[Pharo-project] Fwd: [Vm-dev] Re: out of memory - cog on windows

Andres Valloud avalloud at smalltalk.comcastbiz.net
Sat Apr 23 20:46:59 CEST 2011


Not off the top of my head, we use VisualStudio for VisualWorks.

On 4/23/11 3:19 , Alain_Rastoul wrote:
> The code for memory allocation is in sqWin32Alloc.c
> There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm
> (512MB)
>    maxReserved = MAX_VIRTUAL_MEMORY;
>    do {
>      pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS);
>      if(!pageBase) {
>        if(maxReserved == nowReserved) break;
>        /* make it smaller in steps of 128MB */
>        maxReserved -= 128*1024*1024;
>        if(maxReserved<  nowReserved) maxReserved = nowReserved;
>      }
>    } while(!pageBase);
>
> I don't know gcc compilation flag for address space>  2G  for 32b apps on 64
> bits ?
> ((MS) /LARGEADDRESSEPACE gcc equivalent ?)
> Do you know it ?
>
> "Andres Valloud"
> <avalloud at smalltalk.comcastbiz.net>  a écrit dans le
> message de news: 4DB247E5.7020404 at smalltalk.comcastbiz.net...
>> Does that mean that, even if your app uses 128mb of RAM, the VM will
>> allocate 512mb of memory regardless?  It seems a bit strange that just 1gb
>> worth of allocations would cause problems.  What about the load address of
>> the VM and the way the memory chunk is allocated?  Has the 32 bit VM been
>> compiled to indicate the app is 32 bit address aware (if not, address
>> space is limited to 2gb)?
>>
>> On 4/22/11 14:44 , Mariano Martinez Peck wrote:
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: *Andreas Raab*<andreas.raab at gmx.de
>>> <mailto:andreas.raab at gmx.de>>
>>> Date: Fri, Apr 22, 2011 at 3:01 PM
>>> Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows
>>> To: vm-dev at lists.squeakfoundation.org
>>> <mailto:vm-dev at lists.squeakfoundation.org>
>>>
>>>
>>>
>>> FWIW, the reason for the 512MB limit has to do with Windows memory
>>> layout. When you're running applications that load dynamic libraries
>>> (directly or indirectly such as when using ODBC which loads further
>>> DLLs) Windows needs (sometimes a lot of) address space to map these DLLs
>>> in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY
>>> from the OS which will consequently not be available to map DLLs into.
>>> We have found that depending on the libraries in use and depending on
>>> the overall system utilization, loading DLLs would fail seemingly "at
>>> random" which, after further investigation, we were able to track to
>>> reserving too much address space upfront. We were able to show
>>> experimentally, that changing the limit from 1GB to 512MB would on some
>>> machines make all the difference.
>>>
>>> [*] This is true in particular for libraries that create more threads as
>>> the default Windows policy is to create threads with the stack size of
>>> the application executable. Thus a 1MB default stack in the application
>>> will by default create all further threads to be allocated with a 1MB
>>> stack size. Of course all of this is subject to various other conditions.
>>>
>>> In the end we concluded that 512MB is a reasonable size for most apps
>>> and with 512MB we've never seen these random failures. You can increase
>>> the limit by recompiling the VM, but if you ship your app in diverse
>>> environments you should be aware of the potential issues.
>>>
>>> Cheers,
>>>     - Andreas
>>>
>>> On 4/21/2011 22:44, Alain_Rastoul wrote:
>>>>
>>>>
>>>>
>>>> Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps
>>>> of 128M until it
>>>> succeeds.
>>>> so I suppose it could be possible to allocate 2Gb and see how it runs
>>>> on a 1Gb system and if this is not too much stress for
>>>> the system (thinking of the pagefile) ?
>>>> Tudor is your vm a cog or non cog vm ?
>>>>
>>>>      "Mariano Martinez Peck"
>>>> <marianopeck at gmail.com
>>>>      <mailto:marianopeck at gmail.com>>  a
>>>> écrit dans le message de news:
>>>>
>>>> BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com
>>>>
>>>> <mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com>...
>>>>
>>>>      ------------------------------------------------------------------------
>>>>
>>>>
>>>>      On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba
>>>>      <tudor.girba at gmail.com
>>>> <mailto:tudor.girba at gmail.com>>  wrote:
>>>>
>>>>          Hi,
>>>>
>>>>          Should I to understand that the only way to enable more memory
>>>>          is to recompile the VM? Does that mean that there is no way to
>>>>          pass this information as a parameter like we can on Mac?
>>>>
>>>>
>>>>      As far as I know you can pass a parameter, but even so, it won't
>>>>      be able to allocate more than 512MB.
>>>>      I can compile the VM for you with this change in 5 minutes. But I
>>>>      am not sure that such simple code would make it work. I think such
>>>>      limit is there because of something. Otherwise, it sounds stupid
>>>>      imposing a limit just because.
>>>>
>>>>          The problem is that I cannot recompile the VM because I have
>>>>          no access to a Windows machine. Is there one available that
>>>>          provides more memory?
>>>>
>>>>
>>>>      I don't think so, but start cc'ing the VM mailing list. You'd
>>>>      probably receive more help.
>>>>
>>>>      Cheers
>>>>
>>>>      Mariano
>>>>
>>>>          Cheers,
>>>>          Doru
>>>>
>>>>
>>>>          On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
>>>>
>>>>          >  Hi Tudor,
>>>>          >
>>>>          >  There is a constant in sqWin32Alloc.h (platforms\win32\vm) :
>>>>          >  #define MAX_VIRTUAL_MEMORY 512*1024*1024
>>>>          >  you can change it to whatever you want and rebuild the vm,
>>>>          >  for exzmple give all the available memory less 256 M .
>>>>          >
>>>>          >  HTH
>>>>          >
>>>>          >  Regards
>>>>          >  Alain
>>>>          >
>>>>          >  "Tudor Girba"
>>>> <tudor.girba at gmail.com
>>>>          <mailto:tudor.girba at gmail.com>>  a
>>>> écrit
>>>>          >  dans le message de news:
>>>>
>>>> 03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com
>>>>
>>>> <mailto:03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com>...
>>>>          >  Hi,
>>>>          >
>>>>          >  We have no specific startUp: methods in Moose. In any case,
>>>>          the issue with
>>>>          >  opening the image does not seem to be related to startUp:.
>>>>          >
>>>>          >  Is it really true that the Cog VM is limited to 512MB of
>>>> memory?
>>>>          >
>>>>          >  Cheers,
>>>>          >  Doru
>>>>          >
>>>>          >
>>>>          >  On 21 Apr 2011, at 14:27, Luc Fabresse wrote:
>>>>          >
>>>>          >>  Hi Doru,
>>>>          >>
>>>>          >>  2011/4/21 Tudor Girba
>>>>          >>  <tudor.girba at gmail.com
>>>> <mailto:tudor.girba at gmail.com>>
>>>>          >>  Hi,
>>>>          >>
>>>>          >>
>>>>          >>
>>>>          >>  On Apr 21, 2011, at 14:06, Mariano Martinez Peck
>>>>          >>  <marianopeck at gmail.com
>>>> <mailto:marianopeck at gmail.com>>  wrote:
>>>>          >>
>>>>          >>>
>>>>          >>>
>>>>          >>>  On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba
>>>>          >>>  <tudor.girba at gmail.com
>>>> <mailto:tudor.girba at gmail.com>>  wrote:
>>>>          >>>>  Hi again,
>>>>          >>>>
>>>>          >>>>  I did not say what the problem was :). The problem was
>>>>          that when
>>>>          >>>>  opening the image on Windows, he got a Space is low
>>>>          message and the
>>>>          >>>>  image was not usable (see attachment).
>>>>          >>>
>>>>          >>>  That's weird. Does moose have something on the startup
>>>>          list?   something
>>>>          >>>  that can be bothering there?
>>>>          >>
>>>>          >>  Not that I know of. Is there a way to check this?
>>>>          >>
>>>>          >>  Classes should be registered using Smalltlak
>>>>          addToStartUpList: aClass
>>>>          >>  Then aClass class>>#startUp: is executed at startup.
>>>>          >>  So implementors of #startUp: on Moose classes should give
>>>>          you the answer.
>>>>          >>
>>>>          >>  #Luc
>>>>          >>
>>>>          >>
>>>>          >>  Actually, using exactly the same process some days ago he
>>>>          produced an
>>>>          >>  image of 190MB. This one works just fine on Windows. The
>>>>          only difference
>>>>          >>  between the two is the size of the loaded data.
>>>>          >>
>>>>          >>  It would be really bad news if the Windows vm would be so
>>>>          severely limited
>>>>          >>  in terms of memory.
>>>>          >>
>>>>          >>  Cheers,
>>>>          >>  Doru
>>>>          >>
>>>>          >>
>>>>          >>>
>>>>          >>>  On Mac it worked just fine.
>>>>          >>>>
>>>>          >>>>  Cheers,
>>>>          >>>>  Doru
>>>>          >>>>
>>>>          >>>>
>>>>          >>>
>>>>          >>>>
>>>>          >>>>
>>>>          >>>>
>>>>          >>>>  On 21 Apr 2011, at 12:52, Tudor Girba wrote:
>>>>          >>>>
>>>>          >>>>>  Hi,
>>>>          >>>>>
>>>>          >>>>>  I received a question from someone running a 200MB image
>>>>          on Windows
>>>>          >>>>>  using Cog 2361.
>>>>          >>>>>
>>>>          >>>>>  If I open the image on Mac, it works just fine.
>>>>          Unfortunately, I do
>>>>          >>>>>  not have a Windows machine around, and I cannot test but
>>>>          I believe it
>>>>          >>>>>  should be solvable by increasing the allocated memory.
>>>>          >>>>>
>>>>          >>>>>  On Mac, I would run it with: ./Croquet -memory 1500m
>>>>          >>>>>
>>>>          >>>>>  Can anyone help me with the right incantation for Windows?
>>>>          >>>>>
>>>>          >>>>>  Cheers,
>>>>          >>>>>  Doru
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>>  --
>>>>          >>>>>  www.tudorgirba.com<http://www.tudorgirba.com>
>>>>          >>>>>
>>>>          >>>>>  "What we can governs what we wish."
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>>  <Space is low.png>
>>>>          >>>>
>>>>          >>>>  --
>>>>          >>>>  www.tudorgirba.com<http://www.tudorgirba.com>
>>>>          >>>>
>>>>          >>>>  "Yesterday is a fact.
>>>>          >>>>  Tomorrow is a possibility.
>>>>          >>>>  Today is a challenge."
>>>>          >>>>
>>>>          >>>>
>>>>          >>>>
>>>>          >>>
>>>>          >>>
>>>>          >>>
>>>>          >>>
>>>>          >>>  --
>>>>          >>>  Mariano
>>>>          >>>  http://marianopeck.wordpress.com
>>>>          >>>
>>>>          >>
>>>>          >
>>>>          >  --
>>>>          >  www.tudorgirba.com<http://www.tudorgirba.com>
>>>>          >
>>>>          >  "Beauty is where we see it."
>>>>          >
>>>>          >
>>>>          >
>>>>          >
>>>>          >
>>>>          >
>>>>          >
>>>>          >
>>>>
>>>>          --
>>>>          www.tudorgirba.com<http://www.tudorgirba.com>
>>>>
>>>>          "If you interrupt the barber while he is cutting your hair,
>>>>          you will end up with a messy haircut."
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>      --
>>>>      Mariano
>>>>      http://marianopeck.wordpress.com
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Mariano
>>> http://marianopeck.wordpress.com
>>>
>>
>>
>
>
>
>
> .
>



More information about the Pharo-project mailing list