[Pharo-project] Fwd: [Vm-dev] Re: out of memory - cog on windows

Alain_Rastoul alr.dev at free.fr
Sat Apr 23 12:19:01 CEST 2011


The code for memory allocation is in sqWin32Alloc.c
There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm 
(512MB)
  maxReserved = MAX_VIRTUAL_MEMORY;
  do {
    pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS);
    if(!pageBase) {
      if(maxReserved == nowReserved) break;
      /* make it smaller in steps of 128MB */
      maxReserved -= 128*1024*1024;
      if(maxReserved < nowReserved) maxReserved = nowReserved;
    }
  } while(!pageBase);

I don't know gcc compilation flag for address space > 2G  for 32b apps on 64 
bits ?
((MS) /LARGEADDRESSEPACE gcc equivalent ?)
Do you know it ?

"Andres Valloud" 
<avalloud at smalltalk.comcastbiz.net> a écrit dans le 
message de news: 4DB247E5.7020404 at smalltalk.comcastbiz.net...
> Does that mean that, even if your app uses 128mb of RAM, the VM will 
> allocate 512mb of memory regardless?  It seems a bit strange that just 1gb 
> worth of allocations would cause problems.  What about the load address of 
> the VM and the way the memory chunk is allocated?  Has the 32 bit VM been 
> compiled to indicate the app is 32 bit address aware (if not, address 
> space is limited to 2gb)?
>
> On 4/22/11 14:44 , Mariano Martinez Peck wrote:
>>
>>
>> ---------- Forwarded message ----------
>> From: *Andreas Raab* <andreas.raab at gmx.de 
>> <mailto:andreas.raab at gmx.de>>
>> Date: Fri, Apr 22, 2011 at 3:01 PM
>> Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows
>> To: vm-dev at lists.squeakfoundation.org
>> <mailto:vm-dev at lists.squeakfoundation.org>
>>
>>
>>
>> FWIW, the reason for the 512MB limit has to do with Windows memory
>> layout. When you're running applications that load dynamic libraries
>> (directly or indirectly such as when using ODBC which loads further
>> DLLs) Windows needs (sometimes a lot of) address space to map these DLLs
>> in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY
>> from the OS which will consequently not be available to map DLLs into.
>> We have found that depending on the libraries in use and depending on
>> the overall system utilization, loading DLLs would fail seemingly "at
>> random" which, after further investigation, we were able to track to
>> reserving too much address space upfront. We were able to show
>> experimentally, that changing the limit from 1GB to 512MB would on some
>> machines make all the difference.
>>
>> [*] This is true in particular for libraries that create more threads as
>> the default Windows policy is to create threads with the stack size of
>> the application executable. Thus a 1MB default stack in the application
>> will by default create all further threads to be allocated with a 1MB
>> stack size. Of course all of this is subject to various other conditions.
>>
>> In the end we concluded that 512MB is a reasonable size for most apps
>> and with 512MB we've never seen these random failures. You can increase
>> the limit by recompiling the VM, but if you ship your app in diverse
>> environments you should be aware of the potential issues.
>>
>> Cheers,
>>    - Andreas
>>
>> On 4/21/2011 22:44, Alain_Rastoul wrote:
>>>
>>>
>>>
>>> Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps
>>> of 128M until it
>>> succeeds.
>>> so I suppose it could be possible to allocate 2Gb and see how it runs
>>> on a 1Gb system and if this is not too much stress for
>>> the system (thinking of the pagefile) ?
>>> Tudor is your vm a cog or non cog vm ?
>>>
>>>     "Mariano Martinez Peck" 
>>> <marianopeck at gmail.com
>>>     <mailto:marianopeck at gmail.com>> a 
>>> écrit dans le message de news:
>>> 
>>> BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com
>>> 
>>> <mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com>...
>>>
>>>     ------------------------------------------------------------------------
>>>
>>>
>>>     On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba
>>>     <tudor.girba at gmail.com 
>>> <mailto:tudor.girba at gmail.com>> wrote:
>>>
>>>         Hi,
>>>
>>>         Should I to understand that the only way to enable more memory
>>>         is to recompile the VM? Does that mean that there is no way to
>>>         pass this information as a parameter like we can on Mac?
>>>
>>>
>>>     As far as I know you can pass a parameter, but even so, it won't
>>>     be able to allocate more than 512MB.
>>>     I can compile the VM for you with this change in 5 minutes. But I
>>>     am not sure that such simple code would make it work. I think such
>>>     limit is there because of something. Otherwise, it sounds stupid
>>>     imposing a limit just because.
>>>
>>>         The problem is that I cannot recompile the VM because I have
>>>         no access to a Windows machine. Is there one available that
>>>         provides more memory?
>>>
>>>
>>>     I don't think so, but start cc'ing the VM mailing list. You'd
>>>     probably receive more help.
>>>
>>>     Cheers
>>>
>>>     Mariano
>>>
>>>         Cheers,
>>>         Doru
>>>
>>>
>>>         On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
>>>
>>>         > Hi Tudor,
>>>         >
>>>         > There is a constant in sqWin32Alloc.h (platforms\win32\vm) :
>>>         > #define MAX_VIRTUAL_MEMORY 512*1024*1024
>>>         > you can change it to whatever you want and rebuild the vm,
>>>         > for exzmple give all the available memory less 256 M .
>>>         >
>>>         > HTH
>>>         >
>>>         > Regards
>>>         > Alain
>>>         >
>>>         > "Tudor Girba" 
>>> <tudor.girba at gmail.com
>>>         <mailto:tudor.girba at gmail.com>> a 
>>> écrit
>>>         > dans le message de news:
>>> 
>>> 03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com
>>> 
>>> <mailto:03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com>...
>>>         > Hi,
>>>         >
>>>         > We have no specific startUp: methods in Moose. In any case,
>>>         the issue with
>>>         > opening the image does not seem to be related to startUp:.
>>>         >
>>>         > Is it really true that the Cog VM is limited to 512MB of 
>>> memory?
>>>         >
>>>         > Cheers,
>>>         > Doru
>>>         >
>>>         >
>>>         > On 21 Apr 2011, at 14:27, Luc Fabresse wrote:
>>>         >
>>>         >> Hi Doru,
>>>         >>
>>>         >> 2011/4/21 Tudor Girba
>>>         >> <tudor.girba at gmail.com 
>>> <mailto:tudor.girba at gmail.com>>
>>>         >> Hi,
>>>         >>
>>>         >>
>>>         >>
>>>         >> On Apr 21, 2011, at 14:06, Mariano Martinez Peck
>>>         >> <marianopeck at gmail.com 
>>> <mailto:marianopeck at gmail.com>> wrote:
>>>         >>
>>>         >>>
>>>         >>>
>>>         >>> On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba
>>>         >>> <tudor.girba at gmail.com 
>>> <mailto:tudor.girba at gmail.com>> wrote:
>>>         >>>> Hi again,
>>>         >>>>
>>>         >>>> I did not say what the problem was :). The problem was
>>>         that when
>>>         >>>> opening the image on Windows, he got a Space is low
>>>         message and the
>>>         >>>> image was not usable (see attachment).
>>>         >>>
>>>         >>> That's weird. Does moose have something on the startup
>>>         list?   something
>>>         >>> that can be bothering there?
>>>         >>
>>>         >> Not that I know of. Is there a way to check this?
>>>         >>
>>>         >> Classes should be registered using Smalltlak
>>>         addToStartUpList: aClass
>>>         >> Then aClass class>>#startUp: is executed at startup.
>>>         >> So implementors of #startUp: on Moose classes should give
>>>         you the answer.
>>>         >>
>>>         >> #Luc
>>>         >>
>>>         >>
>>>         >> Actually, using exactly the same process some days ago he
>>>         produced an
>>>         >> image of 190MB. This one works just fine on Windows. The
>>>         only difference
>>>         >> between the two is the size of the loaded data.
>>>         >>
>>>         >> It would be really bad news if the Windows vm would be so
>>>         severely limited
>>>         >> in terms of memory.
>>>         >>
>>>         >> Cheers,
>>>         >> Doru
>>>         >>
>>>         >>
>>>         >>>
>>>         >>> On Mac it worked just fine.
>>>         >>>>
>>>         >>>> Cheers,
>>>         >>>> Doru
>>>         >>>>
>>>         >>>>
>>>         >>>
>>>         >>>>
>>>         >>>>
>>>         >>>>
>>>         >>>> On 21 Apr 2011, at 12:52, Tudor Girba wrote:
>>>         >>>>
>>>         >>>>> Hi,
>>>         >>>>>
>>>         >>>>> I received a question from someone running a 200MB image
>>>         on Windows
>>>         >>>>> using Cog 2361.
>>>         >>>>>
>>>         >>>>> If I open the image on Mac, it works just fine.
>>>         Unfortunately, I do
>>>         >>>>> not have a Windows machine around, and I cannot test but
>>>         I believe it
>>>         >>>>> should be solvable by increasing the allocated memory.
>>>         >>>>>
>>>         >>>>> On Mac, I would run it with: ./Croquet -memory 1500m
>>>         >>>>>
>>>         >>>>> Can anyone help me with the right incantation for Windows?
>>>         >>>>>
>>>         >>>>> Cheers,
>>>         >>>>> Doru
>>>         >>>>>
>>>         >>>>>
>>>         >>>>> --
>>>         >>>>> www.tudorgirba.com <http://www.tudorgirba.com>
>>>         >>>>>
>>>         >>>>> "What we can governs what we wish."
>>>         >>>>>
>>>         >>>>>
>>>         >>>>>
>>>         >>>>> <Space is low.png>
>>>         >>>>
>>>         >>>> --
>>>         >>>> www.tudorgirba.com <http://www.tudorgirba.com>
>>>         >>>>
>>>         >>>> "Yesterday is a fact.
>>>         >>>> Tomorrow is a possibility.
>>>         >>>> Today is a challenge."
>>>         >>>>
>>>         >>>>
>>>         >>>>
>>>         >>>
>>>         >>>
>>>         >>>
>>>         >>>
>>>         >>> --
>>>         >>> Mariano
>>>         >>> http://marianopeck.wordpress.com
>>>         >>>
>>>         >>
>>>         >
>>>         > --
>>>         > www.tudorgirba.com <http://www.tudorgirba.com>
>>>         >
>>>         > "Beauty is where we see it."
>>>         >
>>>         >
>>>         >
>>>         >
>>>         >
>>>         >
>>>         >
>>>         >
>>>
>>>         --
>>>         www.tudorgirba.com <http://www.tudorgirba.com>
>>>
>>>         "If you interrupt the barber while he is cutting your hair,
>>>         you will end up with a messy haircut."
>>>
>>>
>>>
>>>
>>>
>>>     --
>>>     Mariano
>>>     http://marianopeck.wordpress.com
>>>
>>
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>>
>
> 






More information about the Pharo-project mailing list