[Pharo-project] Fwd: [Vm-dev] Re: out of memory - cog on windows

Igor Stasenko siguctua at gmail.com
Sat Apr 23 15:17:44 CEST 2011


On 23 April 2011 13:19, Alain_Rastoul <alr.dev at free.fr> wrote:
> The code for memory allocation is in sqWin32Alloc.c
> There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm
> (512MB)
>  maxReserved = MAX_VIRTUAL_MEMORY;
>  do {
>    pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS);
>    if(!pageBase) {
>      if(maxReserved == nowReserved) break;
>      /* make it smaller in steps of 128MB */
>      maxReserved -= 128*1024*1024;
>      if(maxReserved < nowReserved) maxReserved = nowReserved;
>    }
>  } while(!pageBase);
>
> I don't know gcc compilation flag for address space > 2G  for 32b apps on 64
> bits ?
> ((MS) /LARGEADDRESSEPACE gcc equivalent ?)
> Do you know it ?

No idea :)

>
> "Andres Valloud"
> <avalloud at smalltalk.comcastbiz.net> a écrit dans le
> message de news: 4DB247E5.7020404 at smalltalk.comcastbiz.net...
>> Does that mean that, even if your app uses 128mb of RAM, the VM will
>> allocate 512mb of memory regardless?  It seems a bit strange that just 1gb
>> worth of allocations would cause problems.  What about the load address of
>> the VM and the way the memory chunk is allocated?  Has the 32 bit VM been
>> compiled to indicate the app is 32 bit address aware (if not, address
>> space is limited to 2gb)?
>>
>> On 4/22/11 14:44 , Mariano Martinez Peck wrote:
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: *Andreas Raab* <andreas.raab at gmx.de
>>> <mailto:andreas.raab at gmx.de>>
>>> Date: Fri, Apr 22, 2011 at 3:01 PM
>>> Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows
>>> To: vm-dev at lists.squeakfoundation.org
>>> <mailto:vm-dev at lists.squeakfoundation.org>
>>>
>>>
>>>
>>> FWIW, the reason for the 512MB limit has to do with Windows memory
>>> layout. When you're running applications that load dynamic libraries
>>> (directly or indirectly such as when using ODBC which loads further
>>> DLLs) Windows needs (sometimes a lot of) address space to map these DLLs
>>> in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY
>>> from the OS which will consequently not be available to map DLLs into.
>>> We have found that depending on the libraries in use and depending on
>>> the overall system utilization, loading DLLs would fail seemingly "at
>>> random" which, after further investigation, we were able to track to
>>> reserving too much address space upfront. We were able to show
>>> experimentally, that changing the limit from 1GB to 512MB would on some
>>> machines make all the difference.
>>>
>>> [*] This is true in particular for libraries that create more threads as
>>> the default Windows policy is to create threads with the stack size of
>>> the application executable. Thus a 1MB default stack in the application
>>> will by default create all further threads to be allocated with a 1MB
>>> stack size. Of course all of this is subject to various other conditions.
>>>
>>> In the end we concluded that 512MB is a reasonable size for most apps
>>> and with 512MB we've never seen these random failures. You can increase
>>> the limit by recompiling the VM, but if you ship your app in diverse
>>> environments you should be aware of the potential issues.
>>>
>>> Cheers,
>>>    - Andreas
>>>
>>> On 4/21/2011 22:44, Alain_Rastoul wrote:
>>>>
>>>>
>>>>
>>>> Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps
>>>> of 128M until it
>>>> succeeds.
>>>> so I suppose it could be possible to allocate 2Gb and see how it runs
>>>> on a 1Gb system and if this is not too much stress for
>>>> the system (thinking of the pagefile) ?
>>>> Tudor is your vm a cog or non cog vm ?
>>>>
>>>>     "Mariano Martinez Peck"
>>>> <marianopeck at gmail.com
>>>>     <mailto:marianopeck at gmail.com>> a
>>>> écrit dans le message de news:
>>>>
>>>> BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com
>>>>
>>>> <mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com>...
>>>>
>>>>     ------------------------------------------------------------------------
>>>>
>>>>
>>>>     On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba
>>>>     <tudor.girba at gmail.com
>>>> <mailto:tudor.girba at gmail.com>> wrote:
>>>>
>>>>         Hi,
>>>>
>>>>         Should I to understand that the only way to enable more memory
>>>>         is to recompile the VM? Does that mean that there is no way to
>>>>         pass this information as a parameter like we can on Mac?
>>>>
>>>>
>>>>     As far as I know you can pass a parameter, but even so, it won't
>>>>     be able to allocate more than 512MB.
>>>>     I can compile the VM for you with this change in 5 minutes. But I
>>>>     am not sure that such simple code would make it work. I think such
>>>>     limit is there because of something. Otherwise, it sounds stupid
>>>>     imposing a limit just because.
>>>>
>>>>         The problem is that I cannot recompile the VM because I have
>>>>         no access to a Windows machine. Is there one available that
>>>>         provides more memory?
>>>>
>>>>
>>>>     I don't think so, but start cc'ing the VM mailing list. You'd
>>>>     probably receive more help.
>>>>
>>>>     Cheers
>>>>
>>>>     Mariano
>>>>
>>>>         Cheers,
>>>>         Doru
>>>>
>>>>
>>>>         On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
>>>>
>>>>         > Hi Tudor,
>>>>         >
>>>>         > There is a constant in sqWin32Alloc.h (platforms\win32\vm) :
>>>>         > #define MAX_VIRTUAL_MEMORY 512*1024*1024
>>>>         > you can change it to whatever you want and rebuild the vm,
>>>>         > for exzmple give all the available memory less 256 M .
>>>>         >
>>>>         > HTH
>>>>         >
>>>>         > Regards
>>>>         > Alain
>>>>         >
>>>>         > "Tudor Girba"
>>>> <tudor.girba at gmail.com
>>>>         <mailto:tudor.girba at gmail.com>> a
>>>> écrit
>>>>         > dans le message de news:
>>>>
>>>> 03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com
>>>>
>>>> <mailto:03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com>...
>>>>         > Hi,
>>>>         >
>>>>         > We have no specific startUp: methods in Moose. In any case,
>>>>         the issue with
>>>>         > opening the image does not seem to be related to startUp:.
>>>>         >
>>>>         > Is it really true that the Cog VM is limited to 512MB of
>>>> memory?
>>>>         >
>>>>         > Cheers,
>>>>         > Doru
>>>>         >
>>>>         >
>>>>         > On 21 Apr 2011, at 14:27, Luc Fabresse wrote:
>>>>         >
>>>>         >> Hi Doru,
>>>>         >>
>>>>         >> 2011/4/21 Tudor Girba
>>>>         >> <tudor.girba at gmail.com
>>>> <mailto:tudor.girba at gmail.com>>
>>>>         >> Hi,
>>>>         >>
>>>>         >>
>>>>         >>
>>>>         >> On Apr 21, 2011, at 14:06, Mariano Martinez Peck
>>>>         >> <marianopeck at gmail.com
>>>> <mailto:marianopeck at gmail.com>> wrote:
>>>>         >>
>>>>         >>>
>>>>         >>>
>>>>         >>> On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba
>>>>         >>> <tudor.girba at gmail.com
>>>> <mailto:tudor.girba at gmail.com>> wrote:
>>>>         >>>> Hi again,
>>>>         >>>>
>>>>         >>>> I did not say what the problem was :). The problem was
>>>>         that when
>>>>         >>>> opening the image on Windows, he got a Space is low
>>>>         message and the
>>>>         >>>> image was not usable (see attachment).
>>>>         >>>
>>>>         >>> That's weird. Does moose have something on the startup
>>>>         list?   something
>>>>         >>> that can be bothering there?
>>>>         >>
>>>>         >> Not that I know of. Is there a way to check this?
>>>>         >>
>>>>         >> Classes should be registered using Smalltlak
>>>>         addToStartUpList: aClass
>>>>         >> Then aClass class>>#startUp: is executed at startup.
>>>>         >> So implementors of #startUp: on Moose classes should give
>>>>         you the answer.
>>>>         >>
>>>>         >> #Luc
>>>>         >>
>>>>         >>
>>>>         >> Actually, using exactly the same process some days ago he
>>>>         produced an
>>>>         >> image of 190MB. This one works just fine on Windows. The
>>>>         only difference
>>>>         >> between the two is the size of the loaded data.
>>>>         >>
>>>>         >> It would be really bad news if the Windows vm would be so
>>>>         severely limited
>>>>         >> in terms of memory.
>>>>         >>
>>>>         >> Cheers,
>>>>         >> Doru
>>>>         >>
>>>>         >>
>>>>         >>>
>>>>         >>> On Mac it worked just fine.
>>>>         >>>>
>>>>         >>>> Cheers,
>>>>         >>>> Doru
>>>>         >>>>
>>>>         >>>>
>>>>         >>>
>>>>         >>>>
>>>>         >>>>
>>>>         >>>>
>>>>         >>>> On 21 Apr 2011, at 12:52, Tudor Girba wrote:
>>>>         >>>>
>>>>         >>>>> Hi,
>>>>         >>>>>
>>>>         >>>>> I received a question from someone running a 200MB image
>>>>         on Windows
>>>>         >>>>> using Cog 2361.
>>>>         >>>>>
>>>>         >>>>> If I open the image on Mac, it works just fine.
>>>>         Unfortunately, I do
>>>>         >>>>> not have a Windows machine around, and I cannot test but
>>>>         I believe it
>>>>         >>>>> should be solvable by increasing the allocated memory.
>>>>         >>>>>
>>>>         >>>>> On Mac, I would run it with: ./Croquet -memory 1500m
>>>>         >>>>>
>>>>         >>>>> Can anyone help me with the right incantation for Windows?
>>>>         >>>>>
>>>>         >>>>> Cheers,
>>>>         >>>>> Doru
>>>>         >>>>>
>>>>         >>>>>
>>>>         >>>>> --
>>>>         >>>>> www.tudorgirba.com <http://www.tudorgirba.com>
>>>>         >>>>>
>>>>         >>>>> "What we can governs what we wish."
>>>>         >>>>>
>>>>         >>>>>
>>>>         >>>>>
>>>>         >>>>> <Space is low.png>
>>>>         >>>>
>>>>         >>>> --
>>>>         >>>> www.tudorgirba.com <http://www.tudorgirba.com>
>>>>         >>>>
>>>>         >>>> "Yesterday is a fact.
>>>>         >>>> Tomorrow is a possibility.
>>>>         >>>> Today is a challenge."
>>>>         >>>>
>>>>         >>>>
>>>>         >>>>
>>>>         >>>
>>>>         >>>
>>>>         >>>
>>>>         >>>
>>>>         >>> --
>>>>         >>> Mariano
>>>>         >>> http://marianopeck.wordpress.com
>>>>         >>>
>>>>         >>
>>>>         >
>>>>         > --
>>>>         > www.tudorgirba.com <http://www.tudorgirba.com>
>>>>         >
>>>>         > "Beauty is where we see it."
>>>>         >
>>>>         >
>>>>         >
>>>>         >
>>>>         >
>>>>         >
>>>>         >
>>>>         >
>>>>
>>>>         --
>>>>         www.tudorgirba.com <http://www.tudorgirba.com>
>>>>
>>>>         "If you interrupt the barber while he is cutting your hair,
>>>>         you will end up with a messy haircut."
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>     --
>>>>     Mariano
>>>>     http://marianopeck.wordpress.com
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Mariano
>>> http://marianopeck.wordpress.com
>>>
>>
>>
>
>
>
>
>



-- 
Best regards,
Igor Stasenko AKA sig.



More information about the Pharo-project mailing list