[Pharo-project] Fwd: [Vm-dev] Re: out of memory - cog on windows

Andres Valloud avalloud at smalltalk.comcastbiz.net
Sat Apr 23 21:04:00 CEST 2011


I thought I'd mention... some supported Windows configurations require 
additional work to allow apps to use more than 2gb.  So, even if you 
compile with (IIRC) /LARGEADDRESSAWARE, it may not always be enough.

On 4/23/11 11:48 , Andres Valloud wrote:
> Some seemingly useful items to start tracing through Google appear here...
>
> http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/b1548cd5e7c15a8b/05628751f5f46111?pli=1
>
> On 4/23/11 6:17 , Igor Stasenko wrote:
>> On 23 April 2011 13:19, Alain_Rastoul<alr.dev at free.fr>   wrote:
>>> The code for memory allocation is in sqWin32Alloc.c
>>> There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm
>>> (512MB)
>>>    maxReserved = MAX_VIRTUAL_MEMORY;
>>>    do {
>>>      pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS);
>>>      if(!pageBase) {
>>>        if(maxReserved == nowReserved) break;
>>>        /* make it smaller in steps of 128MB */
>>>        maxReserved -= 128*1024*1024;
>>>        if(maxReserved<   nowReserved) maxReserved = nowReserved;
>>>      }
>>>    } while(!pageBase);
>>>
>>> I don't know gcc compilation flag for address space>   2G  for 32b apps on 64
>>> bits ?
>>> ((MS) /LARGEADDRESSEPACE gcc equivalent ?)
>>> Do you know it ?
>>
>> No idea :)
>>
>>>
>>> "Andres Valloud"
>>> <avalloud at smalltalk.comcastbiz.net>   a écrit dans le
>>> message de news: 4DB247E5.7020404 at smalltalk.comcastbiz.net...
>>>> Does that mean that, even if your app uses 128mb of RAM, the VM will
>>>> allocate 512mb of memory regardless?  It seems a bit strange that just 1gb
>>>> worth of allocations would cause problems.  What about the load address of
>>>> the VM and the way the memory chunk is allocated?  Has the 32 bit VM been
>>>> compiled to indicate the app is 32 bit address aware (if not, address
>>>> space is limited to 2gb)?
>>>>
>>>> On 4/22/11 14:44 , Mariano Martinez Peck wrote:
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: *Andreas Raab*<andreas.raab at gmx.de
>>>>> <mailto:andreas.raab at gmx.de>>
>>>>> Date: Fri, Apr 22, 2011 at 3:01 PM
>>>>> Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows
>>>>> To: vm-dev at lists.squeakfoundation.org
>>>>> <mailto:vm-dev at lists.squeakfoundation.org>
>>>>>
>>>>>
>>>>>
>>>>> FWIW, the reason for the 512MB limit has to do with Windows memory
>>>>> layout. When you're running applications that load dynamic libraries
>>>>> (directly or indirectly such as when using ODBC which loads further
>>>>> DLLs) Windows needs (sometimes a lot of) address space to map these DLLs
>>>>> in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY
>>>>> from the OS which will consequently not be available to map DLLs into.
>>>>> We have found that depending on the libraries in use and depending on
>>>>> the overall system utilization, loading DLLs would fail seemingly "at
>>>>> random" which, after further investigation, we were able to track to
>>>>> reserving too much address space upfront. We were able to show
>>>>> experimentally, that changing the limit from 1GB to 512MB would on some
>>>>> machines make all the difference.
>>>>>
>>>>> [*] This is true in particular for libraries that create more threads as
>>>>> the default Windows policy is to create threads with the stack size of
>>>>> the application executable. Thus a 1MB default stack in the application
>>>>> will by default create all further threads to be allocated with a 1MB
>>>>> stack size. Of course all of this is subject to various other conditions.
>>>>>
>>>>> In the end we concluded that 512MB is a reasonable size for most apps
>>>>> and with 512MB we've never seen these random failures. You can increase
>>>>> the limit by recompiling the VM, but if you ship your app in diverse
>>>>> environments you should be aware of the potential issues.
>>>>>
>>>>> Cheers,
>>>>>      - Andreas
>>>>>
>>>>> On 4/21/2011 22:44, Alain_Rastoul wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps
>>>>>> of 128M until it
>>>>>> succeeds.
>>>>>> so I suppose it could be possible to allocate 2Gb and see how it runs
>>>>>> on a 1Gb system and if this is not too much stress for
>>>>>> the system (thinking of the pagefile) ?
>>>>>> Tudor is your vm a cog or non cog vm ?
>>>>>>
>>>>>>       "Mariano Martinez Peck"
>>>>>> <marianopeck at gmail.com
>>>>>>       <mailto:marianopeck at gmail.com>>   a
>>>>>> écrit dans le message de news:
>>>>>>
>>>>>> BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com
>>>>>>
>>>>>> <mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com>...
>>>>>>
>>>>>>       ------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>>       On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba
>>>>>>       <tudor.girba at gmail.com
>>>>>> <mailto:tudor.girba at gmail.com>>   wrote:
>>>>>>
>>>>>>           Hi,
>>>>>>
>>>>>>           Should I to understand that the only way to enable more memory
>>>>>>           is to recompile the VM? Does that mean that there is no way to
>>>>>>           pass this information as a parameter like we can on Mac?
>>>>>>
>>>>>>
>>>>>>       As far as I know you can pass a parameter, but even so, it won't
>>>>>>       be able to allocate more than 512MB.
>>>>>>       I can compile the VM for you with this change in 5 minutes. But I
>>>>>>       am not sure that such simple code would make it work. I think such
>>>>>>       limit is there because of something. Otherwise, it sounds stupid
>>>>>>       imposing a limit just because.
>>>>>>
>>>>>>           The problem is that I cannot recompile the VM because I have
>>>>>>           no access to a Windows machine. Is there one available that
>>>>>>           provides more memory?
>>>>>>
>>>>>>
>>>>>>       I don't think so, but start cc'ing the VM mailing list. You'd
>>>>>>       probably receive more help.
>>>>>>
>>>>>>       Cheers
>>>>>>
>>>>>>       Mariano
>>>>>>
>>>>>>           Cheers,
>>>>>>           Doru
>>>>>>
>>>>>>
>>>>>>           On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
>>>>>>
>>>>>>           >   Hi Tudor,
>>>>>>           >
>>>>>>           >   There is a constant in sqWin32Alloc.h (platforms\win32\vm) :
>>>>>>           >   #define MAX_VIRTUAL_MEMORY 512*1024*1024
>>>>>>           >   you can change it to whatever you want and rebuild the vm,
>>>>>>           >   for exzmple give all the available memory less 256 M .
>>>>>>           >
>>>>>>           >   HTH
>>>>>>           >
>>>>>>           >   Regards
>>>>>>           >   Alain
>>>>>>           >
>>>>>>           >   "Tudor Girba"
>>>>>> <tudor.girba at gmail.com
>>>>>>           <mailto:tudor.girba at gmail.com>>   a
>>>>>> écrit
>>>>>>           >   dans le message de news:
>>>>>>
>>>>>> 03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com
>>>>>>
>>>>>> <mailto:03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com>...
>>>>>>           >   Hi,
>>>>>>           >
>>>>>>           >   We have no specific startUp: methods in Moose. In any case,
>>>>>>           the issue with
>>>>>>           >   opening the image does not seem to be related to startUp:.
>>>>>>           >
>>>>>>           >   Is it really true that the Cog VM is limited to 512MB of
>>>>>> memory?
>>>>>>           >
>>>>>>           >   Cheers,
>>>>>>           >   Doru
>>>>>>           >
>>>>>>           >
>>>>>>           >   On 21 Apr 2011, at 14:27, Luc Fabresse wrote:
>>>>>>           >
>>>>>>           >>   Hi Doru,
>>>>>>           >>
>>>>>>           >>   2011/4/21 Tudor Girba
>>>>>>           >>   <tudor.girba at gmail.com
>>>>>> <mailto:tudor.girba at gmail.com>>
>>>>>>           >>   Hi,
>>>>>>           >>
>>>>>>           >>
>>>>>>           >>
>>>>>>           >>   On Apr 21, 2011, at 14:06, Mariano Martinez Peck
>>>>>>           >>   <marianopeck at gmail.com
>>>>>> <mailto:marianopeck at gmail.com>>   wrote:
>>>>>>           >>
>>>>>>           >>>
>>>>>>           >>>
>>>>>>           >>>   On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba
>>>>>>           >>>   <tudor.girba at gmail.com
>>>>>> <mailto:tudor.girba at gmail.com>>   wrote:
>>>>>>           >>>>   Hi again,
>>>>>>           >>>>
>>>>>>           >>>>   I did not say what the problem was :). The problem was
>>>>>>           that when
>>>>>>           >>>>   opening the image on Windows, he got a Space is low
>>>>>>           message and the
>>>>>>           >>>>   image was not usable (see attachment).
>>>>>>           >>>
>>>>>>           >>>   That's weird. Does moose have something on the startup
>>>>>>           list?   something
>>>>>>           >>>   that can be bothering there?
>>>>>>           >>
>>>>>>           >>   Not that I know of. Is there a way to check this?
>>>>>>           >>
>>>>>>           >>   Classes should be registered using Smalltlak
>>>>>>           addToStartUpList: aClass
>>>>>>           >>   Then aClass class>>#startUp: is executed at startup.
>>>>>>           >>   So implementors of #startUp: on Moose classes should give
>>>>>>           you the answer.
>>>>>>           >>
>>>>>>           >>   #Luc
>>>>>>           >>
>>>>>>           >>
>>>>>>           >>   Actually, using exactly the same process some days ago he
>>>>>>           produced an
>>>>>>           >>   image of 190MB. This one works just fine on Windows. The
>>>>>>           only difference
>>>>>>           >>   between the two is the size of the loaded data.
>>>>>>           >>
>>>>>>           >>   It would be really bad news if the Windows vm would be so
>>>>>>           severely limited
>>>>>>           >>   in terms of memory.
>>>>>>           >>
>>>>>>           >>   Cheers,
>>>>>>           >>   Doru
>>>>>>           >>
>>>>>>           >>
>>>>>>           >>>
>>>>>>           >>>   On Mac it worked just fine.
>>>>>>           >>>>
>>>>>>           >>>>   Cheers,
>>>>>>           >>>>   Doru
>>>>>>           >>>>
>>>>>>           >>>>
>>>>>>           >>>
>>>>>>           >>>>
>>>>>>           >>>>
>>>>>>           >>>>
>>>>>>           >>>>   On 21 Apr 2011, at 12:52, Tudor Girba wrote:
>>>>>>           >>>>
>>>>>>           >>>>>   Hi,
>>>>>>           >>>>>
>>>>>>           >>>>>   I received a question from someone running a 200MB image
>>>>>>           on Windows
>>>>>>           >>>>>   using Cog 2361.
>>>>>>           >>>>>
>>>>>>           >>>>>   If I open the image on Mac, it works just fine.
>>>>>>           Unfortunately, I do
>>>>>>           >>>>>   not have a Windows machine around, and I cannot test but
>>>>>>           I believe it
>>>>>>           >>>>>   should be solvable by increasing the allocated memory.
>>>>>>           >>>>>
>>>>>>           >>>>>   On Mac, I would run it with: ./Croquet -memory 1500m
>>>>>>           >>>>>
>>>>>>           >>>>>   Can anyone help me with the right incantation for Windows?
>>>>>>           >>>>>
>>>>>>           >>>>>   Cheers,
>>>>>>           >>>>>   Doru
>>>>>>           >>>>>
>>>>>>           >>>>>
>>>>>>           >>>>>   --
>>>>>>           >>>>>   www.tudorgirba.com<http://www.tudorgirba.com>
>>>>>>           >>>>>
>>>>>>           >>>>>   "What we can governs what we wish."
>>>>>>           >>>>>
>>>>>>           >>>>>
>>>>>>           >>>>>
>>>>>>           >>>>>   <Space is low.png>
>>>>>>           >>>>
>>>>>>           >>>>   --
>>>>>>           >>>>   www.tudorgirba.com<http://www.tudorgirba.com>
>>>>>>           >>>>
>>>>>>           >>>>   "Yesterday is a fact.
>>>>>>           >>>>   Tomorrow is a possibility.
>>>>>>           >>>>   Today is a challenge."
>>>>>>           >>>>
>>>>>>           >>>>
>>>>>>           >>>>
>>>>>>           >>>
>>>>>>           >>>
>>>>>>           >>>
>>>>>>           >>>
>>>>>>           >>>   --
>>>>>>           >>>   Mariano
>>>>>>           >>>   http://marianopeck.wordpress.com
>>>>>>           >>>
>>>>>>           >>
>>>>>>           >
>>>>>>           >   --
>>>>>>           >   www.tudorgirba.com<http://www.tudorgirba.com>
>>>>>>           >
>>>>>>           >   "Beauty is where we see it."
>>>>>>           >
>>>>>>           >
>>>>>>           >
>>>>>>           >
>>>>>>           >
>>>>>>           >
>>>>>>           >
>>>>>>           >
>>>>>>
>>>>>>           --
>>>>>>           www.tudorgirba.com<http://www.tudorgirba.com>
>>>>>>
>>>>>>           "If you interrupt the barber while he is cutting your hair,
>>>>>>           you will end up with a messy haircut."
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>       --
>>>>>>       Mariano
>>>>>>       http://marianopeck.wordpress.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mariano
>>>>> http://marianopeck.wordpress.com
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>
>



More information about the Pharo-project mailing list