[Pharo-project] Fwd: [Vm-dev] Re: out of memory - cog on windows

Andres Valloud avalloud at smalltalk.comcastbiz.net
Sat Apr 23 20:48:52 CEST 2011


Some seemingly useful items to start tracing through Google appear here...

http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/b1548cd5e7c15a8b/05628751f5f46111?pli=1

On 4/23/11 6:17 , Igor Stasenko wrote:
> On 23 April 2011 13:19, Alain_Rastoul<alr.dev at free.fr>  wrote:
>> The code for memory allocation is in sqWin32Alloc.c
>> There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm
>> (512MB)
>>   maxReserved = MAX_VIRTUAL_MEMORY;
>>   do {
>>     pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS);
>>     if(!pageBase) {
>>       if(maxReserved == nowReserved) break;
>>       /* make it smaller in steps of 128MB */
>>       maxReserved -= 128*1024*1024;
>>       if(maxReserved<  nowReserved) maxReserved = nowReserved;
>>     }
>>   } while(!pageBase);
>>
>> I don't know gcc compilation flag for address space>  2G  for 32b apps on 64
>> bits ?
>> ((MS) /LARGEADDRESSEPACE gcc equivalent ?)
>> Do you know it ?
>
> No idea :)
>
>>
>> "Andres Valloud"
>> <avalloud at smalltalk.comcastbiz.net>  a écrit dans le
>> message de news: 4DB247E5.7020404 at smalltalk.comcastbiz.net...
>>> Does that mean that, even if your app uses 128mb of RAM, the VM will
>>> allocate 512mb of memory regardless?  It seems a bit strange that just 1gb
>>> worth of allocations would cause problems.  What about the load address of
>>> the VM and the way the memory chunk is allocated?  Has the 32 bit VM been
>>> compiled to indicate the app is 32 bit address aware (if not, address
>>> space is limited to 2gb)?
>>>
>>> On 4/22/11 14:44 , Mariano Martinez Peck wrote:
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: *Andreas Raab*<andreas.raab at gmx.de
>>>> <mailto:andreas.raab at gmx.de>>
>>>> Date: Fri, Apr 22, 2011 at 3:01 PM
>>>> Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows
>>>> To: vm-dev at lists.squeakfoundation.org
>>>> <mailto:vm-dev at lists.squeakfoundation.org>
>>>>
>>>>
>>>>
>>>> FWIW, the reason for the 512MB limit has to do with Windows memory
>>>> layout. When you're running applications that load dynamic libraries
>>>> (directly or indirectly such as when using ODBC which loads further
>>>> DLLs) Windows needs (sometimes a lot of) address space to map these DLLs
>>>> in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY
>>>> from the OS which will consequently not be available to map DLLs into.
>>>> We have found that depending on the libraries in use and depending on
>>>> the overall system utilization, loading DLLs would fail seemingly "at
>>>> random" which, after further investigation, we were able to track to
>>>> reserving too much address space upfront. We were able to show
>>>> experimentally, that changing the limit from 1GB to 512MB would on some
>>>> machines make all the difference.
>>>>
>>>> [*] This is true in particular for libraries that create more threads as
>>>> the default Windows policy is to create threads with the stack size of
>>>> the application executable. Thus a 1MB default stack in the application
>>>> will by default create all further threads to be allocated with a 1MB
>>>> stack size. Of course all of this is subject to various other conditions.
>>>>
>>>> In the end we concluded that 512MB is a reasonable size for most apps
>>>> and with 512MB we've never seen these random failures. You can increase
>>>> the limit by recompiling the VM, but if you ship your app in diverse
>>>> environments you should be aware of the potential issues.
>>>>
>>>> Cheers,
>>>>     - Andreas
>>>>
>>>> On 4/21/2011 22:44, Alain_Rastoul wrote:
>>>>>
>>>>>
>>>>>
>>>>> Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps
>>>>> of 128M until it
>>>>> succeeds.
>>>>> so I suppose it could be possible to allocate 2Gb and see how it runs
>>>>> on a 1Gb system and if this is not too much stress for
>>>>> the system (thinking of the pagefile) ?
>>>>> Tudor is your vm a cog or non cog vm ?
>>>>>
>>>>>      "Mariano Martinez Peck"
>>>>> <marianopeck at gmail.com
>>>>>      <mailto:marianopeck at gmail.com>>  a
>>>>> écrit dans le message de news:
>>>>>
>>>>> BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com
>>>>>
>>>>> <mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g at mail.gmail.com>...
>>>>>
>>>>>      ------------------------------------------------------------------------
>>>>>
>>>>>
>>>>>      On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba
>>>>>      <tudor.girba at gmail.com
>>>>> <mailto:tudor.girba at gmail.com>>  wrote:
>>>>>
>>>>>          Hi,
>>>>>
>>>>>          Should I to understand that the only way to enable more memory
>>>>>          is to recompile the VM? Does that mean that there is no way to
>>>>>          pass this information as a parameter like we can on Mac?
>>>>>
>>>>>
>>>>>      As far as I know you can pass a parameter, but even so, it won't
>>>>>      be able to allocate more than 512MB.
>>>>>      I can compile the VM for you with this change in 5 minutes. But I
>>>>>      am not sure that such simple code would make it work. I think such
>>>>>      limit is there because of something. Otherwise, it sounds stupid
>>>>>      imposing a limit just because.
>>>>>
>>>>>          The problem is that I cannot recompile the VM because I have
>>>>>          no access to a Windows machine. Is there one available that
>>>>>          provides more memory?
>>>>>
>>>>>
>>>>>      I don't think so, but start cc'ing the VM mailing list. You'd
>>>>>      probably receive more help.
>>>>>
>>>>>      Cheers
>>>>>
>>>>>      Mariano
>>>>>
>>>>>          Cheers,
>>>>>          Doru
>>>>>
>>>>>
>>>>>          On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
>>>>>
>>>>>          >  Hi Tudor,
>>>>>          >
>>>>>          >  There is a constant in sqWin32Alloc.h (platforms\win32\vm) :
>>>>>          >  #define MAX_VIRTUAL_MEMORY 512*1024*1024
>>>>>          >  you can change it to whatever you want and rebuild the vm,
>>>>>          >  for exzmple give all the available memory less 256 M .
>>>>>          >
>>>>>          >  HTH
>>>>>          >
>>>>>          >  Regards
>>>>>          >  Alain
>>>>>          >
>>>>>          >  "Tudor Girba"
>>>>> <tudor.girba at gmail.com
>>>>>          <mailto:tudor.girba at gmail.com>>  a
>>>>> écrit
>>>>>          >  dans le message de news:
>>>>>
>>>>> 03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com
>>>>>
>>>>> <mailto:03B9389F-C719-44D0-B106-2AC78B120F56 at gmail.com>...
>>>>>          >  Hi,
>>>>>          >
>>>>>          >  We have no specific startUp: methods in Moose. In any case,
>>>>>          the issue with
>>>>>          >  opening the image does not seem to be related to startUp:.
>>>>>          >
>>>>>          >  Is it really true that the Cog VM is limited to 512MB of
>>>>> memory?
>>>>>          >
>>>>>          >  Cheers,
>>>>>          >  Doru
>>>>>          >
>>>>>          >
>>>>>          >  On 21 Apr 2011, at 14:27, Luc Fabresse wrote:
>>>>>          >
>>>>>          >>  Hi Doru,
>>>>>          >>
>>>>>          >>  2011/4/21 Tudor Girba
>>>>>          >>  <tudor.girba at gmail.com
>>>>> <mailto:tudor.girba at gmail.com>>
>>>>>          >>  Hi,
>>>>>          >>
>>>>>          >>
>>>>>          >>
>>>>>          >>  On Apr 21, 2011, at 14:06, Mariano Martinez Peck
>>>>>          >>  <marianopeck at gmail.com
>>>>> <mailto:marianopeck at gmail.com>>  wrote:
>>>>>          >>
>>>>>          >>>
>>>>>          >>>
>>>>>          >>>  On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba
>>>>>          >>>  <tudor.girba at gmail.com
>>>>> <mailto:tudor.girba at gmail.com>>  wrote:
>>>>>          >>>>  Hi again,
>>>>>          >>>>
>>>>>          >>>>  I did not say what the problem was :). The problem was
>>>>>          that when
>>>>>          >>>>  opening the image on Windows, he got a Space is low
>>>>>          message and the
>>>>>          >>>>  image was not usable (see attachment).
>>>>>          >>>
>>>>>          >>>  That's weird. Does moose have something on the startup
>>>>>          list?   something
>>>>>          >>>  that can be bothering there?
>>>>>          >>
>>>>>          >>  Not that I know of. Is there a way to check this?
>>>>>          >>
>>>>>          >>  Classes should be registered using Smalltlak
>>>>>          addToStartUpList: aClass
>>>>>          >>  Then aClass class>>#startUp: is executed at startup.
>>>>>          >>  So implementors of #startUp: on Moose classes should give
>>>>>          you the answer.
>>>>>          >>
>>>>>          >>  #Luc
>>>>>          >>
>>>>>          >>
>>>>>          >>  Actually, using exactly the same process some days ago he
>>>>>          produced an
>>>>>          >>  image of 190MB. This one works just fine on Windows. The
>>>>>          only difference
>>>>>          >>  between the two is the size of the loaded data.
>>>>>          >>
>>>>>          >>  It would be really bad news if the Windows vm would be so
>>>>>          severely limited
>>>>>          >>  in terms of memory.
>>>>>          >>
>>>>>          >>  Cheers,
>>>>>          >>  Doru
>>>>>          >>
>>>>>          >>
>>>>>          >>>
>>>>>          >>>  On Mac it worked just fine.
>>>>>          >>>>
>>>>>          >>>>  Cheers,
>>>>>          >>>>  Doru
>>>>>          >>>>
>>>>>          >>>>
>>>>>          >>>
>>>>>          >>>>
>>>>>          >>>>
>>>>>          >>>>
>>>>>          >>>>  On 21 Apr 2011, at 12:52, Tudor Girba wrote:
>>>>>          >>>>
>>>>>          >>>>>  Hi,
>>>>>          >>>>>
>>>>>          >>>>>  I received a question from someone running a 200MB image
>>>>>          on Windows
>>>>>          >>>>>  using Cog 2361.
>>>>>          >>>>>
>>>>>          >>>>>  If I open the image on Mac, it works just fine.
>>>>>          Unfortunately, I do
>>>>>          >>>>>  not have a Windows machine around, and I cannot test but
>>>>>          I believe it
>>>>>          >>>>>  should be solvable by increasing the allocated memory.
>>>>>          >>>>>
>>>>>          >>>>>  On Mac, I would run it with: ./Croquet -memory 1500m
>>>>>          >>>>>
>>>>>          >>>>>  Can anyone help me with the right incantation for Windows?
>>>>>          >>>>>
>>>>>          >>>>>  Cheers,
>>>>>          >>>>>  Doru
>>>>>          >>>>>
>>>>>          >>>>>
>>>>>          >>>>>  --
>>>>>          >>>>>  www.tudorgirba.com<http://www.tudorgirba.com>
>>>>>          >>>>>
>>>>>          >>>>>  "What we can governs what we wish."
>>>>>          >>>>>
>>>>>          >>>>>
>>>>>          >>>>>
>>>>>          >>>>>  <Space is low.png>
>>>>>          >>>>
>>>>>          >>>>  --
>>>>>          >>>>  www.tudorgirba.com<http://www.tudorgirba.com>
>>>>>          >>>>
>>>>>          >>>>  "Yesterday is a fact.
>>>>>          >>>>  Tomorrow is a possibility.
>>>>>          >>>>  Today is a challenge."
>>>>>          >>>>
>>>>>          >>>>
>>>>>          >>>>
>>>>>          >>>
>>>>>          >>>
>>>>>          >>>
>>>>>          >>>
>>>>>          >>>  --
>>>>>          >>>  Mariano
>>>>>          >>>  http://marianopeck.wordpress.com
>>>>>          >>>
>>>>>          >>
>>>>>          >
>>>>>          >  --
>>>>>          >  www.tudorgirba.com<http://www.tudorgirba.com>
>>>>>          >
>>>>>          >  "Beauty is where we see it."
>>>>>          >
>>>>>          >
>>>>>          >
>>>>>          >
>>>>>          >
>>>>>          >
>>>>>          >
>>>>>          >
>>>>>
>>>>>          --
>>>>>          www.tudorgirba.com<http://www.tudorgirba.com>
>>>>>
>>>>>          "If you interrupt the barber while he is cutting your hair,
>>>>>          you will end up with a messy haircut."
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>      --
>>>>>      Mariano
>>>>>      http://marianopeck.wordpress.com
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Mariano
>>>> http://marianopeck.wordpress.com
>>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>



More information about the Pharo-project mailing list