[Pharo-project] Remove #class as a Compiler optimization

Nicolas Cellier nicolas.cellier.aka.nice at gmail.com
Tue Apr 5 21:27:21 CEST 2011


Sorry, no time to read by now, so maybe I should better have shut up...
Nonetheless, it much depends on services provided by OS.
If we only access the wall clock, then there will be noise due to
other OS processes...
... but this noise is unlikely following a gaussian with 0 mean, does it ?
In this case, it makes sense to filter out the outliers, and I would
say this simple stupid method could possibly work:
- take the median
- and the standard deviation after filtering out a fixed percentage of outliers
But you probably have more ellaborated and well thought method :)

Nicolas

2011/4/5 Camillo Bruni <camillo.bruni at inria.fr>:
>
> On 2011-04-05, at 15:02, Nicolas Cellier wrote:
>
>> Better than average, take the median
>
> NO! thats why you provide the standard deviation...
>
> you should not optimize the results prematurely but provide numbers that let you decide on the quality of the benchmarks. what you can do is optimize for systematic errors, like warm up and stuff, but nevertheless you should provide the raw results...
>
> read the paper...
>
>> Nicolas
>>
>> 2011/4/5 Camillo Bruni <camillo.bruni at inria.fr>:
>>> This is exactly why you have to provide some confidence interval / deviation, otherwise it is hard to make any reasonable conclusion.
>>>
>>> run it 100 times and take the average and provide the standard deviation.
>>>
>>> I am not a big fan of relying on incomplete benchmarking results:
>>>
>>> Please read: http://portal.acm.org/citation.cfm?id=1297033
>>>
>>> http://www.squeaksource.com/p.html provides a basic benchmarking framework under the NBenchmark package. You subclass from PBenchmarkSuite implement a method #benchXXX and run it.
>>>
>>> r := PBFloat run: 100.
>>> r asString
>>>
>>> which will give decent results back :). This way it is much easier to make sense out of the numbers.
>>>
>>> So here again to remember:
>>>
>>> - number of samples
>>> - average run times
>>> - standard deviation
>>>
>>> If one of these results is missing the benchmark results are incomplete.
>>>
>>> best regards,
>>> camillo
>>>
>>>
>>>
>>> On 2011-04-05, at 13:56, Igor Stasenko wrote:
>>>
>>>> VariableNode initialize.
>>>> Compiler recompileAll.
>>>>
>>>> [
>>>> TestCase allSubclasses do: [ :cls|
>>>>        cls isAbstract
>>>>                ifFalse: [cls suite run]].
>>>> ] timeToRun
>>>>
>>>> 178938
>>>> 183963
>>>>
>>>>
>>>>
>>>> (ParseNode classVarNamed:  'StdSelectors') removeKey: #class ifAbsent: [].
>>>> Compiler recompileAll.
>>>>
>>>> [
>>>> TestCase allSubclasses do: [ :cls|
>>>>        cls isAbstract
>>>>                ifFalse: [cls suite run]].
>>>> ] timeToRun
>>>>
>>>> 187168
>>>> 184992
>>>>
>>>> the deviation is too big to see if its really so big overhead.
>>>>
>>>> if you compare worst , you got  187/178 ~ 5%
>>>> and if you compare the best you got
>>>> 184/183  ~ 0.5%
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Igor Stasenko AKA sig.
>>>>
>>>
>>>
>>>
>>
>
>
>



More information about the Pharo-project mailing list