[Pharo-project] Remove #class as a Compiler optimization

Camillo Bruni camillo.bruni at inria.fr
Tue Apr 5 15:07:01 CEST 2011


On 2011-04-05, at 15:02, Nicolas Cellier wrote:

> Better than average, take the median

NO! thats why you provide the standard deviation... 

you should not optimize the results prematurely but provide numbers that let you decide on the quality of the benchmarks. what you can do is optimize for systematic errors, like warm up and stuff, but nevertheless you should provide the raw results...

read the paper...

> Nicolas
> 
> 2011/4/5 Camillo Bruni <camillo.bruni at inria.fr>:
>> This is exactly why you have to provide some confidence interval / deviation, otherwise it is hard to make any reasonable conclusion.
>> 
>> run it 100 times and take the average and provide the standard deviation.
>> 
>> I am not a big fan of relying on incomplete benchmarking results:
>> 
>> Please read: http://portal.acm.org/citation.cfm?id=1297033
>> 
>> http://www.squeaksource.com/p.html provides a basic benchmarking framework under the NBenchmark package. You subclass from PBenchmarkSuite implement a method #benchXXX and run it.
>> 
>> r := PBFloat run: 100.
>> r asString
>> 
>> which will give decent results back :). This way it is much easier to make sense out of the numbers.
>> 
>> So here again to remember:
>> 
>> - number of samples
>> - average run times
>> - standard deviation
>> 
>> If one of these results is missing the benchmark results are incomplete.
>> 
>> best regards,
>> camillo
>> 
>> 
>> 
>> On 2011-04-05, at 13:56, Igor Stasenko wrote:
>> 
>>> VariableNode initialize.
>>> Compiler recompileAll.
>>> 
>>> [
>>> TestCase allSubclasses do: [ :cls|
>>>        cls isAbstract
>>>                ifFalse: [cls suite run]].
>>> ] timeToRun
>>> 
>>> 178938
>>> 183963
>>> 
>>> 
>>> 
>>> (ParseNode classVarNamed:  'StdSelectors') removeKey: #class ifAbsent: [].
>>> Compiler recompileAll.
>>> 
>>> [
>>> TestCase allSubclasses do: [ :cls|
>>>        cls isAbstract
>>>                ifFalse: [cls suite run]].
>>> ] timeToRun
>>> 
>>> 187168
>>> 184992
>>> 
>>> the deviation is too big to see if its really so big overhead.
>>> 
>>> if you compare worst , you got  187/178 ~ 5%
>>> and if you compare the best you got
>>> 184/183  ~ 0.5%
>>> 
>>> 
>>> --
>>> Best regards,
>>> Igor Stasenko AKA sig.
>>> 
>> 
>> 
>> 
> 




More information about the Pharo-project mailing list