bb.util
Class Benchmark.Params

java.lang.Object
  extended by bb.util.Benchmark.Params
All Implemented Interfaces:
Cloneable
Enclosing class:
Benchmark

public static class Benchmark.Params
extends Object
implements Cloneable

Holds parameters which specify how the benchmarking is carried out.


Field Summary
protected  double cleanFractionThreshold
          Maximum fraction of time that can be spent doing a final JVM cleanup after all measurements have been performed before a warning is issued that the measurements may be too small (because they failed to include cleanup costs).
protected  double confidenceLevel
          If manyExecutions is true, then specifies the confidence level to use when calculating the confidence intervals for the statistics.
protected  boolean consoleFeedback
          Specifies whether or not console feedback should be given regarding the benchmarking progress.
protected  boolean estimateNoiseFloor
          Specifies whether or not Benchmark.noiseTask will be run.
protected  double executionTimeGoal
          If manyExecutions is true, then is used to determine how many executions of Benchmark.task to perform for each timing.
protected  boolean manyExecutions
          Specifies whether or not many executions of Benchmark.task should be performed (as opposed to executing it just once).
protected  boolean measureCpuTime
          Specifies whether or not the execution time is measured as CPU time or elapsed ("wall clock") time.
protected  long numberActions
          Records how many actions are performed by each invocation of Benchmark.task.
protected  int numberMeasurements
          If manyExecutions is true, then once Benchmark.numberExecutions has been determined, specifies the number of timing measurements to perform.
protected  double sdFractionThreshold
          Maximum fraction of the standard deviation (sd) present in Benchmark.statsBlock which may be due to effects besides Benchmark.task's intrinsic variation before a warning is issued that the sd is too high.
protected  double warmupTime
          If manyExecutions is true, then specifies the minimum amount of time that should execute Benchmark.task for before start recording measurements.
 
Constructor Summary
Benchmark.Params()
          Creates a new instance with all fields left at their default initializations.
Benchmark.Params(boolean manyExecutions)
          Simply assigns manyExecutions to the corresponding field.
Benchmark.Params(boolean manyExecutions, long numberActions)
          Simply assigns all the parameters to their corresponding fields.
Benchmark.Params(long numberActions)
          Simply assigns numberActions to the corresponding field.
 
Method Summary
 Object clone()
          Contract: returns a new Params instance that contains equivalent data to this instance.
 boolean equals(Object obj)
          Determines equality based on whether or not obj's Class is the same as this instance's Class, and every field of it equals that of this instance.
 double getCleanFractionThreshold()
          Accessor for cleanFractionThreshold.
 double getConfidenceLevel()
          Accessor for confidenceLevel.
 boolean getConsoleFeedback()
          Accessor for consoleFeedback.
 boolean getEstimateNoiseFloor()
          Accessor for estimateNoiseFloor.
 double getExecutionTimeGoal()
          Accessor for executionTimeGoal.
 boolean getManyExecutions()
          Accessor for manyExecutions.
 boolean getMeasureCpuTime()
          Accessor for measureCpuTime.
 long getNumberActions()
          Accessor for numberActions.
 int getNumberMeasurements()
          Accessor for numberMeasurements.
 double getSdFractionThreshold()
          Accessor for sdFractionThreshold.
 double getWarmupTime()
          Accessor for warmupTime.
 int hashCode()
          Returns a value based on all of the non-boolean fields.
 void setCleanFractionThreshold(double cleanFractionThreshold)
          Mutator for cleanFractionThreshold.
 void setConfidenceLevel(double confidenceLevel)
          Mutator for confidenceLevel.
 void setConsoleFeedback(boolean consoleFeedback)
          Mutator for consoleFeedback.
 void setEstimateNoiseFloor(boolean estimateNoiseFloor)
          Mutator for estimateNoiseFloor.
 void setExecutionTimeGoal(double executionTimeGoal)
          Mutator for executionTimeGoal.
 void setManyExecutions(boolean manyExecutions)
          Mutator for manyExecutions.
 void setMeasureCpuTime(boolean measureCpuTime)
          Mutator for measureCpuTime.
 void setNumberActions(long numberActions)
          Mutator for numberActions.
 void setNumberMeasurements(int numberMeasurements)
          Mutator for numberMeasurements.
 void setSdFractionThreshold(double sdFractionThreshold)
          Mutator for sdFractionThreshold.
 void setWarmupTime(double warmupTime)
          Mutator for warmupTime.
 
Methods inherited from class java.lang.Object
finalize, getClass, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

measureCpuTime

protected boolean measureCpuTime
Specifies whether or not the execution time is measured as CPU time or elapsed ("wall clock") time.

Default value is false (i.e. measure elapsed time).


manyExecutions

protected boolean manyExecutions
Specifies whether or not many executions of Benchmark.task should be performed (as opposed to executing it just once).

In general, should be true in order to obtain the most accurate results. However, if either task takes very long to execute, or if it is desired to measure initialization effects like class loading and hotpsot warmup, then should be false.

Default value is true.


warmupTime

protected double warmupTime
If manyExecutions is true, then specifies the minimum amount of time that should execute Benchmark.task for before start recording measurements.

Units: seconds.

Contract: must be > 0, and is never NaN or infinite.

Default value is 10.0 (seconds), which is the minimum time recomended by Cliff Click, the HotSpot architect (see p. 33 of his 2002 JavaOne talk).


executionTimeGoal

protected double executionTimeGoal
If manyExecutions is true, then is used to determine how many executions of Benchmark.task to perform for each timing. Specifically, for each measurement, task must be executed enough times to meet or exceed the amount of time specified here.

This value should be large enough that inaccuracy in timing measurements are small. Since this class uses System.nanoTime for its timings, and this clock does not (across many platforms) reliably have better than a few 10s of ms accuracy, this value should probably be at least 1 second in order to guarantee measurement error under 1%. Only use a smaller value if you know that System.nanoTime on all the platforms that you deploy on has greater accuracy.

Units: seconds.

Contract: must be > 0, and is never NaN or infinite.

Default value is 1 (second).


numberMeasurements

protected int numberMeasurements
If manyExecutions is true, then once Benchmark.numberExecutions has been determined, specifies the number of timing measurements to perform.

Contract: must be > 0.

Default value is 60. The following considerations determined this value:

  1. 15-25 is about the optimum number of samples for an accuracy versus time tradeoff, according to this one-sided confidence interval table. Now, that table is only valid for mean confidence intervals on a Gaussian distribution. So, in order to accurately handle general distributions (which are likely to require more samples, especially skewed ones) and statistics besides the mean (also likely to require more samples) using the bootstrap technique (which is known to have coverage issues, especially with small numbers of samples), the number of samples should be at least 30.
  2. it, along with executionTimeGoal, determines how long all of the measurements will take. As discussed here, you want the total benchmarking process to last at least 1 minute in order to accurately sample garbage collection and object finalization behavior. Given executionTimeGoal's default value of 1 second, that means this value should be at least 60.
  3. need at least 50 samples to do the autocorrelation tests in diagnoseSerialCorrelation
Conclusion: need at least max(30, 60, 50) = 60.


cleanFractionThreshold

protected double cleanFractionThreshold
Maximum fraction of time that can be spent doing a final JVM cleanup after all measurements have been performed before a warning is issued that the measurements may be too small (because they failed to include cleanup costs).

Contract: must be > 0, and is never NaN or infinite.

Default value is 0.01 (i.e. 1%).


confidenceLevel

protected double confidenceLevel
If manyExecutions is true, then specifies the confidence level to use when calculating the confidence intervals for the statistics. The percent confidence level is 100 times this quantity.

Units: none, is a dimensionless number.

Contract: must be inside the open interval (0, 1), and is never NaN or infinite.

Default value is 0.95, the standard value used in statistics.

See Also:
article on confidence intervals

sdFractionThreshold

protected double sdFractionThreshold
Maximum fraction of the standard deviation (sd) present in Benchmark.statsBlock which may be due to effects besides Benchmark.task's intrinsic variation before a warning is issued that the sd is too high.

Contract: must be > 0, and is never NaN or infinite.

Default value is 0.01 (i.e. 1%).


numberActions

protected long numberActions
Records how many actions are performed by each invocation of Benchmark.task. Only the user knows this number, hence it must be specifed by them.

The number of actions in each timing measurement is numberActions * Benchmark.numberExecutions.

The only effect of this value is on the reporting of action execution statistics.

An example of why this field is needed comes from measuring array element access times where the task internally loops once over an array of length n. To get the access time per element, numberActions needs to be set equal to the array length.

Contract: must be > 0.

Default value is 1.


consoleFeedback

protected boolean consoleFeedback
Specifies whether or not console feedback should be given regarding the benchmarking progress.

Default value is true.


estimateNoiseFloor

protected boolean estimateNoiseFloor
Specifies whether or not Benchmark.noiseTask will be run.

Default value is false.

Constructor Detail

Benchmark.Params

public Benchmark.Params()
Creates a new instance with all fields left at their default initializations.


Benchmark.Params

public Benchmark.Params(boolean manyExecutions)
Simply assigns manyExecutions to the corresponding field. All other fields are left at their default initializations.


Benchmark.Params

public Benchmark.Params(long numberActions)
                 throws IllegalArgumentException
Simply assigns numberActions to the corresponding field. All other fields are left at their default initializations.

Throws:
IllegalArgumentException - if numberActions violates the field's contract

Benchmark.Params

public Benchmark.Params(boolean manyExecutions,
                        long numberActions)
                 throws IllegalArgumentException
Simply assigns all the parameters to their corresponding fields. All other fields are left at their default initializations.

Throws:
IllegalArgumentException - if any paramameter violates the field's contract
Method Detail

getMeasureCpuTime

public boolean getMeasureCpuTime()
Accessor for measureCpuTime.


setMeasureCpuTime

public void setMeasureCpuTime(boolean measureCpuTime)
Mutator for measureCpuTime.


getManyExecutions

public boolean getManyExecutions()
Accessor for manyExecutions.


setManyExecutions

public void setManyExecutions(boolean manyExecutions)
Mutator for manyExecutions.


getWarmupTime

public double getWarmupTime()
Accessor for warmupTime.


setWarmupTime

public void setWarmupTime(double warmupTime)
                   throws IllegalArgumentException
Mutator for warmupTime.

Throws:
IllegalArgumentException - if warmupTime violates its contract

getExecutionTimeGoal

public double getExecutionTimeGoal()
Accessor for executionTimeGoal.


setExecutionTimeGoal

public void setExecutionTimeGoal(double executionTimeGoal)
                          throws IllegalArgumentException
Mutator for executionTimeGoal.

Throws:
IllegalArgumentException - if executionTimeGoal violates its contract

getNumberMeasurements

public int getNumberMeasurements()
Accessor for numberMeasurements.


setNumberMeasurements

public void setNumberMeasurements(int numberMeasurements)
                           throws IllegalArgumentException
Mutator for numberMeasurements.

Throws:
IllegalArgumentException - if numberMeasurements violates its contract

getCleanFractionThreshold

public double getCleanFractionThreshold()
Accessor for cleanFractionThreshold.


setCleanFractionThreshold

public void setCleanFractionThreshold(double cleanFractionThreshold)
                               throws IllegalArgumentException
Mutator for cleanFractionThreshold.

Throws:
IllegalArgumentException - if cleanFractionThreshold violates its contract

getConfidenceLevel

public double getConfidenceLevel()
Accessor for confidenceLevel.


setConfidenceLevel

public void setConfidenceLevel(double confidenceLevel)
                        throws IllegalArgumentException
Mutator for confidenceLevel.

Throws:
IllegalArgumentException - if confidenceLevel violates its contract

getSdFractionThreshold

public double getSdFractionThreshold()
Accessor for sdFractionThreshold.


setSdFractionThreshold

public void setSdFractionThreshold(double sdFractionThreshold)
                            throws IllegalArgumentException
Mutator for sdFractionThreshold.

Throws:
IllegalArgumentException - if sdFractionThreshold violates its contract

getNumberActions

public long getNumberActions()
Accessor for numberActions.


setNumberActions

public void setNumberActions(long numberActions)
                      throws IllegalArgumentException
Mutator for numberActions.

Throws:
IllegalArgumentException - if numberActions violates its contract

getConsoleFeedback

public boolean getConsoleFeedback()
Accessor for consoleFeedback.


setConsoleFeedback

public void setConsoleFeedback(boolean consoleFeedback)
Mutator for consoleFeedback.


getEstimateNoiseFloor

public boolean getEstimateNoiseFloor()
Accessor for estimateNoiseFloor.


setEstimateNoiseFloor

public void setEstimateNoiseFloor(boolean estimateNoiseFloor)
Mutator for estimateNoiseFloor.


clone

public Object clone()
Contract: returns a new Params instance that contains equivalent data to this instance. Any object fields are deep copied. Every subclass must support cloning and must preserve this deep copy behavior.

Overrides:
clone in class Object

equals

public boolean equals(Object obj)
Determines equality based on whether or not obj's Class is the same as this instance's Class, and every field of it equals that of this instance.

The choice was made to use the this.getClass() != obj.getClass() idiom instead of the !(obj instanceof Params) idiom in order to allow subclasses to override this method and obey the equals contract. The disadvantage of this idiom is behavior that can surprise users (no polymorphic equivalence is possible).

Overrides:
equals in class Object
See Also:
"the essay stored in the file equalsImplementation.txt", Josh Bloch interview, Josh Bloch email response

hashCode

public int hashCode()
Returns a value based on all of the non-boolean fields.

Overrides:
hashCode in class Object