|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectbb.util.Benchmark
public class Benchmark
Benchmarks the performance of a "task",
defined here as arbitrary Java code that is contained inside either a Callable
or Runnable
.
The task may be anything from a microbenchmark to a full blown application.
This class implements all of the principles described in the two-part article
Robust Java benchmarking, Part 1: Issues
and Robust Java benchmarking, Part 2: Statistics And solutions.
A related design goal was to automate as many aspects as possible, and minimize input from and interaction with the user. For example, this class tries to detect as many issues as possible (e.g. it checks if the measurements have outliers).
System.out.println("My task: " + new Benchmark(task));
where task is either a Callable
or Runnable
created by the user to encapsulate the code to be measured.
This single argument Benchmark
constructor will carry out multiple execution time measurements
and perform statistics calculations on them, so that upon return the new instance is populated with results.
This code also makes an implicit call to toString
which will describe the essential statistics.
If custom tuning of the measurement parameters is required, then more complicated constructors may be used.
One simple tuning is if the task internally executes the same identical action many times,
and you want statistics for that individual action (as opposed to the task as a whole).
This is easily done with code like this:
System.out.println("My task: " + new Benchmark(task, n));
where n
is the number of actions that you know task performs.
See Benchmark.Params.numberActions
for more discussion.
Another simple tuning is if the task should only be executed once.
This is easily done with code like this:
System.out.println("My task: " + new Benchmark(task, false));
There will be no statistics reported in this case, just the single time measurement.
See Benchmark.Params.manyExecutions
for more discussion.
For uncommon situations, ultimate tunability over every measurement parameter
may be achieved by constructing an instance of the Benchmark.Params
inner class,
modifying it as desired,
and then directly supplying it along with task to the appropriate constructor.
For example, here is how to configure Benchmark
to measure CPU time:
Benchmark.Params params = new Benchmark.Params();
params.setMeasureCpuTime(true); // default is false (i.e. measure elapsed time)
System.out.println("Some task's CPU time: " + new Benchmark(task, params));
Benchmark
's toString
method was implicitly invoked to get a summary result report.
However, if very detailed results are needed, then toStringFull
may be used instead.
Furthermore, various accessors are available if you need customized results.
Note that all the toStringXXX
methods of this class, as well as its inner classes,
have the property that regardless of whether or not the result contains multiple lines,
the final char is never a newline.
This means that the user should always call println
on the results
if a PrintWriter
is being used for output.
Another important concern is the accuracy of those estimates, since inaccurate results, obviously, limit the conclusions that can be drawn (e.g. that one task executes faster than another). This class determines the accuracy of its estimates by producing confidence intervals for both the mean and the sd.
Execute
.
Nested Class Summary | |
---|---|
protected static class |
Benchmark.JvmState
Records some characteristics of the JVM state. |
static class |
Benchmark.LfsrAdvancer
The run method of this class simply advances the internal state of an Lfsr instance. |
protected static class |
Benchmark.Measurement
Records information about an execution time measurement. |
static class |
Benchmark.Params
Holds parameters which specify how the benchmarking is carried out. |
static class |
Benchmark.Stats
Holds execution time statistics. |
static class |
Benchmark.UnitTest
See the Overview page of the project's javadocs for a general description of this unit test class. |
Field Summary | |
---|---|
protected Object |
callResult
If task is a Callable , then records the last result of executing task.call() . |
protected String |
cleanIssues
Records any issues that were found while doing a final JVM clean after all measurements were carried out. |
protected Benchmark.Measurement[] |
measurements
If params . |
protected static Map<Benchmark.Params,Benchmark.Stats> |
noiseMap
Maps a Params instance
to the block statistics
that result from benchmarking noiseTask . |
protected static Runnable |
noiseTask
Task used to estimate the execution time noise floor. |
protected long |
numberExecutions
Records how many executions of task were performed for each time measurement. |
protected String |
outlierIssues
Records any outlier issues that were found in the measurements. |
protected Benchmark.Params |
params
Controls how this instance's benchmarking is done. |
protected String |
serialCorrelationIssues
Records any serial correlation issues that were found in the measurements. |
protected Benchmark.Stats |
statsAction
If params . |
protected Benchmark.Stats |
statsBlock
If params . |
protected Object |
task
Packages the target code that will be benchmarked. |
protected double |
timeExecFirst
First execution time for task . |
Constructor Summary | |
---|---|
protected |
Benchmark()
No-arg constructor. |
|
Benchmark(Callable<T> task)
Convenience constructor that simply calls . |
|
Benchmark(Callable<T> task,
Benchmark.Params params)
Constructor that measures the execution time of a Callable task. |
|
Benchmark(Callable<T> task,
boolean manyExecutions)
Convenience constructor that simply calls . |
|
Benchmark(Callable<T> task,
boolean manyExecutions,
long numberActions)
Convenience constructor that simply calls . |
|
Benchmark(Callable<T> task,
long numberActions)
Convenience constructor that simply calls . |
|
Benchmark(Runnable task)
Convenience constructor that simply calls . |
|
Benchmark(Runnable task,
Benchmark.Params params)
Identical to the Callable constructor except that task is a Runnable . |
|
Benchmark(Runnable task,
boolean manyExecutions)
Convenience constructor that simply calls . |
|
Benchmark(Runnable task,
boolean manyExecutions,
long numberActions)
Convenience constructor that simply calls . |
|
Benchmark(Runnable task,
long numberActions)
Convenience constructor that simply calls . |
Method Summary | |
---|---|
protected void |
appendIssues(String issues,
StringBuilder sb)
Helper method for toStringFull which handles issues fields like cleanIssues . |
protected void |
calculateStats()
Calculates statsBlock from measurements . |
protected void |
checkState()
Checks that all the state of this instance satisfies its contracts. |
protected void |
cleanJvm()
Attempts to restore the JVM to a pristine state by aggressively performing object finalization and garbage collection. |
protected void |
cleanJvmFinal()
Measures how long it takes to execute a call to cleanJvm . |
protected void |
clearUserMsgs()
Clears any output that may have been written by calls to sendUserMsg . |
protected void |
determineNumberExecutions()
Determines how many executions of task are required in order for the sum
of their execution times to equal params . |
protected void |
diagnoseOutliers()
Diagnoses measurements
for the presence of outliers. |
protected String |
diagnoseSd(double mean,
double sd)
Diagnoses the sd parameter
to see if it really represents the intrinsic variation in task 's execution time
or if it is just reflecting environmental noise. |
protected void |
diagnoseSerialCorrelation()
Diagnoses measurements for the presence of serial correlation. |
protected void |
doMeasurementFirst()
Performs the first measurement of task 's execution time. |
protected void |
doMeasurements()
Measures many many executions of task . |
Object |
getCallResult()
Returns callResult . |
double |
getFirst()
Returns the action execution time as determined from the first measurement, namely,
|
double |
getMean()
Returns the mean of the action execution times. |
long |
getNumberActionsPerMeasurement()
Returns the number of action executions in each measurement. |
double |
getSd()
Returns the standard deviation of the action execution times. |
Benchmark.Stats |
getStats()
Alias for getStatsAction . |
Benchmark.Stats |
getStatsAction()
Returns statsAction . |
Benchmark.Stats |
getStatsBlock()
Returns statsBlock . |
protected double[] |
getTimes()
Returns an array of just the execution times stored in measurements . |
protected String |
issueSummary()
Returns a summary of any issues that occured. |
protected Benchmark.Measurement |
measure(long n)
Measures the execution time of n calls of task . |
protected void |
osSpecificPreparation()
Calls operating system specific commands to prepare for benchmarking. |
protected void |
perform(Object task,
Benchmark.Params params)
Carries out the benchmarking of task
according to the specifications found in params . |
protected void |
preventDce()
Attempts to prevent a sophisticated compiler from using dead-code elimination to skip computations that should be benchmarked. |
protected void |
sendUserMsg(String msg)
Displays a non-critical message to the user. |
protected double |
timeDiffSeconds(long t1,
long t2)
Robustly calculates and returns the difference t2 - t1 in seconds. |
protected long |
timeNs()
Returns the time since some fixed but arbitrary offset, in nanoseconds. |
String |
toString()
Returns a String representation of this instance. |
String |
toStringFull()
Returns a String representation of this instance with full details and explanations. |
protected void |
warmupJvm()
In order to give hotspot optimization a chance to complete, task is executed many times (with no recording of the execution time). |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait |
Field Detail |
---|
protected static Runnable noiseTask
Must have minimal "true" execution time variation. Instead, should only be subject to external noise (e.g. from operating system context switches and/or time measurement errors). So, in a perfect environment, this task should always have the same execution time (after some JVM warmup phase). Also, desire the task's steady state execution time to be in a reasonable range, say between 1 ms and 1 s.
Perhaps one way to write a low noise task is if it satisfies these properties:
It is probably best if task repeatedly performs a very simple but irreducable computation. This computation should not give the JIT compiler many oportunities for optimization. It also must guarantee to always be performed (i.e. no dead-code elimination).
Contract: must satisfy the contract of task
.
Additionally, to meet the no garbage collection condition,
is restricted to being a Runnable
whose run method should create no new objects.
protected static Map<Benchmark.Params,Benchmark.Stats> noiseMap
Params
instance
to the block statistics
that result from benchmarking noiseTask
.
In particular, if params
is a key of this Map
,
then new Benchmark(noiseTask, params).statsBlock
is the value.
Contract: is never null
.
protected Object task
Contract: is never null
and is always either a Callable
or Runnable
instance.
To ensure that the computation that it carries out is never be subject to
dead-code elimination (DCE),
the following rules must be obeyed:
Callable
, then the computation done by call must be used to generate the result returned by call
Runnable
, then the computation done by run
must be stored in some internal state.
The toString
method must be reimplemented, overriding Object
's, and must use that state in its result.
preventDce
should guarantee that DCE never occurs.
protected Benchmark.Params params
Contract: is never null
.
protected double timeExecFirst
task
.
In general, this value includes all initialization effects (e.g. class loading, hotspot warmup, etc).
Warning: the value is actually just the first measurement of task that was done by this Benchmark
instance.
So, if other code in this same JVM session has previously executed task,
then this value will not, in fact, reflect initialization effects.
Units: seconds.
Contract: is >= 0, and is never NaN or infinite.
protected long numberExecutions
task
were performed for each time measurement.
Contract: is > 0.
protected Benchmark.Measurement[] measurements
params
.getManyExecutions
returns true
,
then this instance performed multiple execution time measurements and stored the results here.
The values are only those from the steady state execution profile stage, and do not include the JVM warmup runs.
Each measurement is over numberExecutions
of task
,
and is stored in this array in the order that it occurred (i.e. this array is not sorted).
Contract: is null
if params.getManyExecutions()
returns false
, else is non-null
with length = params.
.
Benchmark.Params.getNumberMeasurements()
protected Object callResult
task
is a Callable
, then records the last result of executing task.call()
.
This is used to prevent dead-code elimination; see task
and preventDce
.
Contract: is always null
if task
is a Runnable
; otherwise, may be anything.
protected String cleanIssues
Contract: is either null
if there are no issues, else is non blank.
protected String outlierIssues
Contract: is either null
if there are no issues, else is non blank.
protected String serialCorrelationIssues
Contract: is either null
if there are no issues, else is non blank.
protected Benchmark.Stats statsBlock
params
.Benchmark.Params.getManyExecutions()
returns true,
holds the block statistics.
In this case, what actually is measured is a block of numberExecutions
calls of task.
The statistics of these measurements are stored here.
Contract: is null
if params.getManyExecutions()
returns false
, else is non-null
.
protected Benchmark.Stats statsAction
params
.Benchmark.Params.getManyExecutions()
returns true,
holds the action statistics.
These are always calculated from statsBlock
and are never directly measured
(altho, if there is but 1 action per measurement, then this instance is the same as statsBlock
).
Contract: is null
if params.getManyExecutions()
returns false
, else is non-null
.
Constructor Detail |
---|
public Benchmark(Callable<T> task) throws IllegalArgumentException, IllegalStateException, Exception
this
(task, new Benchmark.Params.Benchmark.Params()
)
.
IllegalArgumentException
IllegalStateException
Exception
public Benchmark(Callable<T> task, boolean manyExecutions) throws IllegalArgumentException, IllegalStateException, Exception
this
(task, new Params
(manyExecutions))
.
IllegalArgumentException
IllegalStateException
Exception
public Benchmark(Callable<T> task, long numberActions) throws IllegalArgumentException, IllegalStateException, Exception
this
(task, new Params
(numberActions))
.
IllegalArgumentException
IllegalStateException
Exception
public Benchmark(Callable<T> task, boolean manyExecutions, long numberActions) throws IllegalArgumentException, IllegalStateException, Exception
this
(task, new Params
(manyExecutions, numberActions))
.
IllegalArgumentException
IllegalStateException
Exception
public Benchmark(Callable<T> task, Benchmark.Params params) throws IllegalArgumentException, IllegalStateException, Exception
perform
(task, params)
.
So, when this constructor finishes, it is fully populated with results.
task
- the Callable whose call
method's execution time will be measured; is assigned to the task
fieldparams
- assigned to the params
field
IllegalArgumentException
- if task == null
; params == null
IllegalStateException
- if some problem is detected
Exception
- (or some subclass) if task.call
throws itpublic Benchmark(Runnable task) throws IllegalArgumentException, IllegalStateException
this
(task, new Benchmark.Params.Benchmark.Params()
)
.
IllegalArgumentException
IllegalStateException
public Benchmark(Runnable task, boolean manyExecutions) throws IllegalArgumentException, IllegalStateException
this
(task, new Params
(manyExecutions))
.
IllegalArgumentException
IllegalStateException
public Benchmark(Runnable task, long numberActions) throws IllegalArgumentException, IllegalStateException
this
(task, new Params
(numberActions))
.
IllegalArgumentException
IllegalStateException
public Benchmark(Runnable task, boolean manyExecutions, long numberActions) throws IllegalArgumentException, IllegalStateException
this
(task, new Params
(manyExecutions, numberActions))
.
IllegalArgumentException
IllegalStateException
public Benchmark(Runnable task, Benchmark.Params params) throws IllegalArgumentException, IllegalStateException, RuntimeException
Callable constructor
except that task
is a Runnable
.
task
- the Runnable
whose run
method's execution time will be measured; is assigned to the task
fieldparams
- assigned to the params
field
IllegalArgumentException
- if task == null
; params == null
IllegalStateException
- if some problem is detected
RuntimeException
- (or some subclass) if some other problem occursprotected Benchmark()
Allows instance creation without causing a benchmark.
For example, subclasses may wish to not have the constructor automatically carry out benchmarking.
Another use is to test instance methods of this class without doing a benchmark first
(e.g. see UnitTest.test_diagnoseSerialCorrelation
).
Method Detail |
---|
protected void perform(Object task, Benchmark.Params params) throws IllegalArgumentException, IllegalStateException, Exception
task
according to the specifications found in params
.
task
- either a Callable
or a Runnable
; packages the target code that will be benchmarked; is assigned to the task
fieldparams
- assigned to the params
field
IllegalArgumentException
- if task == null
;
task
is neither a Callable
or Runnable
;
params == null
IllegalStateException
- if some problem is detected
Exception
- (or some subclass) if task
is a Callable
and task.call
throws itprotected void osSpecificPreparation()
The current implementation only does something if the operating system is determined to be from Microsoft.
In this case, it forces pending idle tasks to execute by running the command
Rundll32.exe advapi32.dll,ProcessIdleTasks
Warning: this may take several minutes to complete, especially if this is the first time that it has been called in a while.
This method never throws an Exception
.
If one occurs, it is caught and its stack trace is printed to System.err
.
protected void doMeasurementFirst() throws Exception
task
's execution time.
The result is recorded in timeExecFirst
.
Exception
- (or some subclass) if task
is a Callable
and task.call
throws itprotected void warmupJvm() throws Exception
task
is executed many times (with no recording of the execution time).
This phase lasts for the time specified by params
.Benchmark.Params.getWarmupTime()
.
Exception
- (or some subclass) if task
is a Callable
and task.call
throws itprotected void determineNumberExecutions() throws Exception
task
are required in order for the sum
of their execution times to equal params
.getExecutionTimeGoal
.
The result is stored in numberExecutions
.
Exception
- (or some subclass) if task
is a Callable
and task.call
throws itprotected void doMeasurements() throws Exception
task
.
Each measurement is over a block of numberExecutions
calls to task
.
The number of measurements is specified by params
.getNumberMeasurements
.
The results are recorded in measurements
.
Exception
- (or some subclass) if task
is a Callable
and task.call
throws itprotected void preventDce()
The implementation here writes data to the console
that always depends on both
and task
.toString()
.
This ensures that as long as task obeys its contract, regardless of whether it is a Runnable or Callable, then bad DCE never occurs.
Actually, to cut down on low level output like this, this method only prints out this information if some conditional logic is passed.
The form of this logic should cause output to almost never appear, however, the JVM does not know this, and so will be forced to do all the computations.
callResult
.toString()
protected void cleanJvmFinal()
cleanJvm
.
Compares this time with total task
execution time held in measurements
,
and issues a warning if this cleanJvm
time is excessive
(since this means that the task
execution times do not properly account for cleanup costs).
protected void diagnoseOutliers() throws IllegalStateException
measurements
for the presence of outliers.
Implementation here uses the interquartile range
to define what constitutes an outlier
(i.e. the boxplot technique)
and assigns a non-null
message to outlierIssues
if any outliers are detected.
IllegalStateException
- if called when measurements == null
protected void diagnoseSerialCorrelation() throws IllegalStateException
measurements
for the presence of serial correlation.
Implementation here computes the autocorrelation function as a function of lag,
counts how many values fall outside their 95% confidence interval,
and assigns a non-null
message to serialCorrelationIssues
if this count exceeds the expected count.
IllegalStateException
- if called when measurements == null
protected void calculateStats() throws IllegalStateException
statsBlock
from measurements
.
Then derives statsAction
from statsBlock
.
IllegalStateException
- if called when measurements == null
protected double[] getTimes() throws IllegalStateException
measurements
.
IllegalStateException
- if called when measurements == null
protected String diagnoseSd(double mean, double sd)
sd
parameter
to see if it really represents the intrinsic variation in task
's execution time
or if it is just reflecting environmental noise.
See the "Standard deviation measurement issues" section of the
article supplement for more details.
mean
- the mean execution time; must be from the block statisticssd
- the standard deviation of the execution time; must be from the block statisticsprotected void checkState() throws IllegalStateException
IllegalStateException
- if any violation is foundprotected void cleanJvm()
protected void sendUserMsg(String msg)
Only use this method for information like program progress. Do not use it for results or error messages.
Implementation here only does something if
params
.getConsoleFeedback
returns true.
If that is the case, it writes msg
to the current console line.
In order to avoid flooding the console with data,
it truncates msg
if necessary in order to fit on a single line,
and overwrites anything that was previously on the current console line.
protected void clearUserMsgs()
sendUserMsg
.
protected Benchmark.Measurement measure(long n) throws IllegalArgumentException, Exception
n
calls of task
.
Units: seconds.
Contract: the result is always >= 0.
IllegalArgumentException
- if n <= 0
Exception
- (or some subclass) if task
is a Callable
and task.call
throws itprotected long timeNs()
params
.getMeasureCpuTime
returns true
,
else elapsed (aka wall clock) time is used.
protected double timeDiffSeconds(long t1, long t2) throws IllegalArgumentException
t2 - t1
in seconds.
t1
- first time, in nanosecondst2
- second time, in nanoseconds
IllegalArgumentException
- if an issue is discoveredpublic double getFirst()
timeExecFirst
/ params
.Benchmark.Params.getNumberActions()
.
public long getNumberActionsPerMeasurement()
params
.getNumberActions()
* numberExecutions
.
public Object getCallResult()
callResult
.
public Benchmark.Stats getStatsBlock() throws IllegalStateException
statsBlock
.
IllegalStateException
- if params.getManyExecutions()
returns false (there are no Stats)public Benchmark.Stats getStatsAction() throws IllegalStateException
statsAction
.
IllegalStateException
- if params.getManyExecutions()
returns false (there are no Stats)public Benchmark.Stats getStats() throws IllegalStateException
getStatsAction
.
This is a convenience method,
since most users will typically only be interested in the action statistics,
therefore this provides an abbreviation.
IllegalStateException
- if params.getManyExecutions()
returns false (there are no Stats)public double getMean() throws IllegalStateException
This is a convenience method that simply returns
.
getStats
().getMean
()
IllegalStateException
- if params.getManyExecutions()
returns false (there are no Stats)public double getSd() throws IllegalStateException
This is a convenience method that simply returns
.
getStats
().getSd
()
Warning: the result may be unreliable.
The toString
and toStringFull
methods will warn if there are issues with it.
See the "Standard deviation warnings" section of the
article supplement for more details.
IllegalStateException
- if params.getManyExecutions()
returns false (there are no Stats)public String toString()
String
representation of this instance.
The implementation here merely summarizes the important information: the action's first execution time, the action's statistics (if available), and high level warnings.
toString
in class Object
protected String issueSummary()
public String toStringFull()
String
representation of this instance with full details and explanations.
protected void appendIssues(String issues, StringBuilder sb)
toStringFull
which handles issues fields like cleanIssues
.
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |