This webpage contains extra material for the Robust Java benchmarking series of articles (Part 1: Issues and Part 2: Statistics and solutions) which were published at IBM developerWorks.
In late 2009, I was doing some benchmarks for a client,
and got strange results that seemed to depend on the order in which various benchmarks were executed.
At the time, I was doing multiple calls to my bb.util.Benchmark class from within the same JVM session.
The tasks being benchmarked were not totally independent, but were differently parameterized versions of some common functionality.
I ended up concluding that hotspot was optimizing for the first execution profile being benchmarked,
but that profile was not optimal for subsequent tasks.
There appears to be no way that you can programmatically tell the JVM to throw away all its profiling and optimizations.
So, the only way to do pristine benchmarks is to do but a single benchmark per JVM session.
As an example of how to do but a single benchmark per JVM session, I have revised the latest version of my article project so that the script files now generate all the scenarios instead of the Java code.
Augmenting the original text is some highly specialized documentation that I have placed into an article supplement. These formats are available:
If anyone knows of a free open source way to convert Microsoft Word 2007 documents to an open format which would preserve the equations in MathML as well as leave hyperlinks working, please email me. I am aware of Microsoft's pdf convertor add-on, but that requires a valid copy of MS Office which I no longer have. I am also aware that OpenOffice 3 can allegedly import from/export to MS Office 2007, however, when I tried to open my docx file with Open Office 3.1 on 2009-05-18, it garbled all of the equations.
The following may be downloaded:
| benchmarking framework (just jars): |
bb.jar, jsci-core.jar, mt-13.jar (download all these individual jar files) Last modified: 2010-06-01 Change to Benchmark.java: the environmental noise is no longer measured by default,
altho this can be reactivated (see the Benchmark.Params.estimateNoiseFloor field).There have also been many changes to other classes in bb.jar. The jar files above are all that you need if you merely want to use the bb.util.Benchmark class.
|
|
|---|---|---|
| benchmarking framework (complete project): |
projectLibrary.zip Last modified: 2010-06-01 (Same changes described above.) Download this if you want my complete code library project, including source code for the bb.util.Benchmark class.
|
projectLibrary_asPublished.zip Last modified: 2008-07-23 This is a version from around the time when the article was published. |
| article code listings: |
projectArticle.zip Last modified: 2009-05-07 Noteworthy changes:
Notes:
|
projectArticle_asPublished.zip Last modified: 2008-07-06 This is a version from around when the article was published. Warning: this project is not self-contained: to run it, must first download the benchmarking framework |
Note: all the text files should be proper text files (that is, use unix '\n' line ends). So, if you are a Windows user, you will need a decent text editor to read them (NotePad/WordPad may not cut it; I highly recommend TextPad)