Java benchmarking article

This webpage contains extra material for the Robust Java benchmarking series of articles (Part 1: Issues and Part 2: Statistics and solutions) which were published at IBM developerWorks.

Best practices

In late 2009, I was doing some benchmarks for a client, and got strange results that seemed to depend on the order in which various benchmarks were executed. At the time, I was doing multiple calls to my bb.util.Benchmark class from within the same JVM session. The tasks being benchmarked were not totally independent, but were differently parameterized versions of some common functionality. I ended up concluding that hotspot was optimizing for the first execution profile being benchmarked, but that profile was not optimal for subsequent tasks. There appears to be no way that you can programmatically tell the JVM to throw away all its profiling and optimizations. So, the only way to do pristine benchmarks is to do but a single benchmark per JVM session.

As an example of how to do but a single benchmark per JVM session, I have revised the latest version of my article project so that the script files now generate all the scenarios instead of the Java code.

Article supplement

Augmenting the original text is some highly specialized documentation that I have placed into an article supplement. These formats are available:

If anyone knows of a free open source way to convert Microsoft Word 2007 documents to an open format which would preserve the equations in MathML as well as leave hyperlinks working, please email me. I am aware of Microsoft's pdf convertor add-on, but that requires a valid copy of MS Office which I no longer have. I am also aware that OpenOffice 3 can allegedly import from/export to MS Office 2007, however, when I tried to open my docx file with Open Office 3.1 on 2009-05-18, it garbled all of the equations.

Java code

The following may be downloaded:

benchmarking framework (just jars): bb.jar, jsci-core.jar, mt-13.jar (download all these individual jar files)
Last modified: 2010-06-01
Change to the environmental noise is no longer measured by default, altho this can be reactivated (see the Benchmark.Params.estimateNoiseFloor field).
There have also been many changes to other classes in bb.jar.
The jar files above are all that you need if you merely want to use the bb.util.Benchmark class.
benchmarking framework (complete project):
Last modified: 2010-06-01
(Same changes described above.)
Download this if you want my complete code library project, including source code for the bb.util.Benchmark class.
Last modified: 2008-07-23
This is a version from around the time when the article was published.
article code listings:
Last modified: 2009-05-07
Noteworthy changes:
  1. clever code enhancements (e.g. several benchmarking artifacts eliminated), some bug fixes (e.g. the old results for IdentityHashMap and WeakHashMap were wrong), and many new results
  2. new class added since publication which benchmarks iterators
  3. new class added since publication which benchmarks size-varying arrays to estimate CPU cache sizes


  1. this project is now self-contained: all resources (e.g. jar files) needed to run the code listings are now included in this download
  2. if you want to find all the new results since publication of the article, look at the .../results/newPostArticleResults directory; each new result should have its own dedicated subdirectory there with a readMe.txt file
Last modified: 2008-07-06
This is a version from around when the article was published.
Warning: this project is not self-contained: to run it, must first download the benchmarking framework

Note: all the text files should be proper text files (that is, use unix '\n' line ends). So, if you are a Windows user, you will need a decent text editor to read them (NotePad/WordPad may not cut it; I highly recommend TextPad)

Please address comments and suggestions concerning this website to
Copyright © 2013 Elliptic Group. All rights reserved.