Java JMH benchmarks: consumed results, fixtures and limited measurements

Java 21+. This benchmark requires JMH 1.37 and its annotation processor; the downloadable Maven project declares them.

Choose a question the fixture can answer

This benchmark compares exact integer membership in a HashSet and an ArrayList for declared sizes and lookup keys. Setup constructs both structures outside the measured method. Returning the boolean result keeps the lookup result observable instead of measuring a discarded computation.

The parameters include a present and an absent key under the selected sizes. A missing lookup scans more of the list than an early present lookup, so one carefully chosen hit cannot establish a universal container ranking. Input distribution belongs in the report.

The source is compiled with JMH 1.37 and its annotation processor in the downloadable Maven project. The processor generates runner metadata. A plain Java main with a nanoTime loop does not provide the same runner or isolation.

Separate smoke verification from a useful experiment

The local check uses one short warmup and measurement iteration to verify the packaged runner can run. Those numbers are not published as a performance conclusion. A useful experiment needs longer measurements, more forks, representative inputs, error estimates and stated JVM and hardware settings.

Dead-code elimination, constant folding, allocation and retained state can change what is measured. The thread-scoped fixture declares per-worker ownership. If the real application shares one contended structure, that is a different benchmark and needs a different state scope and operation mix.

Working program

package in.aitrove.bench;
import org.openjdk.jmh.annotations.*;
import java.util.*;
import java.util.concurrent.TimeUnit;
@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class MembershipBenchmark {
    @Param({"64","4096"}) public int size;
    @Param({"63","8192"}) public int key;
    Set<Integer> indexed;List<Integer> scanned;
    @Setup public void prepare(){
        indexed=new HashSet<>();scanned=new ArrayList<>();
        for(int value=0;value<size;value++){indexed.add(value);scanned.add(value);}
    }
    @Benchmark public boolean indexedLookup(){return indexed.contains(key);}
    @Benchmark public boolean scannedLookup(){return scanned.contains(key);}
}

Measurement result

Measured scores depend on the selected parameters, JVM, machine and measurement configuration.

Costs and boundaries

Fixture construction and retained storage are O(n) for n inserted values. A list membership scan is O(n); hash membership has expected constant lookup work under its normal assumptions. Actual nanoseconds and allocation behavior require measurement and are not supplied as universal facts here.

Common Mistakes

• Do not use smoke-run scores as a production ranking.
• Setup work should be included or excluded according to the question, not by accident.
• A returned result still needs an actual JMH-generated runner to run this class.

Read next

Operation choice, Diagnostic observations.

Run the packaged runner

mvn package
java -jar target/benchmarks.jar -wi 3 -i 5 -f 3