Benchmarks

This page reports two sets of benchmarks: comparisons with other open-source Pauli and Majorana propagation engines, and an internal pytest suite for detecting performance regressions.

Results

All results were obtained on Leonardo (CINECA); the experimental conditions are specified under Methods.

Pauli propagation

The Pauli benchmark simulates the Trotterized time evolution of a two-dimensional transverse-field Ising model and evaluates a ZZ correlator on a fixed pair of qubits. One experiment varies the circuit depth at fixed lattice size; the other varies the lattice size at fixed depth.

Fixed lattice, increasing depth

This experiment uses a 12×12 lattice (144 qubits), 28 Trotter points, dt = 0.05, a coefficient threshold of 1e-6, and no weight cutoff. The final operator contains approximately 33 million terms.

The observables produced by the coefficient-threshold implementations differ by at most 3.8e-4. Qiskit pauli-prop, which truncates by term count instead, differs by 2.8e-3. The cuPauliProp operator resides in GPU memory; consequently, the table reports its peak GPU allocation rather than the host RSS (which we measured at 552 MB).

Increasing lattice size

This experiment holds the circuit depth and truncation threshold fixed while increasing the lattice from 6×6 (36 qubits) to 18×18 (324 qubits). Each model size and implementation runs in a separate subprocess with a 300 s limit.

The 300 s limit excludes QuEra ppvm and Qiskit pauli-prop at all lattice sizes, and excludes PauliPropagation.jl above 10×10. We remark that this is not an implementation limit for the packages but a consequence of the timeout limit.

Majorana propagation

The Majorana benchmark simulates 20 Trotter layers of a one-dimensional Fermi-Hubbard model with 60 spinful sites. Runtime, expectation value, term count, and memory are recorded after each layer. Both implementations use 24 threads.

Caveats

• CPU parallelism differs among implementations. QuEra ppvm and Qiskit's pauli-prop use one core during propagation (0.99 measured active cores), whereas monoprop and PauliPropagation.jl use multiple cores. The ppvm Python bindings expose the serial indexmap variant rather than the Rayon-parallel config::dashmap variant; Qiskit's _accelerate extension has no parallel propagation path. Runtime ratios are therefore wall-clock ratios, not per-core ratios.
• The Julia benchmarks use the fastest documented containers. These are VectorPauliSum with Performance.propagate! (development branch 0.8.0 or later) and VectorMajoranaSum with AcceleratedKernels. Their dictionary-backed alternatives are serial. The vector Pauli path does not merge duplicate strings, so its reported term count is a storage count rather than the number of distinct operator terms.
• The CPU and GPU measurements use different node types. Each scaling curve is obtained on a single node. The cuPauliProp runtime comparison nevertheless includes both hardware and implementation effects.

Methods

The benchmark drivers are maintained in packages/bench-third-party, a standalone uv project with its own lockfile. It is excluded from the repository workspace because it requires CUDA-specific wheels and a narrower Python version range than monoprop.

Hardware

The measurements use exclusive nodes on Leonardo (CINECA). CPU and GPU measurements use different partitions because Leonardo's DCGP nodes do not contain GPUs.

The benchmarks use each engine's best known configuration. monoprop uses all 56 cores of one socket; PauliPropagation.jl uses 28 threads because higher thread counts reduce its throughput.

Each result record includes host, threads, and library_version; the plotting scripts report these fields alongside derived metrics.

How memory is measured

CPU memory denotes peak resident set size (RSS), obtained from the Linux kernel's VmHWM counter. RSS is the physical memory resident for one process; pages shared with other processes are counted in full. For cuPauliProp, whose operator resides on the GPU, working memory instead denotes peak device allocation.

• The measurement is exact. The kernel updates VmHWM whenever RSS increases, including during native code that does not hold the Python GIL.
• The measurement interval is reset for each step. Before resetting VmHWM, the harness runs gc.collect() and malloc_trim for Python, or GC.gc() and malloc_trim for Julia. The resulting value excludes peaks retained from earlier steps.
• All CPU implementations use the same measurement procedure over the timed region.

RSS includes the interpreter and loaded libraries. Comparisons of growth across steps therefore carry less fixed overhead than comparisons of absolute values.

Implementation-specific estimates are also recorded as native_memory: operator_memory_bytes() for monoprop, Base.summarysize for PauliPropagation.jl, and device-pool usage for cuPauliProp. These estimates cover different allocations and are therefore suitable only for comparison within a single implementation.

Reproducing

The Python implementations share one uv environment:

cd packages/bench-third-party
uv sync    # monoprop (editable), qiskit, ppvm, pauli-prop, cuquantum-python-cu12

Julia is required only for the Julia comparisons:

curl -fsSL https://install.julialang.org | sh          # juliaup + latest stable Julia
julia -e 'using Pkg; Pkg.add(["JSON", "ProgressMeter"])'

The benchmark and plotting commands are:

# Pauli, fixed lattice; edit settings.json to change the model
cd pauli_prop
uv run python run_model.py        # CPU engines -> results.json
julia run_model.jl                # optional; merges PauliPropagation.jl in
uv run python run_model.py --backends cupauliprop -o results_gpu.json   # on a GPU node
uv run python plot_results.py

# Pauli, lattice sweep; 300 s limit per (size, engine)
uv run python run_scaling.py --output scaling_cpu.jsonl --timeout 300 \
    --backends monoprop juliapp ppvm qiskit --threads monoprop=56 juliapp=28
uv run python run_scaling.py --output scaling_gpu.jsonl --timeout 300 \
    --backends cupauliprop                                   # on a GPU node

# The first input takes precedence when a backend occurs in both files
uv run python plot_scaling.py scaling_cpu.jsonl scaling_gpu.jsonl
uv run python plot_scaling.py scaling_cpu.jsonl scaling_gpu.jsonl \
    --memory-key working_set_MB
uv run python plot_speedup.py scaling_cpu.jsonl scaling_gpu.jsonl

# Majorana
cd ../majorana_prop
monoprop_NUM_THREADS=24 JULIA_NUM_THREADS=24 bash run_benchmarks.sh

# Two-panel headline figure from both committed result sets
cd .. && uv run python plot_headline.py

CPU and GPU scaling results are stored separately and combined during plotting. If an implementation occurs in more than one input, the first input takes precedence; placing the CPU file first ensures that the monoprop curve contains only CPU-node measurements. plot_scaling.py warns when a curve contains multiple hosts and does not label an axis as host RSS unless all included records use that metric.

All Pauli model parameters are defined in settings.json and are shared by all implementations. Output schemas, field definitions, and implementation-specific limitations are documented in packages/bench-third-party/README.md.

Internal benchmarks

The internal pytest benchmark suite measures the runtime and peak RSS of core monoprop operations. It is separate from the functional test suite and runs with just bench.

Running

Each run requires a label, which identifies its column in results/REPORT.md. This permits direct comparison of serial, threaded, and MPI configurations. Additional arguments are forwarded to pytest.

uv sync --group bench                          # once

just bench serial                              # serial run, column "serial"
just bench-smoke                               # quick sanity check (tiny sizes)
just bench serial --num-modes 64 --bench-rounds 10

monoprop_NUM_THREADS=10 just bench serial-t10  # cap the partition count

uv run --group bench monoprop-bench-report benches/results  # rebuild report, no re-run

The suite contains two benchmark groups:

• bench_random.py measures build_graph, pare, energy, gradient, and inplace on a random problem. Configurable with --gen-length (4), --obs-terms (10000), --num-generators (100), --num-modes (128), --cutoff (6), --seed (0) and --bench-rounds (1); defaults in parentheses.
• bench_models.py measures two fixed models marked slow: a 120-qubit Fermi-Hubbard 29-step Trotter run (test_model[hubbard]) and a 127-qubit kicked-Ising run over 20 layers (test_model[pauli]). Override any config field with --<model>-<field>, e.g. --pauli-num-layers 30; just bench --help lists them all.

Each run writes pytest-benchmark timings to results/time-<label>.json and the remaining metrics to <label>.json. monoprop-bench-report combines all labels in results/REPORT.md.

MPI

For MPI benchmarks, barriers delimit each timed operation; runtime is therefore the makespan across ranks. Memory is the sum of the per-rank peak RSS values. This is an upper bound on the job-wide peak: shared pages are counted once per rank, and rank-local peaks are summed even when they occur at different times.

just bench-build-mpi                           # build once (MPI on)
monoprop_NUM_THREADS=2 just bench-mpi r5t2 5 --map-by slot:PE=2 --bind-to core

Continuous benchmarking

The bench_main.yml workflow benchmarks every commit on main. Bencher records the time series and reports threshold violations. The workflow invokes the same recipes available locally:

just bench-ci ci-linux          # default sizes, more rounds, no slow models
just bench-bmf ci-linux         # the same results as Bencher Metric Format JSON

monoprop-bench-bmf combines both artifacts because each Bencher upload accepts one adapter. The standard python_pytest adapter includes timings but omits memory and operator metrics. Four measures are recorded:

For a fixed seed and problem size, terms is deterministic; any change therefore indicates an algorithmic change. Timing alerts from shared runners are recorded but do not fail the build.

Testbeds

Bencher identifies each series by (branch, testbed, measure). A testbed therefore denotes one fixed machine configuration; changing that configuration would introduce a discontinuity in every series.

The bare-metal workflow provisions a dedicated node for each run. Its lower measurement variance supports tighter timing thresholds and permits execution of the fixed models omitted on the shared runner.

The bench-markers and bench-rounds inputs of the reusable bench.yml workflow select the benchmarks and repetition count. These values should remain fixed after establishing a testbed, because changing them changes the meaning of its time series:

monoprop_BENCH_MARKERS= monoprop_BENCH_ROUNDS=3 just bench-ci ci-bare-metal