aeronet Benchmarks

This document describes the purpose, scope, and usage of the benchmarking suite for aeronet.

Goals

• Track performance regressions for core HTTP and WebSocket request handling.
• Provide comparative harnesses against other C++ HTTP frameworks (planned: oatpp, drogon).
• Keep CI benchmark runs reproducible with pinned tooling and generated artifacts.
• Offer reproducible local runs with JSON + HTML output for ad-hoc analysis.

Non‑Goals (Current Phase)

• Producing authoritative cross-platform numbers (cloud CI noise is high).
• Shipping benchmark binaries in packages.
• Providing full-feature client load generation (wrk/vegeta do that better externally).

CI and GitHub Pages

Benchmark CI is executed by .github/workflows/benchmarks-gh-pages.yml.

It runs:

• HTTP/1.1 benchmarks (wrk)
• HTTP/2 benchmarks (h2load, h2c + h2-tls)
• WebSocket benchmarks (k6)

The same workflow publishes rendered dashboards and badge endpoint JSON files to GitHub Pages:

• HTTP/1.1: https://sjanel.github.io/aeronet/benchmarks/
• HTTP/2 h2c: https://sjanel.github.io/aeronet/benchmarks/h2/benchmarks_h2c.html
• HTTP/2 TLS: https://sjanel.github.io/aeronet/benchmarks/h2/benchmarks_h2-tls.html
• WebSocket: https://sjanel.github.io/aeronet/benchmarks/ws/

For WebSocket runs, if aeronet reports benchmark check failures/errors, the benchmark CI job fails by design.

WebSocket Benchmarks

The WebSocket scripted benchmark harness lives under benchmarks/scripted-servers/ and is driven by:

• run_ws_benchmarks.py (orchestration)
• k6 scenario scripts under benchmarks/scripted-servers/k6/
• render_ws_benchmarks_html.py (dashboard generation)

Run locally from the benchmark build directory:

./run_ws_benchmarks.py --duration 30s --vus 50 --output ./ws-results

Artifacts produced:

• Timestamped JSON summary: ws_benchmark_*.json
• Stable JSON pointer for CI/artifacts: ws_benchmark_latest.json
• HTML report: ws_benchmark_*.html
• Shields badge payload: ws_benchmark_badge.json

Build Activation

By default benchmarks build only when aeronet is the main project (top‑level) OR when you explicitly enable them:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DAERONET_BUILD_BENCHMARKS=ON
cmake --build build --target run-aeronet-bench

In CI (environment variable CI defined) they are skipped unless you override:

-DAERONET_BENCH_FORCE_CI=ON

Targets

Target	Description
aeronet-bench-internal	Google Benchmark based micro / roundtrip benchmarks (contains BENCHMARK_MAIN).
aeronet-bench-throughput	Simple standalone executable printing JSON-ish timing for a trivial loopback workload.
run-aeronet-bench	Convenience target: runs internal benchmarks (aggregates only).
run-aeronet-bench-json	Emits Google Benchmark JSON (aeronet-benchmarks.json).
run-aeronet-bench-throughput	Runs the throughput skeleton.
aeronet-bench-frameworks	Comparative simple GET size= handler (aeronet + optional drogon/oatpp).
run-aeronet-bench-frameworks	Runs comparative benchmark with default args.

JSON Output

Generate structured benchmark data:

cmake --build build --target run-aeronet-bench-json
cat build/aeronet-benchmarks.json

Google Benchmark natively supports JSON; we simply redirect stdout.

Comparative Frameworks

An initial comparative benchmark aeronet-bench-frameworks is available. It spins up:

• aeronet (always) – /data?size=N returning an iota-generated std::string of length N.
• Drogon (if -DAERONET_BENCH_ENABLE_DROGON=ON) – identical endpoint.
• Oatpp (if -DAERONET_BENCH_ENABLE_OATPP=ON) – identical endpoint.

Each iteration selects a random payload size in [min,max] and performs a blocking HTTP/1.1 request over loopback. Metrics collected:

• Total wall time
• Requests per second
• Aggregate bytes and MB/s (body bytes only)

Activate extra frameworks:

cmake -S . -B build_bench -DCMAKE_BUILD_TYPE=Release -DAERONET_BUILD_BENCHMARKS=ON \
  -DAERONET_BENCH_ENABLE_DROGON=ON -DAERONET_BENCH_ENABLE_OATPP=ON
cmake --build build_bench --target aeronet-bench-frameworks
./build_bench/aeronet-bench-frameworks --iters=5000 --min=64 --max=8192

Planned enhancements:

• Percentile latency (collect micro timings per request)
• Concurrent client connections
• Streaming & TLS variants (requires AERONET_ENABLE_OPENSSL=ON)

Planned Roadmap

Phase	Item	Status
A	Initial harness + minimal roundtrip bench	DONE
A	Throughput skeleton	DONE
B	Parse-only microbench (expose internal parser)	TODO
B	Multi-threaded in-process client loop	TODO
C	Simulated degraded network (latency, bandwidth)	TODO
C	Streaming benchmarks	TODO
D	Comparative: oatpp / drogon basic handler	PARTIAL (basic size endpoint & driver)
D	JSON consolidation & history (append-only .jsonl)	TODO
E	TLS handshake & request benchmarks	TODO

Guidelines for Adding a Benchmark

1. Add source under benchmarks/internal/ (micro) or benchmarks/e2e/ (macro / throughput).
2. Append the file to AERONET_BENCH_INTERNAL_SOURCES (for microbench) or create a new executable.
3. Prefer small, isolated scopes - avoid mixing multiple subsystems unless explicitly measuring end-to-end.
4. Keep runtime bounded: default iterations should finish in < 2 seconds on a typical dev laptop.
5. Use benchmark::DoNotOptimize(value) to prevent undesired optimization.
6. Use counters (state.counters["name"]) for derived metrics where helpful.

Caveats

• Loopback measurements elide network variability; real deployment performance can differ.
• Single-thread server design: multi-core scaling requires multiple processes/instances; benchmarks will eventually include multi-instance harnesses.
• Comparative numbers should always record compiler, flags, CPU model, and temperature (when publishing externally).

Example Quick Run

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DAERONET_BUILD_BENCHMARKS=ON
cmake --build build --target run-aeronet-bench
./build/aeronet-bench-throughput

Future Ideas

• Automatic detection of regression (simple % threshold) when explicitly requested.

Profiling with perf

scripts/profile_benchmark.sh records a command or an existing PID, exports perf.script, and generates flamegraph.svg when Brendan Gregg's FlameGraph scripts are installed. It uses DWARF call graphs by default, which work with optimized binaries even when frame pointers are unavailable.

Build benchmarks as RelWithDebInfo with frame pointers:

./scripts/profile_benchmark.sh --build

Record any finite benchmark command:

./scripts/profile_benchmark.sh --freq 400 -- ./build-profile/aeronet-bench-internal-router

Attach to an already-running server and open the result in Hotspot:

./scripts/profile_benchmark.sh --pid "$(pidof aeronet-bench-server)" --hotspot

Hotspot is discovered on PATH, under ~/Downloads/hotspot-*.AppImage, or under ~/Applications/hotspot-*.AppImage. Use --hotspot-bin FILE to override it.

Scripted server and client profiles

The scripted runners can profile the measured process directly. Server runs attach after warmup and record the server while wrk or h2load drives it. Client runs record the selected finite client driver, including its in-process warmup, and do not record the shared server.

# One server workload, with an SVG flamegraph and Hotspot view.
python3 benchmarks/scripted-servers/run_benchmarks.py \
  --server aeronet --scenario headers --duration 15s \
  --profile --profile-install-flamegraph --profile-hotspot

# One client workload.
python3 benchmarks/scripted-clients/run_client_benchmarks.py \
  --client aeronet --scenario small-get --duration 15s \
  --profile --profile-install-flamegraph --profile-hotspot

Profiling artifacts are stored below the runner's output directory:

results/profiles/<run>/<protocol>/<server-or-client>/<scenario>/
  perf.data
  perf.script
  flamegraph.svg

With --repeat N, server profiles get an additional sample-N/ directory so each measured run remains separate. Avoid --profile-hotspot with broad multi-scenario runs because it opens one GUI window per profile.

Permissions and tools

Install perf using the package matching the running kernel. On Ubuntu or Debian:

sudo apt install linux-tools-$(uname -r) linux-tools-common

The scripted runners cannot prompt for sudo after starting child processes. Allow per-process perf recording before running them:

sudo sysctl kernel.perf_event_paranoid=1

For a single direct command, profile_benchmark.sh --sudo -- ... is also supported. The runner preflight and the helper both report the current kernel.perf_event_paranoid value when access is denied.

Pass --profile-install-flamegraph in a scripted run, or --install-flamegraph to the helper, to clone FlameGraph into ${XDG_CACHE_HOME:-~/.cache}/aeronet/FlameGraph. Existing recordings can be processed again without rerunning the workload:

./scripts/profile_benchmark.sh --input path/to/perf.data --install-flamegraph --hotspot

The helper also accepts --flamegraph-dir DIR, --call-graph dwarf|fp|lbr, --event EVENT, and --output-dir DIR; run it with --help for the complete interface.

HTTP/2 Performance Testing Plan with h2load

Purpose: Establish performance baseline and regression detection for HTTP/2

Overview

This document outlines HTTP/2 performance testing scenarios using h2load, a benchmark tool specifically designed for HTTP/2 load testing.

h2load Test Scenarios

Scenario 1: Baseline RPC-Style (Default)

Test: Simple JSON request/response with stream multiplexing

h2load -n 100000 -c 100 -m 100 \
  -H 'Content-Type: application/json' \
  https://localhost:8443/api/users/1

• -n 100000 : Total requests
• -c 100 : Concurrent connections
• -m 100 : Max streams per connection (multiplexing)

Metrics to Track:

• Requests per second (req/s)
• Average latency (ms)
• Min/Max latency
• Memory usage per connection
• CPU utilization

Baseline Goal: Establish baseline metrics for future regression detection

---

Scenario 2: Large Body Streaming

Test: Large payload response (e.g., 10 MiB download)

h2load -n 1000 -c 50 -m 50 \
  -H 'Accept-Encoding: gzip,zstd' \
  https://localhost:8443/files/large.bin

Metrics to Track:

• Throughput (bytes/sec)
• Latency under streaming
• Compression efficiency
• Memory allocation patterns during streaming

Expected: Zero-copy sendfile should be observable via strace/perf

---

Scenario 3: Compression Efficiency

Test: Compression negotiation and throughput

h2load -n 10000 -c 100 -m 100 \
  -H 'Accept-Encoding: zstd' \
  https://localhost:8443/api/small-payload

Metrics to Track:

• Compression ratio (bytes sent / original)
• CPU overhead of compression
• Throughput (req/s) with/without compression
• Wall-clock time

Expected: zstd to have best compression/speed tradeoff

---

Scenario 4: High Stream Multiplexing

Test: Maximum concurrent streams stress

h2load -n 50000 -c 10 -m 500 \
  https://localhost:8443/api/echo

• -m 500 : Max concurrent streams (high multiplexing)
• -c 10 : Fewer connections (more multiplexing per conn)

Metrics to Track:

• Requests per second
• Stream state management overhead
• Flow control window utilization
• HPACK decoder efficiency

Expected: Near-linear performance scaling with streams (no bottlenecks)

---

Scenario 5: Small Payload Latency

Test: Ultra-low-latency requests (e.g., health checks, metrics)

h2load -n 100000 -c 100 -m 100 \
  https://localhost:8443/health

Metrics to Track:

• Tail latencies (p50, p95, p99)
• Min latency (event loop responsiveness)
• GC/allocation pauses

Expected: Sub-millisecond latencies (< 1ms p99)

---

Scenario 6: TLS Handshake Overhead

Test: Connection establishment with TLS

# Warm handshakes (reused connections)
h2load -n 10000 -c 100 -m 100 \
  https://localhost:8443/api/echo

# Cold handshakes (new connections every N requests)
h2load -n 10000 -c 100 -m 1 \
  https://localhost:8443/api/echo

Metrics to Track:

• Handshake time (ms)
• Time to first byte (TTFB) with/without handshake
• TLS 1.3 0-RTT impact (if enabled)
• Session ticket resumption benefit

Expected: <50ms handshake, <10ms TTFB for subsequent requests

---

Scenario 7: Header Compression (HPACK)

Test: Header compression efficiency

# Small headers (default)
h2load -n 10000 -c 100 -m 100 \
  https://localhost:8443/api/echo

# Large headers (many custom headers)
h2load -n 10000 -c 100 -m 100 \
  -H 'X-Custom-1: value-1' \
  -H 'X-Custom-2: value-2' \
  ... (repeat 50+ custom headers) \
  https://localhost:8443/api/echo

Metrics to Track:

• Bytes per request (with large headers)
• Compression ratio (static + dynamic table)
• Dynamic table eviction rate
• HPACK encoder/decoder CPU time

Expected: >60% header compression ratio

---

Scenario 8: Flow Control Behavior

Test: Slow client reading (flow control window exhaustion)

# Simulate slow reading: --rate-period=1ms (one read per millisecond)
h2load -n 1000 -c 10 -m 10 \
  --rate=1000 \
  https://localhost:8443/files/medium-file

Metrics to Track:

• Throughput under flow control constraints
• WINDOW_UPDATE frame frequency
• Buffer memory usage
• Stream stall time

Expected: Proper backpressure (no excessive buffering)

Performance Regression Detection (ideas)

Baseline Metrics

To be established in first run:

{
  "scenario": "baseline_rpc",
  "date": "2026-01-17",
  "metrics": {
    "req_per_sec": 45000,  // Expected value
    "avg_latency_ms": 2.3,
    "p99_latency_ms": 5.1,
    "memory_per_conn_kb": 128
  }
}

Regression Threshold

Fail build if:

• req/s < baseline * 0.95 (5% regression)
• p99_latency_ms > baseline * 1.1 (10% increase)
• memory_per_conn > baseline * 1.2 (20% increase)

---

Profiling & Diagnostics

When performance regressions are detected:

1. CPU Profiling with perf

perf record -F 99 -p <server_pid> -- h2load ... 
perf report

2. Flame Graph Generation

perf script | inferno-collapse-perf | inferno-flamegraph > /tmp/perf.svg

3. Memory Profiling with Valgrind/Massif

valgrind --tool=massif --massif-out-file=massif.out <aeronet_server>
ms_print massif.out

4. strace for Syscall Overhead

strace -e trace=sendfile,writev,write -c <aeronet_server>