Skip to main content

​
Performance overview

This page gives you the benchmark framing needed to evaluate NodalMerge performance without overfitting to single-run numbers. Use it as the entry point before diving into attribution or operation-specific benchmark artifacts.

​
What benchmarks should answer

NodalMerge benchmarking should answer:
  1. Is current behavior within acceptable performance budgets?
  2. Did a change cause meaningful regression?
  3. Which layer likely caused the change?
  4. Is the system still safe to promote under expected workload shape?
If a benchmark result cannot answer one of these, refine the benchmark.

​
Benchmark dimensions that matter

At minimum, benchmark matrices should vary:
  • Peer cardinality
  • Operation mix (map/list/blob)
  • Payload sizes
  • Auth/policy posture
  • Runtime host surface
This avoids drawing global conclusions from a single easy scenario.

​
Scenario classes

Maintain at least these scenario classes:
  • Steady-state sync: typical collaborative command mix
  • Auth/policy-enabled: capability and governance overhead posture
  • Blob-sensitive: payload-heavy behavior
  • Replay/validation: deterministic reconstruction cost
  • Lifecycle-sensitive: persistence/GC/topology side effects under load
Each class highlights different bottlenecks.

​
Baseline vs candidate methodology

For credible comparisons:
  1. Keep scenario matrix fixed.
  2. Run baseline and candidate in alternating sequence when possible.
  3. Capture environment metadata (hardware/runtime/OS).
  4. Compare per-dimension deltas, not only aggregate averages.
Alternating runs reduce false positives from thermal/load drift.

​
Reading results safely

When interpreting results:
  • Look at map/list/blob separately
  • Compare p95/p99 and not only mean
  • Validate stability across peer-count tiers
  • Cross-check with runtime attribution signals
A “faster average” can still hide tail regressions that hurt user experience.

​
Suggested decision bands

Use configurable decision bands for release gating:
  • Green: deltas within agreed operation-specific budgets
  • Yellow: investigate and justify before promotion
  • Red: block promotion pending fix or explicit risk acceptance
Keep thresholds explicit per operation family and topology profile.

​
Benchmark artifact hygiene

Treat benchmark outputs as release artifacts:
  • Timestamped result files
  • Baseline/candidate pair tracking
  • Scenario metadata embedded in results
  • Clear provenance of tooling/version
This makes trend analysis and audit discussions straightforward.

​
Common benchmark anti-patterns

  • Comparing non-equivalent environments
  • Using one benchmark flavor to represent all workloads
  • Ignoring auth/policy-enabled scenarios
  • Ignoring blob-heavy cases
  • Treating microbench wins as guaranteed product-level wins

​
What to do after a regression

  1. Confirm reproducibility with rerun.
  2. Identify affected dimension (map/list/blob, peers, host, auth mode).
  3. Check runtime attribution to isolate likely layer.
  4. Decide: optimize, accept with rationale, or revert.
Tie actions to explicit evidence instead of intuition.

​
Latest results

Run date: 2026-07-01. First full re-run since the activesync → nodalmerge rename (the benchmark harness itself already used current naming; the matrix config’s env var keys did not — see caveat below). Environment: AMD Ryzen 9 5900X (12C/24T), 64 GB RAM, Windows 11 Pro. This is a different host than the 2026-05-22 numbers previously recorded for this project (an ASUS ProArt P16 laptop) — do not diff these results against that note’s absolute numbers or deltas; only same-host runs are comparable. Treat this run as a fresh same-host baseline. Scenario: Run-BenchmarkMatrix.ps1 rows baseline-default-auth and auth-enforced-room-lock-tokened (single A/B pair, not alternated). Targets: rust-combined-server, dotnet-host-runtime-alias. Peers: 2/10/20. Command mix: map60/list60/blob20@4k, 3 iterations per cell. (rust-integrated-hosted-server was skipped — it’s optional and requires MONGO_URI.) Aggregate deltas (auth-enforced vs baseline, positive = auth is slower): Auth overhead was not observable in this run — deltas are flat to negative across every dimension, well inside single-A/B-pair noise. This differs from the 2026-05-22 note’s alternated (A-B-A-B) result of roughly +2% to +8% overhead, but the two are not directly comparable (different host, single-pair vs 2x-averaged). Re-run with alternation on this host before concluding overhead has actually changed. Known caveat: blob operations in this harness log direct upload failed; falling back to ws on every call and fall back to the WebSocket blob path. Blob timings above reflect that fallback path, not the direct-upload path — this has been true since at least the 2026-05-22 run and is a harness/environment limitation, not a regression.

​
Realtime WS host baseline (2026-07-22)

Same-host stats snapshot for the realtime WebSocket + persistence path after two host latency fixes landed on the native Rust hosts: TCP_NODELAY enabled on every accepted connection (Kestrel does this by default; without it Nagle plus delayed-ACK adds a small-frame round-trip stall). These are absolute stats, not a baseline/candidate delta — see the methodology caveats below. Environment: AMD Ryzen 9 5900X (12C/24T), 64 GB RAM, Windows 11 Pro. Same host as the 2026-07-01 run above; do not diff against other-host notes. Scenario: run_bench.ps1 (SDK scenario runner). Targets rust-integrated-hosted-server (Mongo-backed dev-server, :7979) and dotnet-host-runtime-alias (Mongo-backed, :8787). Peers: 6. Command mix: map/list/blob at 6/12/30 ops, 1 KB blobs, ws-only transport, 15 iterations per cell, 4 warmup ops, 2 ms op delay. Results (avg / p95 ms, lower is better): Map: List: Blob: Warmup averaged ~46 ms on both hosts. Map and blob are near-parity at 12 ops; list remains .NET-favored across the sweep, and .NET keeps an edge at 30 ops. Caveats:
  • Single run per cell, not alternated A/B. Per this page’s own methodology, treat this as a directional same-host snapshot, not release-grade gating evidence. Thermal/load drift of several ms between runs is expected — repeat runs of the identical build moved the unchanged .NET numbers by 5–8 ms, so sub-~20 ms differences here are within noise.
  • Blob path is the WebSocket fallback. Blob timings reflect the direct upload failed; falling back to ws fallback, not the direct-upload path — the same harness limitation noted in the run above.
  • Target differs from the 2026-07-01 entry. rust-integrated-hosted-server is the Mongo-backed dev-server; the earlier “Latest results” block used rust-combined-server (no Mongo). The two result blocks are not directly comparable.
  • Provenance: build from branch host-latency-nodelay-connect-tax (TCP_NODELAY on all native hosts + per-connection connect-tax removed).