Docs

Running the rig

The bloating stack is two pieces of software, shipped as the two PRs below. This page is the operator's guide: enable the plugin, build the Go binaries, run the loop, and replay the result on any EL client.

① Nethermind plugin

StateDiffsWriter — writes per-block trie diffs to a BlockDiffs column family.

nethermind#11788 →

② Go orchestrator + sidecar

Closed-loop bloating driver, the RocksDB-secondary tracker, and cross-EL replay.

eth-perf-research#77 →

1 · Nethermind StateDiffsWriter plugin

The plugin is the only Nethermind-side change. On every committed block it walks the trie diff and appends one RLP record to a dedicated BlockDiffs RocksDB column family. It keeps no in-process tracker state — the sidecar reads that CF and does all the accounting.

Configuration

Config section StateDiffsWriter (disabled by default):

Key	Default	Meaning
`Enabled`	`false`	Turn the per-block diff writer on.
`KeepLastNBlocks`	`1000000`	How many recent blocks of diffs to retain in the CF. Lower for tight disk budgets; must exceed the sidecar's worst-case catch-up window.
`PruneIntervalSeconds`	`600`	Background pruner sweep interval. `0` or negative disables pruning.

Enable via any of Nethermind's standard config surfaces:

# configuration JSON
{ "StateDiffsWriter": { "Enabled": true, "KeepLastNBlocks": 1000000 } }

# environment variables
NETHERMIND_STATEDIFFSWRITERCONFIG_ENABLED=true
NETHERMIND_STATEDIFFSWRITERCONFIG_KEEPLASTNBLOCKS=1000000

# CLI
--StateDiffsWriter.Enabled true

BlockDiffs wire format

Each value is a strict-additive RLP list, keyed by 8-byte big-endian block number:

[ blockNumber, postStateRoot, codeHashChanges[], slotCountChanges[],
  accountTrieBytesDelta?, storageTrieBytesDelta?, accountsAddedDelta? ]

Trailing fields can be added without breaking older readers; the sidecar's decoder tolerates the non-canonical long encodings C# RLP emits.

2 · Go orchestrator & sidecar

Two co-located binaries in orchestrator/ of eth-perf-research.

Build

# orchestrator (the bloating driver)
make build                  # -> bin/orchestrator
make linux                  # static linux/amd64

# sidecar (the state-composition tracker)
make build-sidecar-cgo      # full grocksdb; needs librocksdb-dev
make docker-sidecar         # debian-slim image with librocksdb9
make build-sidecar          # stub backend (no CGO) for tests on macOS

make all                    # both

Run the bloater

orchestrator run is the closed-loop controller. It commits blocks into Nethermind via testing_commitBlockV1, packing transaction "verbs" until the live state composition (read from the sidecar) reaches the target in target.yaml.

ORCH_DEPLOY_PRIVATE_KEY=0x... \
bin/orchestrator run \
  --rpc-url        http://nethermind:8545 \
  --state-dir      /var/orch/state \
  --target-yaml    /etc/orch/target.yaml \
  --genesis-sha256 <hex>

Narrow the active verb set with ORCH_VERBS (comma-separated):

ORCH_VERBS=eoatx,storagespam,uniswap_swaps bin/orchestrator run ...

Available verbs: eoatx, calltx, deploytx, factorydeploytx, storagespam, erc20_bloater, erc20tx, uniswap_swaps, gasburnertx, storagerefundtx.

Run the sidecar

One binary, four modes. Production runs --mode=all beside Nethermind, sharing its FlatDb in RocksDB secondary mode:

bin/sidecar \
  --mode=all \
  --db          /mnt/nm/flat \
  --code-db     /mnt/nm/code \
  --snapshot-dir /var/lib/sidecar \
  --spill-dir   /mnt/scratch/spill          # IMPORTANT: the bootstrap slot-spill defaults to the
                                            # container root (/tmp); on big states that ENOSPC-wedges.
                                            # Point it at roomy disk (or set TMPDIR). ~400 GB at 10x.
  --dedup-dir   /mnt/scratch/codehash-dedup   # roomy: ~700 GB at 10x

Mode	Purpose
`bootstrap`	One-shot full FlatDb scan → `snapshot.bin`, exit.
`tail`	Restore last snapshot, apply `BlockDiffs`, re-snapshot periodically.
`serve`	Read-only JSON-RPC over an existing snapshot.
`all`	Bootstrap-if-needed + tail + serve. Production default.

JSON-RPC on :9001:

Method	Returns
`statecomp_lite`	Tier-1 counters + lag — the controller's signal.
`statecomp_get`	Full report: tier-2 byte/depth counters + slot histogram.
`statecomp_health`	Liveness: lag, snapshot age, RSS.

curl -s localhost:9001 -H content-type:application/json \
  --data '{"jsonrpc":"2.0","id":1,"method":"statecomp_get"}' | jq .result

Offline structural scan — `statescan`

orchestrator/cmd/statescan is the exact, offline counterpart to the live sidecar: a structural root→leaf walk of a frozen FlatDb (opened read-only as a RocksDB secondary, never writes the source). It classifies every trie node at its true structural depth (branch+extension edges from the root) and reports the account + storage node-type/depth tables and the per-contract storage count — the FlatDb keys storage by account, so each contract's trie is walked individually (clones sharing a storage root are not deduplicated). This is the source of the #29/#35 state-structure numbers and the trie-depth charts.

# walk a frozen NM FlatDb (just the mainnet/flat RocksDB; TMPDIR must be on roomy disk)
STATESCAN_PARALLEL=12 STATESCAN_BLOCK_CACHE_MIB=8192 \
  statescan /mnt/nm/mainnet/flat
# -> ACCOUNT/STORAGE node-type + depth tables, contracts-with-storage count
# STATESCAN_ACCOUNT_ONLY=1 skips the (long) storage walk

3 · Cross-EL replay

Each run emits replay artifacts so the exact state can be reproduced on any Engine-API client and checked bit-for-bit.

Artifact	What it is
`payloads.rlp`	4-byte-LE length-prefixed `ExecutionPayloadV3` frames — the committed block stream.
`journal.bin`	Protobuf binlog + BLAKE3 chain hash — per-batch controller state, idempotent restart.
`run-manifest.json`	`genesis_sha256`, `chain_identity_hash`, `target_history`, `final_state_root`.
`pending-batch.bin.<id>`	Crash-recovery snapshot of the in-flight batch.

Replay onto another client

# 1. boot the target EL on the matching genesis (verify genesis_sha256), JWT on Engine API

# 2. rebuild a canonical, contiguous, hash-linked payloads file. 'cohere' refetches
#    each canonical block from the source NM (debug_getRawBlock) -- needed because raw
#    run payloads carry zeroed logsBloom and won't replay as-is; pass an empty --in to
#    fetch the whole range. ('heal' is the lighter variant: only fills block-number gaps.)
heal-payloads cohere \
  --in payloads.rlp --out payloads.cohered.rlp \
  --rpc-url http://<source-nm>:8545 --from <firstBlock> --head <lastBlock>

# 3. replay + verify the final state root
bin/orchestrator replay \
  --payloads payloads.cohered.rlp \
  --manifest run-manifest.json \
  --rpc-url  http://<target-el>:8551 \
  --jwt-path jwt.hex

Replay drives engine_newPayloadV4 + engine_forkchoiceUpdatedV3. Outcomes:

ErrReplayInvalid — a payload was rejected (status != VALID).
ErrReplayMismatch — all blocks accepted, but final stateRoot ≠ manifest.
nil — bit-exact replay verified.

Audit a journal

bin/orchestrator verify-journal --path journal.bin
# re-derives the BLAKE3 chain hash, prints final hash + record count

4 · Snap-sync benchmarking

Milestone states are frozen snapshots with no live chain, so a stock node can't snap-sync from them. Runs use an isolated single-peer network (frozen source → target over devp2p) driven by a repeating fake-CL engine_forkchoiceUpdatedV3 loop (~every 12 s) — there is no consensus client.

Nethermind target — a fixed pivot pinned at (or just below) the source head, served by a peer sitting at that pivot, reconstructing stateRoot(H) bit-for-bit (cf. nethermind#11943). The pivot is set via Sync.PivotHash/PivotNumber/PivotTotalDifficulty; the FCU's finalized block becomes the snap pivot, so a tiny head↔finalized split (head=H, finalized=H−16) is required or NM can't resolve the pivot header and never starts the state walk.
Geth target — no client change; it picks its own pivot and heals, driven by the same fake-CL forkchoice loop pinned at the source head.

# source: frozen NM serving snap (no CL). Discovery MUST be on -- NetworkingEnabled
# alone never starts the RLPx server -- and the inbound-IP throttle MUST be off or
# the source rejects the target after a few handshakes (-> 0 peers, persisted in
# peers/SimpleFileDb.db). Serves the pivot within Sync.SnapServingMaxDepth (=128) of head.
nethermind --Init.DiscoveryEnabled=true --Sync.NetworkingEnabled=true \
  --FlatDb.Enabled=true --Pruning.Mode=None \
  --Network.FilterPeersByRecentIp=false --Network.FilterPeersBySameSubnet=false

# NM target
nethermind --Sync.SnapSync=true --Init.DiscoveryEnabled=true \
  --Sync.PivotNumber=<H> --Sync.PivotHash=<hash(H)> --Sync.PivotTotalDifficulty=<td> \
  --Network.StaticPeers=<source-enode> --Network.OnlyStaticPeers=true \
  --Network.FilterPeersByRecentIp=false --Network.FilterPeersBySameSubnet=false

# then drive the snap (fake CL), repeating every ~12s:
#   engine_forkchoiceUpdatedV3({headBlockHash: H, safeBlockHash: H-16, finalizedBlockHash: H-16})

5 · RPC load benchmarking & mock-CL driver

A separate Python framework (state-benchmarks#34) covers the two things the Go stack doesn't: Locust-driven RPC load tests against a milestone node, and a standalone mock consensus layer — the Engine-API driver that the snap-sync and replay runs need on a CL-less network.

Locust load tests (src/load_tests/locustfile.py) — hammer a node with eth_getBalance / eth_getStorageAt / eth_call / eth_getProof and aggregate latency percentiles (src/metrics/aggregator.py).
Mock CL (src/mock_cl/) — a JWT-authed Engine-API client with two drivers: PivotDriver repeatedly forkchoice-updates an EL to a frozen pivot so it snap-syncs (the §4 fake-CL loop), and ReplayDriver feeds recorded ExecutionPayloads through engine_newPayloadV1–V4 + engine_forkchoiceUpdatedV1–V3 and measures per-block latency.

End-to-end

Full design rationale: orchestrator/docs/v19-sidecar-architecture.md in eth-perf-research#77. Both PRs are draft at time of writing; flags and defaults track the branches replace-statecomp-with-statediffswriter and feature/bloating-orchestrator.