System overview
The rig is a closed loop: a Go orchestrator drives a Nethermind node toward a mainnet-shaped target, a lean plugin writes per-block trie diffs to disk, and a separate sidecar consumes those diffs to track state composition and feed the controller back.
The controller compares the live composition to the mainnet-shaped target and scores the bloating “verbs”, picking the transaction mix that closes the gap fastest.
verb mix ← argmax( score | target − current )Orchestrator (Go)
A single static binary. Closed-loop controller scores bloating “verbs” against a target composition, submits transactions through the Engine API, and records every committed block as a replayable payload. Replaces the earlier Python prototype.
StateDiffsWriter plugin
Runs in Nethermind. On every committed block it walks the trie diff and writes a compact RLP
record to a dedicated BlockDiffs RocksDB column family — and nothing else. No in-engine
tracker, no scan on startup. (nethermind#11788)
Sidecar (Go)
Opens the node’s RocksDB in secondary mode, bootstrap-scans once, then tails the
BlockDiffs CF. Maintains the running state composition and serves it over a statecomp_get
JSON-RPC plus a Prometheus endpoint. Restartable independently of the chain.
Why the tracker moved out of the engine
The previous design kept the composition tracker inside Nethermind (the StateComposition
plugin). It worked at mainnet size and held up through ~3.5×, but broke at 5×.
This shape has driven a live 5× bloating run for 16+ hours: ~858 GB state, sensor tracking chain head with 0 blocks behind, and 0 engine rescans across multiple graceful restarts.
Design constraints
Wire format (BlockDiffs column family)
Each value is a strict-additive RLP list keyed by 8-byte big-endian block number:
[ blockNumber, postStateRoot, codeHashChanges[], slotCountChanges[],
accountTrieBytesDelta?, storageTrieBytesDelta?, accountsAddedDelta? ]Trailing fields can be added without breaking older consumers; the sidecar’s length-prefixed
decoder tolerates the non-canonical long encodings C# RLP emits.
What lives where
| Concern | Location |
|---|---|
| Closed-loop controller, verbs | orchestrator/internal/controller, orchestrator/internal/verbs (Go) |
| Diff emission | Nethermind StateDiffsWriter plugin (#11788) |
| State-composition tracker | orchestrator/cmd/sidecar — RocksDB secondary (#77) |
| Offline structural state scan | orchestrator/cmd/statescan — root→leaf FlatDb walk, per-contract node-types/depth (feeds #29/#35) |
| Cross-EL replay + payload cohere | orchestrator/cmd/orchestrator replay, orchestrator/cmd/heal-payloads (cohere/heal) |
| RPC load tests + mock CL | state-benchmarks Locust framework + Engine-API mock-CL driver (#34) |
| Run artifacts | journal.bin, payloads.rlp, run-manifest.json |
| Reports | this site, /results |
Next: the bloating verbs and how the controller decides what to send. →