The share that made me want to build a logbook
Someone posted their embedded KV engine, Mace, written in Rust: a transactional store that chases predictable B+ tree read latency and LSM-style write throughput at the same time. Bw-tree ordered index, MVCC snapshot isolation, append-only WAL with async checkpoints, blob separation for big values, optional zstd, CRC checks. Solid stuff for a two-year side project.
What caught my attention wasn't the feature list. It was the confession buried in the middle: they picked between sled and RocksDB, couldn't get RocksDB running because it was too heavy, and ended up building toward RocksDB as a target without ever running a proper side-by-side against other stores. Then they dropped a table anyway.
The numbers they shared were specific: relaxed durability, snapshot reads, 16-byte keys, 128-byte values, 1M keys, 8 threads. W1 (95% read, 5% write) came in at 2,941,301 ops/s for Mace against 1,100,399 for RocksDB, a 2.67x gap. W2 with a zipfian distribution stretched to 2.88x. But by W3 (50/50) it was 1.97x, W4 (5% read, 95% write) dropped to 1.49x, and the scan workload W6 landed at 1.62x. To their credit they said it plainly: read-heavy and scan look good, write-heavy shrinks the advantage.
That honesty is the whole story. A 2.67x number means nothing until you can see the exact conditions that produced it. Change the value size to 4KB, flip durability to fsync-per-write, drop the thread count, and the ranking can invert. Most benchmark posts show the winning row and hide the parameter sheet.
The real deliverable is the parameter sheet, not the chart
When I've shipped storage comparisons before, the argument was never about who was faster. It was about whether the two systems were doing the same work. Someone always asks: were both running with the same durability mode? Same key distribution?
Was one warmed up and the other cold? Without those answers pinned to each row, a benchmark is a vibe, not a measurement.
So the useful thing to build from the Mace share isn't another benchmark tool. It's a journal. Every run gets recorded with its full settings, immutably, and the comparison page reads from that record instead of from memory. You stop debating and start pointing at rows.
The structure is boring on purpose:
- Each run is one row.
- Each row carries the engine name, workload label (W1 through W6, or whatever you define), key size, value size, key count, thread count, durability mode, read/write ratio, key distribution, and the measured ops/s.
- Nothing gets overwritten. A re-run is a new row with a new timestamp.
That schema is what lets you say something defensible later. When a reader challenges a figure, you don't hand-wave. You show the exact row.
Building the harness
The harness itself is small. You need something that runs your workload, captures the parameters it ran under, and appends a row. I keep this in Python because the glue is trivial and I can run Python online without babysitting a local toolchain or fighting a heavy build the way the Mace author fought RocksDB.
A sketch of the recording layer:
import sqlite3
import time
def init_db(path="bench.db"):
conn = sqlite3.connect(path)
conn.execute("""
CREATE TABLE IF NOT EXISTS runs (
id INTEGER PRIMARY KEY,
ts TEXT NOT NULL,
engine TEXT NOT NULL,
workload TEXT NOT NULL,
key_size INTEGER NOT NULL,
value_size INTEGER NOT NULL,
key_count INTEGER NOT NULL,
threads INTEGER NOT NULL,
durability TEXT NOT NULL,
read_ratio REAL NOT NULL,
distribution TEXT NOT NULL,
ops_per_sec REAL NOT NULL
)
""")
conn.commit()
return conn
def record_run(conn, params, ops_per_sec):
conn.execute("""
INSERT INTO runs (ts, engine, workload, key_size, value_size,
key_count, threads, durability, read_ratio,
distribution, ops_per_sec)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
time.strftime("%Y-%m-%dT%H:%M:%S"),
params["engine"], params["workload"], params["key_size"],
params["value_size"], params["key_count"], params["threads"],
params["durability"], params["read_ratio"],
params["distribution"], ops_per_sec,
))
conn.commit()The important discipline: the same dictionary that configures the run is the dictionary that gets logged. There's no separate step where you type the parameters into a spreadsheet by hand and quietly get them wrong. If the run used a 128-byte value, the row says 128, because the run and the record read from the same source.
Re-populating the Mace share into this schema, W1 becomes two rows, one per engine, sharing identical key_size, value_size, key_count, threads, durability, and distribution, differing only on engine and ops_per_sec. Now the 2.67x is a computed ratio between two rows you can inspect, not a headline.
Editing runs by hand without breaking the record
Real benchmark journals get messy. You mistype an engine version. You realize a run used the wrong durability flag and needs a correction note. You want to prune a warmup run that skewed the picture. For that I keep the database open in a SQLite editor so I can fix a cell or delete a bad row directly, then let the comparison page pick up the change on its next read.
A couple of habits keep the hand-editing honest:
- Never edit an ops_per_sec value to make a story look better. If a number was measured wrong, delete the row and re-run.
- Treat corrections as new rows where you can. The timestamp column exists so the history stays legible.
This is the part where discipline beats tooling. The database will happily let you rewrite reality. The point of the journal is that you choose not to.
Publishing the comparison as a page people can actually read
The last piece is turning the table into something shareable. I export the rows to a static HTML page with the parameter sheet visible on every comparison, then run HTML online to host it so a link is enough for anyone to review the full context, not just the winning row.
The layout I'd use:
- A grouped view by workload, so W1 shows both engines with the shared parameters printed once above the pair.
- The ratio computed and displayed, but always next to the raw ops/s for both engines.
- A fixed disclaimer block on the page, not in fine print.
That disclaimer matters, and it's the boundary I won't cross. The Mace figures, and any figures you generate on your own machine, are the author's own numbers under the author's own conditions. They are explicitly unverified. Different hardware, a different build, a warmed versus cold cache, and a different RocksDB tuning would all move the ranking. Publishing the parameters is what makes the claim reproducible; it does not make the claim verified. Say that on the page, plainly.
Where the boundaries actually sit
A few honest limits so you don't plan around things that aren't there.
The harness described here logs and compares numbers you feed it. It does not compile or run a Rust engine like Mace itself; the actual benchmark of a native storage engine happens in that engine's own toolchain, and you're recording the results into the journal. The Python side is the recording, aggregation, and publishing layer, and the SQLite file is your source of truth for it. If you want the harness to drive a Python-implemented store or a Python-callable backend directly, that fits; driving a Rust binary's internals does not.
The hosted HTML page is a static comparison view. It reads a snapshot of your data at publish time. It's a report, not a live dashboard wired into a running cluster.
And the whole thing is only as trustworthy as your willingness to log the ugly runs alongside the flattering ones. The Mace author showed W4 dropping to 1.49x right next to the 2.88x on W2. That's the model. A benchmark journal that only records wins is just marketing with extra steps.
Build the recorder, pin every parameter to every row, let yourself correct mistakes in the open, and publish the sheet with the numbers. You end up with something better than a fast headline: a claim someone can check.