Building a Coding Agent Cost Monitor After FrontierHarness Revealed 5.6x Budget Variance
Runta's FrontierHarness eval ran the same model—Kimi K3—through 9 different agent harnesses on 30 coding tasks. Pass rates ranged from 50.0% to 66.7%. Cost per solved task ranged from $1.05 to $18.34.
Two harnesses, DSH Creator and Claude Code, both passed 19 out of 30 tasks. Identical 63.3% success rate. DSH Creator spent $3.28 per solved task. Claude Code spent $18.34. That's 5.6x the budget for the same result.
The variance came from how each harness managed context, retries, and tool calls. The model stayed constant. The tasks stayed constant. The harness became the cost driver.
The Problem: You Don't Know What You're Paying Until You've Already Paid
If you're evaluating agent frameworks or building your own orchestration layer, you need to compare configurations before committing. But most setups don't log cost at task granularity. You see total spend at the end of the month, not per-task breakdowns that show where the budget went.
Failed tasks are expensive. In one local run using the NXS harness, 8 failed tasks consumed 53.4% of recorded spend and 46.4% of tool calls. Success tasks finished in a median of 3 minutes 35 seconds. Failed tasks ran for 19 minutes 52 seconds before stopping.
When a harness retries the same operation, context grows, cache hit rate drops, and token costs accumulate. If you're running evals across multiple configurations, you need per-task logs that capture token counts, retry cycles, and cost before the next billing cycle closes.
What You Can Build: A Python Cost Tracker with Task-Level Granularity
You can run Python online and build a tracker that logs every agent execution as a structured record. Each record stores task ID, harness configuration, token usage (prompt + completion), retry count, success or failure, and calculated cost based on your model's pricing.
Store results in an SQLite editor database with a schema like this:
CREATE TABLE task_runs (
run_id TEXT PRIMARY KEY,
task_name TEXT,
harness_config TEXT,
model TEXT,
prompt_tokens INTEGER,
completion_tokens INTEGER,
retry_count INTEGER,
duration_seconds REAL,
success BOOLEAN,
cost_usd REAL,
timestamp TEXT
);Wrap your agent execution loop so every task call logs to this table. If you're using a framework that exposes token counts in response metadata, parse them into the schema. If not, estimate from returned message length and known token ratios for your model.
Once you have 20-30 tasks logged across two or more configurations, query for cost-per-solved-task:
SELECT
harness_config,
COUNT(*) FILTER (WHERE success) as solved,
SUM(cost_usd) FILTER (WHERE success) as total_cost,
SUM(cost_usd) FILTER (WHERE success) / COUNT(*) FILTER (WHERE success) as cost_per_solve
FROM task_runs
GROUP BY harness_config;You'll see which configuration wastes budget on long retry loops and which one solves tasks efficiently. If one config shows 3x the cost per solve, you know before scaling.
Adding a Hosted Dashboard
Use web app hosting to turn the SQLite database into a live comparison dashboard. Read from the database, render a table or chart that shows cost-per-solve by configuration, and update it every time you finish a batch of tasks.
You can plot:
- Cost per solved task by harness
- Retry count distribution (unverified: whether retry reduction correlates with lower cost in your environment)
- Token usage by task type
- Success rate vs. median cost
The dashboard doesn't need to be complex. A single HTML page with a query endpoint and a chart library is enough to see the variance.
What the Boundary Is
VicroCode supports Python execution, SQLite storage, and hosting for HTML dashboards. It does not provide native integrations with agent frameworks like LangGraph, distributed tracing systems, or production observability stacks. If your agent runs outside VicroCode, you'll instrument it separately and push logs to the VicroCode-hosted database via API.
This setup works for local eval runs, internal comparisons, and small-scale harness testing. For production multi-agent deployments with real-time alerting or cross-cloud log aggregation, you'll need additional infrastructure.
Why This Matters Now
One developer running NXS locally burned through half the recorded budget on 8 failed tasks. Another user's subscription to a model plan consumed 50% of monthly credits in two days when using a pro-tier model (unverified: whether usage pattern or model tier was the primary driver).
Cost variance isn't an edge case. It's the default when harnesses manage retries, context windows, and tool call chains differently. Before you choose a harness or tune your own orchestration layer, log the per-task breakdown. Otherwise, you're flying blind until the bill arrives.