Building a Token Cost Tracker Before You Commit to an Agent Harness
Caijing ran a telling experiment in August 2024. They subscribed to domestic and international token plans, then used an open-source agent framework to run the same task repeatedly until the weekly quota dried up. The task was straightforward—extract ten financial metrics from 24 quarters of earnings reports—but the token consumption wasn't.
Alibaba's ¥499 plan delivered 115M tokens per week. OpenAI's comparable tier gave 2.5B. That's a 22x difference in usable quota, and it showed up fast when the test ran continuously.
The gap came from three places. First, raw quota allocation: domestic plans simply handed out less. Second, cache hit rates: Alibaba's plan registered under 90% cache hits, while OpenAI stayed above 95%. Cache misses burn tokens at roughly 10-15x the rate of hits. Third, output ratios: Alibaba's 2.4% output share versus OpenAI's 0.2% meant more tokens spent generating responses relative to input processing.
If you're building AI agent development tooling or evaluating harness frameworks, those differences compound quickly. A developer mentioned building a nine-part video series walking through harness implementation from scratch—循环上下文 in episode one, tool functions in episode two. That's a solid learning path, but if the underlying token economics break your budget halfway through episode five, the harness design becomes academic.
Before you commit to a framework or a subscription tier, build a cost monitor that logs what actually burns tokens in your workflow.
What to Track
You need four numbers per task run:
**Cache hit rate.** Every agent call that reuses context should log whether the cache served it or the model reprocessed it. A 5% difference in cache efficiency can double your monthly bill.
**Output ratio.** Compare output tokens to total tokens consumed. If your agent generates verbose responses or produces structured JSON with redundant keys, output tokens climb. The Caijing test showed this ranging from 0.2% to 2.4%—a 12x spread.
**Per-task token burn.** Log total tokens consumed for each discrete task. If you're running financial report extraction, log tokens per report. If you're summarizing support tickets, log tokens per ticket. Aggregate stats hide the tasks that silently eat quota.
**Time-to-quota-exhaustion.** Run a test workload and measure how long it takes to hit your weekly or monthly cap. Caijing's test revealed that Alibaba's plan lasted days, not weeks, under continuous load.
These four metrics tell you whether a plan or model will survive production use.
Building the Monitor
VicroCode lets you run Python online and store structured logs in SQLite, which makes this a natural fit for a lightweight tracker.
Start with a Python script that wraps your agent calls. After each task completes, log the metrics to a local SQLite database. Structure the schema around task runs:
- `task_id`: unique identifier for the task
- `model`: which model served the request
- `cache_hit`: boolean, true if cache served the response
- `input_tokens`: tokens sent in the request
- `output_tokens`: tokens generated in the response
- `total_tokens`: sum of input and output
- `timestamp`: when the task ran
Run your typical workload—extracting data, summarizing documents, generating code—and let the logger capture a week's worth of real usage. Then query the database.
Calculate average cache hit rate:
SELECT AVG(CASE WHEN cache_hit THEN 1.0 ELSE 0.0 END) FROM task_logs;Calculate output ratio:
SELECT SUM(output_tokens) * 100.0 / SUM(total_tokens) FROM task_logs;Calculate tokens per task:
SELECT task_id, SUM(total_tokens) FROM task_logs GROUP BY task_id;If you're working with multiple models or plans, add a `plan` column and segment the analysis. Compare domestic and international tiers side by side using the same task set.
VicroCode's SQLite editor makes it easy to inspect the logs interactively and adjust queries as you discover patterns. You can also export the results to a CSV or visualize trends over time if token burn spikes unexpectedly.
What the Data Tells You
Once you have a week of logs, the numbers clarify what subscription math obscures.
If your cache hit rate is below 90%, investigate whether your agent is structured to reuse context. Some frameworks rebuild the full context window on every call instead of appending deltas. That design choice quietly doubles your token spend.
If your output ratio is above 1%, check whether your prompts are generating unnecessary verbosity. Agents that return full explanations, markdown formatting, or nested JSON when a simple answer would suffice will burn through quota faster than agents tuned for concise responses.
If certain tasks consume 10x more tokens than others, prioritize optimization there. The Caijing test used a uniform task, but real workloads vary. A financial report with 50 tables might cost 10x what a two-page summary costs. Knowing which tasks are expensive lets you batch them, cache aggressively, or offload them to cheaper models.
If your weekly quota runs out in three days under realistic load, you know the plan won't survive production. Scale your cost projections accordingly before committing to a harness that assumes continuous availability.
Boundary Conditions
This approach works when your tasks are Python-based and you can instrument the agent calls. If your harness is written in another language or relies on a closed vendor SDK, you'll need to adapt the logging layer.
VicroCode supports Python execution and SQLite storage, but it doesn't natively run JavaScript frameworks, Java agents, or deployed cloud functions. If your harness depends on those, you'll need to export logs from the runtime and import them into SQLite for analysis.
The tracker also assumes you can replay a representative workload. If your agent responds to unpredictable user queries, log production traffic instead of synthetic tests. The metric definitions remain the same; only the data source changes.
Why This Matters Now
Developers are publishing harness tutorials, building agent frameworks, and sharing architectures that assume token availability is predictable. The Caijing test showed it isn't. A ¥499 domestic plan and a $200 international plan might look comparable on paper, but one delivers 22x more usable quota in practice.
If you build a harness on optimistic assumptions about cache efficiency and output ratios, you'll discover the real cost when production traffic hits and the quota evaporates mid-week. By that point, refactoring the agent to be more efficient is harder than building cost tracking into the prototype.
Log the four metrics early. Run the test workload before you commit to a plan or framework. Let the data guide architecture decisions instead of discovering constraints after deployment.
The evidence from Caijing's test is unverified in the sense that VicroCode hasn't independently reproduced it, but the methodology is sound and the pattern is recognizable. Token economics vary widely across providers, and surface-level pricing doesn't reveal consumption behavior. A monitor that captures cache rates, output ratios, and per-task burn gives you the ground truth you need to choose a harness and a plan that will actually survive contact with production workloads.