Someone posted that their codex run burned through $1,200 in a single afternoon. The diagnosis in the thread was blunt: the agent ran the full test suite every time it wrote a small feature, and that seemed to be the default behavior. One reply suggested the obvious fix, put a line in `agents.md` telling it to prefer targeted tests over full runs. Another pushed back that a full suite mostly burns local CPU, not tokens, so the cost story isn't that simple.
Both replies are right in a narrow way, and both miss the practical problem. A note in `agents.md` is a suggestion the model can ignore the moment it decides the change "touches core logic." And whether the cost shows up as tokens or wall-clock, the waste is real: you don't need to re-run the payment integration tests because you renamed a variable in a date formatter. The agent just doesn't know that, so it plays it safe and runs everything.
The thing worth building here isn't a smarter model. It's a gate the agent has to walk through before it's allowed to run the suite.
Reframe the complaint as a decision point
Full-suite-on-every-change is a decision the agent makes with no information. It sees changed files, it has no map of what those files affect, so it defaults to "test all of it." If you give the agent a cheap way to ask "given these changed files, what's the minimal set of tests I actually need?
, you turn a reflex into a lookup.
That lookup is small enough to build and own yourself. You need three pieces: a place to store a file-to-test mapping, a function that takes changed paths and returns the matching tests, and an endpoint the agent can call as a tool.
What the selector actually is
Start with the map. A plain table does the job: one row per mapping rule, a source path or glob on one side, the test targets it implicates on the other. SQLite is a good fit because the map is something you'll edit constantly as you learn which changes break which tests, and you want to see and fix it by hand without a migration dance. Using the in-platform SQLite editor you can open the table, add a rule after a surprise failure, and correct a bad mapping the same afternoon you notice it. No ORM, no deploy step for a data change.
A minimal schema is enough to be useful:
CREATE TABLE file_test_map (
id INTEGER PRIMARY KEY,
path_glob TEXT NOT NULL, -- e.g. 'src/payments/%'
test_target TEXT NOT NULL, -- e.g. 'tests/payments/test_charge.py'
priority INTEGER DEFAULT 0 -- higher = run first
);Then the selection logic. It takes a list of changed paths, matches each against the globs, collects the distinct test targets, and returns them ordered by priority. This is ordinary string matching and set logic, which is exactly what a small hosted script handles well. You can prototype and run Python online to get the matching right against a few real diffs before you wire anything up, then keep that same script as the backend.
def select_tests(changed_paths, rows):
hits = []
for path in changed_paths:
for row in rows:
if fnmatch(path, row["path_glob"].replace("%", "*")):
hits.append((row["priority"], row["test_target"]))
seen, ordered = set(), []
for _, target in sorted(hits, reverse=True):
if target not in seen:
seen.add(target)
ordered.append(target)
return orderedLast piece: expose it. Host that function as an API endpoint and register it as a tool the agent calls. Now, before the agent runs anything, it hands over the changed file paths and gets back a scoped list. The whole point of leaning into AI agent development this way is that the agent keeps its autonomy, you've just given it a cheaper first option than "run everything."
The boundary, said plainly
This selector does not run your tests. It can't execute another repo's native test runner, because it doesn't live where your code lives and doesn't have your repo's environment, dependencies, or fixtures. What it returns is a selection: the names of the tests that are in scope for this change. The agent still runs them itself, in the repo, with whatever command your project uses.
That's a feature, not a dodge. The selector stays a small, portable, language-agnostic lookup, and the actual execution stays where it belongs. You could point three different repos at the same endpoint with three different maps.
So the flow becomes: agent edits files, agent calls the tool with changed paths, tool returns the minimal set, agent runs that set first. You can still have it fall back to a full suite before a merge or when the selector returns nothing confident, but the per-feature reflex is gone.
Where it earns its keep, and where it doesn't
I'd set expectations honestly. Whether this saves someone the specific kind of money in that $1,200 complaint is unverified, it depends entirely on how your runner is billed and how good your map is. A map with sloppy or missing rules will under-select and let real breakage through, which is worse than wasting money. The map is the product; the code around it is almost trivial. Budget your effort accordingly: start conservative, over-include when unsure, and tighten rules only as you build confidence from real runs.
It also won't catch the failures that come from genuinely global changes, config, shared types, build tooling. For those you want a rule that matches broadly on purpose and pulls in a wide set. The selector doesn't need to be clever everywhere, it needs to be correct about the common case, which is the small, local change that currently triggers a full run for no reason.
That's the whole trick. The agent was never going to reason its way to "this change is safe to scope" on its own. Give it a lookup it can trust, keep the map editable, and let it ask before it spends.