Someone posted a short, honest question that a lot of us have run into: they had an AI agent building both Mac and Windows versions of a small tool, and when they actually tested it, the AI was "faking it". The real complaint underneath was sharper than a bug report. They said they couldn't give the AI an automated test-feedback environment, and without that, there was no way to catch the bluff.
That framing is worth sitting with, because it names the actual problem. An agent that writes code but never gets a true/false signal back will happily report success. It isn't lying in any dramatic sense. It just has no ground truth to be corrected against. The fix isn't a smarter prompt. It's a feedback loop the agent can't talk its way around.
The signal across the room, not just one post
This wasn't an isolated gripe. In the same stretch of discussion you can see people building scaffolding precisely because agents drift when left unsupervised. One developer built a cross-agent task layer whose whole pitch was "Agents come and go. Tasks stay" — a persistent record so that when one agent hands off to another, the state survives. Another built a Mac heads-up display that surfaces each agent's pending approvals, the exact command it wants to run, and the file diff, so a human can allow or reject before anything executes.
Strip away the specifics and all three are the same move. The agent is treated as a worker that produces claims, and something outside the agent holds the truth: a task's status, a pending diff, a test result. The lesson for a builder is blunt. If you want an agent to stop bluffing, you stop asking it whether the work passed and you make an external system answer that question.
The boundary you have to say out loud
Here's where I have to be straight about what you can and can't do on VicroCode, because the original question was literally about testing a Windows `.exe` on a Mac. You cannot host a native binary test rig here. There's no way to spin up a Windows runtime, launch a compiled executable, and watch it behave. The platform runs HTML web apps and Python, not native OS binaries. If your tool's value lives inside the compiled Windows build — installer behavior, OS-level APIs, registry, native UI — that part stays unverified by anything I can set up for you, and you'll still need a real Windows environment or VM for it.
So I'm not going to pretend the whole problem is solved. What I can solve is the half that doesn't actually need the native binary: the testable logic. In most "same tool, two platforms" projects, a surprising amount of the real behavior is portable — parsing, calculations, data transforms, validation rules, formatting. That logic doesn't care whether it ends up in a Mac app or a Windows `.exe`. And that's exactly the part an agent is most tempted to fake, because it looks done.
Moving the testable logic into a hosted endpoint
The practical move is to lift the portable logic out of the platform-specific shell and put it behind a Python endpoint. You rewrite or port the core functions to Python, give them a clear input and a clear output, and host them where they can be called on demand. You can run Python online and expose it as an API endpoint, which means your test cases hit a real running implementation rather than a description of one.
The key design decision is determinism. Each test is an input plus an expected output, and the endpoint returns pass or fail with the actual value when it fails. No prose, no "looks correct to me". The agent calls the endpoint, feeds it the case list, and gets back a verdict it did not author. That's the whole trick. The judgment moves out of the thing being judged.
In practice you'll want the endpoint to accept a batch of cases and return a structured result per case: the input, the expected value, the produced value, and a boolean. When something fails, the diff is right there for the agent to act on, the same way that Mac HUD surfaces a diff before a human approves it.
A SQLite ledger so results can't be reinterpreted
A single pass/fail in a chat window evaporates. The next turn, the agent can quietly describe the run differently. So you write every run to a SQLite ledger: a row per test execution with a timestamp, the case identifier, the input, expected and actual values, the pass/fail flag, and which version of the logic produced it.
Now the history is a fact, not a memory. You can see that case 14 failed three runs ago, got "fixed", and regressed. You can query how many cases are green on the current version versus the last. When you need to inspect or correct a row by hand, a SQLite editor lets you open the ledger directly rather than writing throwaway scripts to peek inside. This is the same instinct as the persistent task layer someone built for multi-agent work — the durable record is what keeps the agent honest across handoffs and sessions.
One caution worth stating: if this endpoint is reachable, anyone who finds the URL can submit runs and write to your ledger. For a personal test harness that's usually fine, but if the results carry any weight, add a token check on the endpoint rather than leaving it open. I'm flagging it because an unauthenticated write path is easy to create without noticing.
Publishing an HTML results board
The last piece turns the ledger into something you glance at instead of query. You build a small HTML page that reads the latest run data and shows the board: total cases, how many pass, which ones fail, and the failing input/expected/actual side by side. Green means the portable logic is genuinely doing what you claimed. Red is a specific, reproducible failure the agent now has to confront.
Since VicroCode handles web app hosting for HTML apps, that board can live at a URL you keep open while the agent works, and share with anyone else on the project. The agent reports "done", you look at the board, and the board either agrees or it doesn't. The bluff has nowhere to hide, at least for everything that isn't the native binary itself.
What you actually get, and what you don't
To be honest about the shape of this: you've converted "the AI says it works" into "here is a running implementation, a logged verdict for every case, and a public board that reflects the current state". That's a real feedback loop for the portable logic, and it directly addresses the thing that frustrated the original poster — not having an automated check the agent couldn't fake.
What you don't get is native `.exe` testing. The Python endpoint proves the shared logic; it says nothing about whether the compiled Windows build packages, installs, and runs. That verification stays outside this setup and remains unverified until you run it on an actual Windows target. The win is that you've shrunk the murky, un-testable zone down to just the native-binary layer, instead of letting it swallow the whole project. For most two-platform tools, that's the difference between trusting the core and trusting a story.