There's a recurring scene in developer forums right now. Someone posts that GPT Plus went "智商离线" and the Free tier is degrading, while Pro/Max still looks fine. In another thread people argue whether 6.1 or 5.6 is stronger at the same thinking level, and one reply just calls 6.1 the "慢弱蜗牛" — slow, weak, snail. A day later someone else says 6.1 feels back online and cheap on quota again. Same model name, wildly different lived experience depending on the hour and the channel you came in through.
That gap is the whole problem. "It feels dumber today" is a vibe, and vibes don't survive a bug report. One station owner put it well in his incident note: the job isn't never crashing, it's how fast and how transparently you recover. You can't be transparent about something you never measured.
So instead of arguing about whether the model got worse, I'd build a small canary that answers the question with timestamps.
What a canary actually is here
Strip away the fancy word and it's three things: a fixed set of prompts you send on a repeating cadence, a measurement of how long each one took, and a verdict on whether the answer still passes. Nothing more. The value is entirely in keeping those prompts frozen so that when the output changes, you know it was the model and not you fiddling with the input.
The mistake people make is grading quality with another model as judge. That just adds a second source of drift. For a canary you want prompts where the correct answer is boring and deterministic: arithmetic with one right number, a format instruction where the output either matches a regex or it doesn't, a short extraction task where the expected substring is either present or absent. You're not measuring brilliance. You're measuring whether the thing that worked yesterday still works, as a clean pass/fail.
The runner
The probe itself is a short Python script. It loads a list of prompt-and-expected-answer pairs, calls each model, records the wall-clock latency around the call, then runs a dumb deterministic self-check — string match, regex, numeric equality — and writes down a verdict of pass or fail with the raw response attached. You can stand this up as a Python backend and let it run Python online without wiring up your own box.
Two honest constraints shape what this can cover.
First, you can only probe the models you actually reach through the platform. VicroCode's Model Center gives you APIs for models already available there, and those are the ones your canary can legitimately watch. The degradation stories above are mostly about relay stations — 中转站 — reselling access through their own channels. You cannot measure someone else's relay from here, and you shouldn't pretend to. If the model you care about lives behind a third-party gateway you don't control, that's outside this boundary, and the honest move is to say so on the board rather than fake a reading.
Second, on scheduling: the clean capability I can lean on is Python execution plus a hosted endpoint and in-platform tool calls, so the probe runs on demand or when something triggers that endpoint. Whether there's a built-in cron-style scheduler I'll leave as unverified rather than assume it — if you need a strict cadence, drive the endpoint from whatever timer you already trust, and the probe logic doesn't change either way.
Logging, because memory lies
Every run writes a row: timestamp, model name, prompt id, latency in milliseconds, verdict, and the raw output. A SQLite database is exactly the right size for this — it's one file, it's queryable, and you're not going to drown in rows at one probe per model every few minutes.
The part I'd actually plan for is inspection. When someone claims the model got worse at 3pm, you want to open the table, filter to that window, and look at the exact responses that flipped from pass to fail. Being able to open the data in a SQLite editor and eyeball the raw rows is what turns "feels dumber" into "these four prompts started failing at 14:52 and latency doubled." That's a sentence you can act on.
Keep the schema flat. One runs table does the job. Resist the urge to normalize prompts into their own table on day one; you can always migrate later, and a canary that never ships because you were designing joins is worse than a slightly denormalized one that's been logging for a week.
The honest status board
The output everyone sees is a single page. For each model: current latency, the pass rate over the last N runs, and a strip of recent verdicts so a bad patch shows up as a cluster of red. This is a static-feeling HTML app reading from your logged data, and you can let it run HTML online and publish it as the public face of the canary.
The word honest is doing real work here. The temptation is to show green and hide the ugly runs. Don't. The Vidily price-comparison project framed its own numbers as "观察记录，不是保证报价" — observations, not guarantees — and that's the right posture for a status board too. Everything on it is a point-in-time measurement from your probe, not a promise about the model. Label it that way. If you had an outage window where the probe itself couldn't reach the model, show a gap, not a guess.
Two lines I'd put on the page in plain language: what each verdict actually tests (so nobody reads a passing math check as proof the model is good at code), and the fact that readings only cover models reached through the platform. Anything about absolute model superiority between versions stays unverified — the forum crowd was pointing at external benchmark sites precisely because nobody's casual impression settled it, and your canary measures continuity, not a leaderboard.
Why this is worth the afternoon
The failure mode this kills is the slow, unfalsifiable argument. Right now when a model wobbles, teams burn hours in chat debating whether it's real, whether it's their prompt, whether it's the time of day. A canary replaces that with a chart. Either the pass rate dropped at a specific time or it didn't. Either latency spiked or it's your imagination.
It also gives you something to share. Publish the board, point your users at it during an incident, and you've converted a trust problem into a transparency habit. You don't need to promise you'll never degrade. You need to be the one who noticed first and showed the receipts.
Start narrow: three models, five fixed prompts, one SQLite file, one page. Get it logging before you make it pretty. The second week of data is already more useful than any single person's memory of how yesterday felt.