Someone ran a scan I keep thinking about: 1000 big Chinese sites, each one hit at `/llms.txt`, just to see who had done their homework for AI crawlers. The headline number is bleak-your-coffee bad. Only 30 sites served a real file. Three percent. But the number that actually matters for anyone building a checker is the next one down: 164 sites returned a clean `200 OK` and then handed back an HTML page. Status code says yes, content says no. The crawler gets a shell and walks away with nothing.
That gap is the whole project. Fetching `/llms.txt` is trivial. Refusing to count a 200-with-HTML as a pass is where a naive scanner quietly lies to you. So I rebuilt the idea on VicroCode as three small pieces that fit together: a hosted Python fetch-and-score endpoint, a SQLite ledger you can hand-correct, and an HTML report page that grades a domain and drafts a starter file when there isn't one. Here's how it went and where the honest boundaries are.
The scoring logic is a content check, not a status check
The survey's own breakdown is the spec. Of the 1000 sites: 30 real files, 164 HTML shells behind a 200, 412 flat 404s, and 390 that simply couldn't be reached because a WAF blocked the request or the connection timed out. The author was careful to say that last bucket is not the same as "missing" — you don't know what's there, so you don't get to score it. That restraint is the part most tools skip.
So the fetch step can't stop at the HTTP code. After pulling the body, you have to classify it. A genuine llms.txt is plain text, usually Markdown-ish, with headings and link lines that carry descriptions. An HTML shell gives itself away immediately: a doctype, a `<html>` tag, a wall of `<div>`s, a content-type of `text/html`. The rule I settled on is boring on purpose — if the body parses as HTML or the content-type is HTML, it's a shell, full stop, regardless of the 200. I'd rather under-credit a borderline file than repeat the exact mistake the survey was built to expose.
I stood this up as a Python backend. You can run Python online to do the fetch, inspect the response headers, sniff the body, and emit a structured verdict — real file, HTML shell, 404, or unreachable — without maintaining a server of your own. The scoring beyond the shell check is deliberately shallow: does each link line carry a description, does each section have a summary. The survey found the common failures were exactly that mundane — a bare URL with no description, a missing one-line summary — the kind of thing it said takes about half an hour to fix. A checker that flags those is more useful than one that invents a clever grade.
WAF and timeout cases get labeled, not scored
Here's the boundary I want to be straight about, because it's easy to paper over. Server-side Python fetching a third-party domain is still just an HTTP client. If a site's WAF blocks it or the request times out, that's a real outcome, and the honest move is to record it as `unreachable` and leave the grade empty. The platform does not give me a way around a WAF, and I wouldn't want one — pretending a blocked domain scored zero would be the same category of lie as counting a shell as a pass. In the survey that was 390 of 1000 sites. On a rebuild it'll vary, and whether a given domain blocks a cloud egress IP is something I can't verify ahead of time. The report has to say "couldn't reach this" and mean it.
One more honest flag: if you host this as a public endpoint, it's an open fetch tool pointed at arbitrary domains. There's no auth on it by default, which means anyone can drive your scans. Before putting it anywhere real I'd add a key check and basic rate limiting through the platform's endpoint hosting, and I'd keep the target list to domains you actually want to audit rather than a wide-open crawler.
A scan ledger you can hand-correct
Automated classification will be wrong sometimes, and you want somewhere to fix it by hand without redeploying anything. That's what the ledger is for. Each scan writes a row — domain, timestamp, HTTP status, content-type, verdict, score, the warnings it raised — into a SQLite database. When the classifier mislabels something, say a text file that happened to open with an HTML comment, you open the SQLite editor and correct the verdict directly in the row. The scan history stays queryable, so you can filter for every HTML-shell case or pull just the unreachable ones to retry later. For a dataset the size of a survey, a single SQLite file is plenty, and keeping the corrections next to the raw results means the whole thing stays auditable instead of becoming a black box.
This matters more than it sounds. The survey author published the full 1000-row dataset precisely so people could search and filter and argue with the method. A ledger you can read and edit by hand is what makes a scan trustworthy to someone who wasn't there when it ran.
The report page, and generating a starter file
The front door is a report page: type a domain, see the grade, see the warnings in plain language. You can run HTML online to host this and wire it to the Python endpoint, so a non-technical operator gets a readable verdict instead of a JSON blob. When a domain comes back with no file at all — the 412 flat-404 bucket in the survey — the page offers a starter `llms.txt`: a skeleton with the site name, a one-line summary slot, and link lines that already prompt for descriptions, so the output doesn't ship the exact bare-URL mistake the survey kept finding. The survey's own tooling did something similar, generating a file in about thirty seconds for sites that had none. Drafting a template is squarely inside what these pieces can do; actually publishing it to the target site is not, since that's someone else's server.
I won't quote a pass rate or a speed for this rebuild — I haven't run it against a thousand domains, and any such number would be unverified until I did. What I can say is the architecture holds: the content check kills the false pass, the ledger makes corrections cheap, and the report turns a raw verdict into something a site owner can act on.
Why this is worth a weekend
The broader signal in the survey is that AI-readiness is mostly unclaimed ground, and the failures are small and fixable rather than deep. A real file on a top site was the exception, not the rule, and even the ones that tried often tripped on duplicate links and missing descriptions — doing it versus doing it well being two different things. For an independent builder, that's a clean opening: a focused tool that grades honestly, keeps its own books, and hands people a correct starting file is useful on day one, and it's small enough that the hard part is judgment, not engineering. The trick is just refusing to let a 200 lie to you.