Why is the complaint actually a retrieval bug?
Someone on v2ex posted a small frustration that I think a lot of builders feel but rarely name. They asked an agent to find "as many passive-income developer platforms as possible," and it handed back the same handful of famous names. Dig into why, and it's not that the agent is dumb. Its search tool leans on whatever the search engine ranks first, which means SEO-winning platforms dominate the results. The poster's conclusion was blunt: the pages that rank best are not necessarily the pages you need.
That's the part worth sitting with. Agent search is great at shallow lookups and genuinely bad at deliberate divergence. When you want the long tail, the just-launched, the thing with no backlinks yet, ranking actively works against you. The same poster sketched a workaround: start from a root page, pull semantically similar pages with vector search, let an AI filter down to the ones that actually match intent, then re-expand from those. They ran it on the passive-income case and said they turned up dozens of possibly-relevant sites, some pre-launch or still in testing. Treat that result as unverified, it's one person's anecdote, but the shape of the idea is sound.
Why does this matter now?
Two other things in the same feed reinforce why the long tail is where the interesting stuff hides. One post argued that individual developers should "make feature-poor things," because you can't win a feature war with a big platform, you win by solving the problem they ignore. Another was just a solo dev shipping an open-source AI try-on site with 1,407 reference images. Neither of those is going to out-rank an incumbent. Neither is going to show up when an agent asks the search engine for the best option. If your discovery process only ever returns the top of the ranking, you never see the people doing exactly this kind of work.
So the opportunity isn't "build a better search engine." It's building a narrow tool that does one thing the big search doesn't: deliberately surface the obscure. Feature-poor, on purpose.
What does the discovery loop actually need?
Break the loop into stages and it maps onto a few concrete pieces:
- A **seed**: one or a few root pages you already trust as on-topic.
- **Expansion**: given a page, find pages that are semantically near it.
- **Filtering**: an AI pass that reads candidates and keeps only the ones matching your real intent, not just surface-similar ones.
- **Re-expansion**: feed the survivors back in as new seeds and go again.
- A **review surface**: somewhere a human can look at each round, kill the junk, and hand-seed the next pass.
The filtering and the review surface are where most of the value sits, because similarity alone will happily wander off into adjacent-but-wrong territory after a round or two.
How would you build it on VicroCode?
Here's how I'd assemble this with the platform, and where I'd be honest about what it can't do.
The expand and filter passes are plain scripts, so I'd run Python online to fetch a page, extract its text, call an embedding model, and run the AI relevance check. Keep each pass small and idempotent: one function takes a batch of candidate URLs, returns kept ones with a short reason. Both the embedding step and the filter step depend on a model being available through Model Center; wire those in rather than assuming any specific provider, and if the model you want isn't on the platform, that's a hard boundary, not something to work around.
For similarity I'd keep a LanceDB knowledge base of page vectors. Every page you fetch gets embedded and stored with its URL and the round it entered on. "Find N similar pages" then becomes a nearest-neighbor query inside the corpus you've built. Round metadata and the human verdicts (keep, drop, promote-to-seed) live naturally in a small SQLite table alongside it, which also gives you a cheap audit trail of why a site survived.
The review board is a static page with a bit of interactivity, so I'd use the web app hosting to publish an HTML board that lists each round's survivors, lets you tick which ones to carry forward, and posts the chosen seeds back to a hosted endpoint that kicks off the next pass. That manual gate is the point, not a limitation. It's what keeps the loop from drifting.
What are the honest limits?
Two of them, and they're real.
First, **this is not a web crawler**. VicroCode gives you Python execution and endpoint hosting, not a sanctioned way to crawl the open web at scale, and you shouldn't try to turn it into one. The expansion step depends on where you get "pages similar to this page" from. If that's an external similarity source you bring in, you own its terms, cost, and rate limits. What the platform contributes is the corpus store, the compute for the passes, the AI filter, and the review surface, not a standing index of the whole internet. Size your ambitions to polite, small-batch fetching.
Second, **seed quality decides everything**. A loop that starts from a generic root drifts toward generic results, the exact failure you were trying to escape. The AI filter buys you some protection, but garbage seeds in still means mediocre survivors out. Budget your attention for the seeding and the per-round pruning, because that's the human judgment the automation can't replace.
If you build it narrow, treat the discovery claims as something you verify yourself rather than trust, and keep a person in the loop each round, you get a small tool that does the one thing ranked search refuses to: show you the stuff nobody optimized for being found.
FAQ
Why does AI search keep citing the same platforms?
Because retrieval favours sources it can already crawl and parse. If your pages are not crawlable or not structured, they never enter the candidate set.
What breaks the loop?
Cheap crawlable pages with clear answers, plus a discovery index that surfaces sources outside the top five.
Is a discovery tool enough?
No. It only helps if the destination page answers a specific question in its first two sentences.