VicroCode
Make Code Create Value
VicroCode is a lightweight online platform for publishing, running, sharing, and monetizing code projects. Launch HTML, Python, SQLite, AI agents, management tools, games, and more without server setup.
Please wait while VicroCode loads. You can also explore the AI programming guide
Loading...

AI MARKET GUIDE

The DIY signal filter nobody admits they need (and how to build one on VicroCode)

Independent devs keep rebuilding the same thing: a pipeline that filters noise with vector search, dedupes hard, and refuses to spam. Here's how to assemble it without a VPS.

If you've spent any time scouting markets or hunting for reusable open-source parts, you know the real tax isn't finding information. It's the volume. One builder put it plainly in a recent writeup: every day is buried under shell-tool mutual promotion and marketing-account anxiety posts, and panning for anything with actual implementation value gets exhausting. So they did what a lot of us quietly do — stopped consuming feeds raw and built a vertical intelligence filter that only surfaces two or three high signal-to-noise cards a day.

That post is worth reading closely, not because the tool is novel, but because the engineering scars are honest. And once you read the scars, you realize the same pipeline can be stood up on a hosted platform without ever touching a Linux box. Let me walk through what they hit, then map it to what's actually buildable on VicroCode.

The problem is dedup, not collection

Collecting items is trivial. The hard part is that the same event gets reported by five outlets inside 48 hours, and you don't want five cards for it. The builder reached for pgvector and plain text embeddings with a cosine similarity threshold. Set it at 0.85 and genuinely different projects in the same niche got killed as duplicates. Loosen it to 0.92 and a reworded press release slipped right through. (Those thresholds are one developer's reported experience, unverified, but the shape of the problem is familiar to anyone who's tuned similarity cutoffs.)

The fix they landed on is the interesting bit: don't trust raw semantic distance alone. First extract entities and normalize them — collapse GitHub repos to a canonical form, fold top-level domains together — and if entities match, merge immediately. Only when entities miss do you fall back to semantic scoring across several axes like implementation feasibility, business viability, and code maturity, which also gives you a natural place to drop pure PR fluff.

Where VicroCode fits, and where it doesn't

The original ran pgvector on a spare VPS. VicroCode doesn't give you pgvector or a Postgres box to SSH into — that's the boundary, and it's a real one. What it gives you instead is a LanceDB knowledge base for the vector side and a SQLite database for everything relational. So you'd restructure rather than lift-and-shift: embeddings and nearest-neighbor lookups live in LanceDB, while your entity table, the canonical-repo map, and the dedup ledger sit in SQLite.

The embedding and scoring calls go through Model Center APIs for the models already available on the platform. I'm deliberately not naming a model or promising a specific one — you use what the platform exposes. The multi-axis scoring logic, the entity normalization, the fold-on-match rule: that's plain Python running online, no environment to provision.

The scoring layer is also the piece that benefits most from treating it as AI agent development rather than a flat script. You want something that can take an item, pull context, score it against your axes, and decide fold-merge-or-drop with reasons you can inspect later. In-platform tool calls let that agent reach your SQLite ledger and LanceDB index as it reasons, instead of you hard-wiring every lookup.

The duplicate-push disaster is the part everyone skips

Here's the scar that stuck with me. In their async queue, the external send (pushing to a channel) succeeded, but the database bookkeeping failed because two concatenated SQL statements tripped the underlying prepared-statement parser. The transaction rolled back, the queue engine decided the task hadn't finished, and exponential backoff kicked in — so the same message got re-pushed to the external endpoint in a tight loop.

The lesson they drew is the right one and it's platform-independent: any side effect that crosses an external network boundary has to be made atomic against local database state. Insert an explicit pending placeholder lock before the outbound call, not after.

On VicroCode this maps cleanly. Your ingestion and dispatch logic lives behind a hosted API endpoint. Before any outbound action, you write a pending row to SQLite keyed on the item's dedup identity; the write succeeding is what authorizes the send; on confirmation you flip it to done. If the item's already pending or done, you short-circuit. One honest caveat on the boundary: pushing out to something like Telegram or a chat channel is your own outbound integration, wired in your code — it isn't a prebuilt connector I can promise. The endpoint hosting and the state machine around it are the confirmed part.

Push discipline is a feature, so encode it

The third scar is behavioral, not technical. The easy failure in any aggregator is thinking everything's worth sharing and never shutting up. They found that past roughly four items a day, a person just goes numb, so they hard-coded the discipline into the backend: a daily cap, a minimum interval between pushes, and total silence overnight with tasks deferred to morning. That's not config you expose and forget to set — it's a physical gate in the code path. SQLite is the natural place to track the daily count and last-push timestamp, and your dispatch function checks the gate before it even considers sending.

Give yourself somewhere to look

A push-only pipeline has a blind spot: when you want to review what got filtered and why, scrolling a chat history is miserable. This is where a small hosted dashboard earns its keep. You can run HTML online as a lightweight web app that reads your SQLite tables and renders the day's cards, the fold decisions, and the items the scorer dropped with its stated reasons. That review surface is also your tuning loop — when a good project gets wrongly merged, you can see it and adjust the entity rules or the axis weights instead of guessing.

A few things stay firmly outside the platform's lines, and it's worth naming them so you don't design around capabilities that aren't there. Fetching external RSS feeds depends on those feeds being reachable; one builder's site kept getting removed from an aggregator because anti-bot rules returned 403 to the crawler. Nothing here bypasses that. You also won't be managing your own daemon, systemd unit, or VPS — the trade is that you skip the ops entirely and give up the low-level control.

The reported outcomes from the original build — two to three clean cards a day, the specific thresholds, the latency of the whole thing — are that developer's experience and remain unverified. What's reproducible isn't the numbers. It's the architecture: entity-first dedup with a semantic fallback, atomic pending locks before every external side effect, and discipline baked into the code instead of left to willpower. Assemble those three on top of Python, SQLite, LanceDB, hosted endpoints, and a small HTML viewer, and you've got the same tool without the server.