Sai's release note for the polox_ai image agent caught my eye because it names a specific, unglamorous problem: editing text inside images that are absolutely crammed with text. The fix in that release was to have an LLM detect the coordinates of each text run, drop numbered markers on a canvas, and pair every marker with its own edit box. That's a smart move. When an image has thirty labels on it, a single "edit text" button is useless — you need to see which box maps to which word before you touch anything.
I rebuilt a version of that workflow to see how far the confirmed VicroCode capabilities carry it. The short answer: the detection-and-mapping half is very buildable, and the honest boundary sits at the pixel rewrite. Let me walk through both.
The need, stated plainly
Text-heavy images break the usual editing loop. A screenshot, a diagram, an annotated photo — each carries multiple independent text regions, and an operator wants to change some without hunting pixel by pixel. Sai's insight was to make the mapping explicit: number every detected region, and give each number a field. The human never guesses which box is which. That's the part worth copying, and it doesn't require anything exotic.
What the canvas layer does
The front end is a plain HTML page. You load an image onto a canvas, then draw rectangles at the coordinates you were handed, each stamped with a number in the corner. Below or beside the canvas, you render one edit box per number, so marker 3 on the image lines up with field 3 in the panel. When someone edits field 3, you can highlight box 3 to confirm the link. This is exactly the kind of self-contained interface you can run HTML online without a build pipeline, which keeps the iteration loop tight while you're still tuning the overlay math.
The fiddly bits here are unglamorous but real: canvas coordinate scaling when the displayed image is smaller than the source, keeping marker labels readable against busy backgrounds, and making the box-to-field highlight obvious on a crowded image. None of that is hard, but it's where the time goes.
Where the detection actually happens
The canvas doesn't detect anything on its own. It needs a records payload: for each text region, a bounding box and the recognized string. That's a job for a Python backend. You send the image to an endpoint, the endpoint calls a model available through the Model Center, and it returns a list of coordinate-plus-text records that the canvas can draw directly. Being able to run Python online and expose it as an API endpoint means the detection logic and the model call live in one place, and the HTML page just fetches records.
A practical note on the model call: LLM-reported coordinates drift. The numbers you get back are often close but not pixel-perfect, and the model can merge two adjacent labels or split one. I'd treat the returned boxes as a first pass that the human nudges on the canvas, not as ground truth. Building the overlay to be draggable early saves you from pretending the detection is more precise than it is. Whether any specific model's coordinate accuracy is good enough for your images is unverified until you test it on your own set.
The editable store
Each marker needs somewhere to live. A SQLite table keyed by marker id — holding the box coordinates, the original detected text, and the edited text — gives you a clean record per region. That lets you reload a session, track which fields changed, and export the edits as structured data. This is the quiet backbone of the whole thing: the canvas is just a view over that store, and the edit boxes are just writes to it. Once it's working you can publish the whole helper as one of your hosted online tools and share it.
The line I won't cross
Here's the honest part. Detecting text, mapping it, and storing edits is the winnable half. Rendering the edited text back into the original image — erasing the old pixels and painting new ones so the result looks native — is inpainting, and that is not something the confirmed VicroCode capabilities cover. There's no image-generation or pixel-compositing capability in the set I'm working from. Claiming otherwise would be inventing a feature.
So what you ship is a detection and annotation helper, not a one-click image rewriter. You give the operator an accurate map of every text region and a clean record of what they want each one to say. Handing that structured output — coordinates plus new strings — to whatever rendering step they already use is a reasonable seam to stop at, and being upfront about it beats promising a seamless rewrite you can't deliver.
Why the boundary is the point
The pattern across a lot of recent open-source releases is the same: builders are getting sharper about splitting a task into the part an LLM does well and the part a deterministic layer must own. You see it in trading systems that give the model proposal rights and keep hard risk checks in Python, and you see it here — let the model detect and read, keep coordinates and edits in a store you control. Copy that discipline, not the marketing. Build the coordinate detector, the numbered canvas, and the keyed store. Draw the pixel-rewrite line honestly, and the tool stays trustworthy.