Stop Asking a Vision Model to Check Your Own Renders

Your vision model approved a broken render. A string comparison caught it.

Here's the setup. An automated pipeline generates text-heavy visuals: quote cards, report pages, slides. To QA them, the obvious move is to screenshot the output and ask a multimodal model, "is this good?"

In one build, a stray negative margin pushed the attribution line on top of the quote. The vision model read the overlapping text perfectly, attribution and all, and flagged nothing. It reads mangled text too well to notice that a human couldn't.

A dumb deterministic check, comparing the words on the render against the words we meant to put there, dropped to 63% coverage. Caught it instantly.

The mistake wasn't a weak model. It was asking for an opinion when we already held the truth. We wrote the source text and we owned the markup. Handing the model a picture turned an exact verification problem into a vibe check.

The fix is to give the model the structure instead of the pixels. Render the asset in a real browser, pull the resolved DOM or accessibility tree, and strip the wrapper soup and styling clutter. What's left is a compact skeleton: the words, their order, their roles (headline, quote, byline), and the emphasis that survived rendering.

On that skeleton you ask two cheap, exact questions before any model gets a say. Did every intended word survive, in order? Did anything foreign creep in, or get clipped out? Only after a card passes do you let a model judge the part it's actually good at: whether the thing is worth saying.

The pixels are the most faithful representation of a visual, but they're often the worst one to reason over. Faithfulness is not usefulness.

This has limits. A skeleton won't tell you the colours clash or the photo is ugly. But for text-bearing assets where you already know what should be there, a smarter model is worse engineering than a string match. It's also why I'll take a generation path whose output I can flatten back and verify over one that's slightly prettier and opaque. When something runs a thousand times unattended, you need the version you can check.

If your pipeline produces or judges visuals, find the one place you hand a model an image of your own output. Flatten one render and check its words against the source.

Tell me what it caught. I read every reply.