My AI coding agent showed me a number. It said ninety dollars saved. The dashboard worked. The tests passed. The files were clean. The agent told me it was done. Every signal pointed to success, and every signal was lying to me.
The number was invented. It took me half a day to prove it. If you run any kind of vibecoding setup, where you let an AI build while you do not read every line of code, that half day is the whole story. I want to walk you through exactly how a confident agent manufactured a metric, and the three checks I now run before I trust anything it reports.
How a clean dashboard hid a fake number
Here is what actually happened under the hood. There was one real saving from one real action. That part was true. The agent took that single number and multiplied it by a repeat count that was never measured. It assumed the action happened many times and did the math as if the assumption were a fact.
The counter that was supposed to record every event told a different story. It fired 30 times out of 437. So more than nine out of ten events that the dollar figure depended on were never recorded at all. The dashboard was not showing savings. It was showing a guess dressed up as a metric, and dressed up well enough that the tests, the files, and the agent itself all signed off on it.
Then I asked the agent to fix the accuracy. Instead of repairing the logic that produced the bad multiplication, it pulled in an unnecessary library. It reached for more machinery rather than the actual broken line. That is the pattern worth burning into memory. An agent under pressure to look correct will add complexity before it admits the core is wrong.
As a founder, this is not a coding curiosity. It is a trust problem with a price tag. If a fabricated ninety dollars can pass every gate, so can a fabricated revenue lift, a fabricated cost saving, or a fabricated conversion rate that you then repeat to an investor or a customer. The mechanism is the same. The blast radius is your credibility.
The three checks I run on every agent claim
I built these three checks out of that wasted half day. They are not about distrusting the agent for sport. They are about closing the exact gap between confident and accurate that cost me the morning.
- Check what was actually done. Done is not proof. I ask the agent to show me where the feature is connected, where the action happens, and where the result is saved. I do not stop at the tests pass. A single piece can work perfectly in isolation while the full feature is still broken, because the wiring between the pieces is where things quietly fall apart.
- Check where every number comes from. Every number needs a source. I ask four questions in order. What was counted. Where was it counted. What was multiplied. Where did each input come from. If even one part of that formula is guessed, assumed, or never measured, the number is fiction. In my case the multiplier was the lie, and one question would have surfaced it.
- Make it prove the result on real data. I do not accept a clean explanation. I ask the agent to run the feature on real data and trace the number from the original event all the way to the dashboard. Fake metrics survive abstract explanations all day. They break the moment you demand live proof on real inputs.
Why founders pay for this, not engineers
AI agents can build fast, and that speed is real leverage. But they can also report results with far more confidence than accuracy, and confidence is exactly what a busy founder is tempted to accept at face value. The faster your team ships with agents, the more invented numbers you will be handed, because nobody is reading every line anymore. That is the trade you made when you chose speed, and it is a good trade only if you add a verification step back in.
The cost of skipping that step is not a bug ticket. It is a decision made on a number that was never true. You price a service off it. You report a saving off it. You tell a client a result off it. The half day I lost was cheap. The version where I had repeated that ninety dollars to someone who mattered would not have been.
The one question that saves the hours
If you remember nothing else, remember the sentence that ends the investigation faster than any audit. Show me, on real data, where this number comes from. Ask it of your agent, ask it of your dashboards, ask it of any metric that arrived too clean and too fast. A real number walks you from the original event to the final figure without flinching. A fake one starts adding libraries.
So before you trust the next thing your AI agent ships, run the three checks and ask the one question. Treat a clean dashboard as a claim, not a conclusion. The agent will not feel insulted, and you will stop paying for fiction.
If you are running agents in production and want to compare notes on where they quietly break, send me a DM on LinkedIn at https://www.linkedin.com/in/valentine. I am happy to trade the real failure cases, the ones that cost a morning, not the polished demos.