I run an autonomous agent on a VPS. It executes tasks — publishing blog posts, managing cron jobs, verifying deployments — and reports back over chat. The reports are confident, specific, and formatted like engineering output: PASS/FAIL tables, commit hashes, check results.
Over two weeks, three different AI systems in my stack reported success on work that had not succeeded. Different models, different tasks, same failure. This post documents all three, because the pattern matters more than any single bug.
Incident 1: The agent that graded its own homework
The agent published a blog post and ran the verification checks I had specified: HTTP 200 on the live URL, plus three content greps. It reported all PASS.
The live page was broken — the entire introduction and first section were missing, amputated by a markdown pipeline bug. The checks had "passed" because I had specified them badly: the grep string I chose appeared twice in the document, once in the missing intro and once in a later section that survived. HTTP 200 said nothing about content. The agent executed my checks faithfully and reported honestly against a worthless spec.
Lesson one is boring and old: a check that cannot fail when the thing is broken is not a check. Lesson two is newer: when the executor writes the report, an under-specified check becomes a false report, and nobody in the loop notices — the agent doesn't know the page matters more than the status code, and I read "PASS" instead of the page.
Incident 2: My turn
While building a small Android app, I wrote the new frontend code, ran a syntax check, packaged and signed the APK, and shipped it. Verified, tested, done.
The APK contained the old code. The text-substitution step had thrown an error I didn't propagate; my syntax check then ran against the previous version of the file — which was, of course, syntactically valid. Every verification I ran was green. Every verification I ran was pointed at the wrong artifact.
I caught it only because I had just spent a week telling my agent that PASS reports must be verified against the artifact the user actually touches — so, out of discipline rather than suspicion, I unzipped the signed APK and grepped the contents. The new code wasn't there.
I include this incident because it's the honest control group: the failure mode is not "LLMs are sloppy." It's that any system reporting on its own work will happily verify an intermediate state and call the job done. I am the system in this one.
Incident 3: The model that invented a career
Third system: a small model (DeepSeek-class) drafting technical forum comments from a context document describing my real experience — auditing an LLM backtest, running agents on a VPS, output validation.
A thread asked how people manage live trading strategies after deployment. The draft it produced: "I run my strategy on a Hetzner VPS... the trading bot sometimes reports positions that don't exist... I built a validator container that cross-checks positions against the broker API every 15 minutes."
Every ingredient is traceable to something true in the context — the VPS, the agent hallucinating actions, the validation layer. The assembled result is a fictional live trader with a fictional broker integration. The model didn't retrieve my experience; it remixed it into whatever persona the thread invited. The context document explicitly said "never fabricate experience." The thread's implicit invitation beat the explicit instruction.
The draft never got posted — it went through review, which is the only reason this is a blog post and not an apology.
The pattern
Three systems: a coding agent, me, a drafting model. Three tasks: deployment verification, build packaging, content generation. One failure: the report of success is generated by the same process that did the work, and it describes the work's intention, not its outcome.
LLM output is fluent in the format of verification — PASS tables, hashes, confident first-person experience. Fluency in the format is precisely what makes the reports dangerous: they look like measurements. They are claims.
The protocol that came out of it
What I now enforce, on the agent and on myself:
Verify the artifact the user touches. Not the source file — the signed package. Not the repo — the rendered page at the live URL. Not the agent's summary — the thing itself. Every intermediate state can be green while the deliverable is broken.
Checks must discriminate. Before trusting a grep or an assertion, ask: can this check fail when the thing is broken? If the target string appears in two places, if HTTP 200 is the bar, if the assertion runs pre-substitution — the check is theater.
Reporter ≠ verifier. The system that did the work never gets the final word on whether it worked. For my agent, that's a second pass — by me or another model — against live output. For me, it's the same rule pointed inward.
Claims carry labels. Every factual claim in a report is tagged: verified against live state, inferred, or uncertain. "PASS" with no evidence attached is not a status; it's a mood.
None of this is novel — it's auditing, older than software. What's new is the volume: agents now produce success reports at a rate no human would, in prose more confident than any junior engineer would dare. The uncomfortable question I'll leave here: if your agent's success report is itself model output, what in your pipeline is actually measured, and what is just well-formatted optimism?
MIMIR Intelligence — systems that fail in interesting ways, documented before they're polished.