For investors deciding whether to fund an AI product.
I’m Zac Carlton. I build houses and commercial buildings in Kansas. For about a year I’ve been trying to run my company on AI agents, and I kept a record of where it broke. The failure that worried me most passed the checks I describe below.
n = 1 companyVerified in my test recordsSources and limits below.
14 of 14 · 17 of 17
Checks that passed while the action they should have caught sat off the list.
0 of 176
Commands where one model passed all my scoring checks. A key I wrote by hand passed them on all 176.
22 of 30
Repeated commands where my scoring rules marked the differences between answers as consequential.
01 · What broke
What broke
Three results from my own system. These are one company’s tests, so read them as that. They aren’t claims about AI in general.
01
14 of 14 17 of 17Validator cases and preflight checks that passed
Verified · my agent, my machine
Sources: my claim register, EVIDENCE-CLAIMS-001, B5, and my correction of September 25.
The list that told the inspector where to look
My agent added an action to a test copy of my system and left it out of the declared list. It then ran the completeness check. It came back PASS. The validator passed 14 of 14 cases. The preflight passed 17 of 17 checks. This was a test copy; the protected system was not changed.
The action was there the whole time. The checks read the list to decide what to look for, and it wasn’t on the list.
Inferred · what I take from this
If the list defines the scope, nothing off the list gets checked, and a pass only tells you about the list.
Drawn from my agent’s test on my machine. The rows stand in for the actions on the list.
02
0 of 176Commands where the model passed all my scoring checks. A hand-written key passed them on 176 of 176.
Verified · builder’s evidence
Source: my claim register, EVIDENCE-CLAIMS-001, B1 and B7.
A contract one model didn’t meet
I wrote down the rules my company runs on, with checks to score the model’s answers. I ran 176 commands through one model. It passed all my scoring checks on none of them.
A key I wrote by hand passed all 176 through the same checks, so those checks can be passed. That’s all it shows.
One model, one setup. It says nothing about models in general.
I could see the test set, and I scored the answers with my own checks. No independent reader judged the raw answers. The record does not establish which run came first, whether a model could match the key, whether the prompt was fair, or how it would do on new commands.
03
22 of 30Repeated commands where my scoring rules marked the differences between answers as consequential
Verified · builder’s evidence
Source: my claim register, EVIDENCE-CLAIMS-001, B2-REPEATABILITY.
Same command, different answer
I gave the model the same 30 commands, three times each. It never gave the same structured answer all three times: 0 of 30. On 22 of the 30, my scoring rules marked the differences between answers as consequential.
These counts come from my scoring rules, not a human judgment of the answers.
Agreement isn’t accuracy. A model can repeat itself and still be wrong.
What I don’t claim
I don’t claim a stronger model fixes this, that my setup is typical, or that anyone else’s product fails the same way. I want someone who didn’t build it to check.
02 · Before you invest
Before you sign, ask who checked it.
If you’re shown test results before investing in a company whose AI takes actions, ask where they came from. These are the questions I’d ask first.
01Who ran the tests, and did the company pay them?
02Where did the list of things to test come from? If it came from the product, what isn’t on it?
03Are results on bad or unusual inputs reported apart from results on normal ones?
04Who paid for the evaluation, and does the report say so?
05If you run the same input again, do you get the same answer?
06Can you read the full report, or only a summary and a badge?
Inferred · my comparison as a contractor
I can’t hand a family the keys until an inspector who doesn’t work for me signs off. Who would do that before you fund an AI product?
Call me before your next check goes into a product the builder says works.
I ask these of anyone building agents. I don’t have complete answers to them either.
GuardrailsHow do you know your guardrails are complete?
Test coverageWhen you measure test coverage for an agent, where does the list of things to cover come from?
AuthorityHow do you know your agent only did what it was allowed to?
PaymentsWhen an agent pays for something, how does anyone know it stayed inside what it was allowed to spend?
DataWhen an agent gets into a database through your tools, how does the customer know it only touched what it was supposed to?
PrioritiesIf you had five verification engineers for six months, what would you have them check?
If you have a real answer to any of these, I want to hear it. If the answer is a word for it, I want to learn the word.
04 · The public record
What the public record shows
Two things anyone can look up, and the question each one leaves open.
Verified · source linked
74% of the IT and business leaders Deloitte surveyed expect their companies to use AI agents at least moderately by 2027. 21% say their organizations have a mature governance model for them.
A December 2025 announcement said the Model Context Protocol was being donated to the Linux Foundation’s new Agentic AI Foundation. In February 2026, NIST started an AI Agent Standards Initiative.
My questionDo any of the standards check what an agent actually did?
05 · How I work
How I work
If I’m going to ask other people who checked their work, I owe you the same answer about mine.
Outside readersI pay engineers who check aircraft and spacecraft software for a living to read my work and tell me where it’s weakest. None of them endorsed anything.
LabelsI use verified, inferred and unproven to distinguish what the records show from my inferences and what the records don’t establish. You’ll see those labels on this page.
PredictionsI write my predictions down and score them.
MistakesI keep a log of the mistakes my AI makes working for me, and what caught each one.
PublishingWhen my method is ready, I’ll publish it. I won’t publish a verdict on any named party.
VerifiedI saw it directly in a record or a source I can point to.
InferredMy inference, and I say so.
UnprovenThe record doesn’t establish it.
06 · Corrections
What I’ve taken back
I’ve said things that turned out to be wrong. Here are two, and what holds instead.
A check that defines its own list will always report itself complete.
Wrong. It can fail its own checks for other reasons. What holds is narrower: a pass only tells you about the list.
Taken back September 19, 2026
The model run had zero errors.
It had zero API errors. 173 of its 176 proposals were rejected when they ran.
Corrected September 2026
If you find something wrong on this page, tell me. I’ll fix it here and say what changed.
07 · Contact
Who I want to hear from
Three kinds of people. If you’re one of them, I’d rather hear from you before you decide than after.
Investors
If you’re about to fund a company whose AI takes actions, send me the test results you were shown. I’ll look for what they leave out.
People who check things for a living
Independent verification and validation, aircraft software, testing and evaluation, safety cases. I’ll pay for your time to read one document and tell me where it’s weakest.
People who build agents
If you’ve solved the list problem, I want to see how.
About
I’m a residential developer, builder and designer, and a multifamily and commercial contractor, with more than 200 completed projects. I hold a Class A general contractor license.
I started IMAGINE in Dodge City, Kansas, in January 2026. The questions on this page came out of trying to run my own company on AI.
