Two Bugs AI Wrote That I Had to Find Myself

I build products solo and AI writes most of the code. That works well enough that the interesting question stopped being whether the code compiles. The interesting question is which bugs survive.

The two that cost me the most time on Stips, a play-money prediction market I designed and built, were both invisible to the model that wrote them. Neither was a syntax error. Neither would have been caught by a test the model could have written for itself.

The Model Has No Clock

Stips generates markets from news on a schedule. A model reads what happened, writes a question people can take a position on, and sets a close date.

The close dates came out already expired.

Not off by a day. Months in the past. The market would land in the database and be dead on arrival, because a model's sense of "now" is whatever felt current in its training data. Ask it to close a market in two weeks and it does the arithmetic honestly, from a starting point that is nowhere near today.

What made this expensive is that everything else about the output was right. Valid JSON. Sensible question. Clean resolution criteria. Reasoning that read like a person who had actually understood the news story. One field was garbage, and it happened to be the field that decided whether any of the rest was usable.

The fix is two rules I now apply by reflex. Today's date goes into the prompt engineering for designers as a stated fact. The model never has to infer it. And no generated date is trusted until it has been compared against real system time. If a close date isn't in the future, the market never gets written.

Time is the obvious version of this bug. It isn't the only one. Anything the model can't observe, a current price, who holds an office, whether a service still exists, what your schema looks like today, gets produced anyway, confidently, in exactly the right shape. Shape is not truth. And the model has no way to flag which of its outputs it actually knows.

Row-Level Security Doesn't Throw, It Filters

Second bug, different flavor. Stips runs on Postgres with row-level security. I did most of my testing signed out, because signing in was slower and I was moving fast.

Signed out, every authenticated query returned an empty list. That looks exactly like a feature nobody has put data into yet. It does not look like a permission problem, because there is no problem to see. RLS doesn't reject the read. It returns the rows you're allowed to see, and when you're allowed to see none, it returns none. Zero rows, zero errors, no stack trace.

So the app rendered empty states everywhere and I read those empty states as "nothing here yet." Every policy mistake, every query missing a user id, every place a session wasn't being passed through was sitting right there in plain sight. They surfaced the moment I tested as a real signed-in user, and they surfaced all at once.

Signed-out testing is a permanent blind spot on anything with row-level security. I test signed in first now. Guest is the special case.

What Review Actually Has to Look For

Reading generated code line by line for correctness is mostly wasted attention. It's usually correct. The failures live in what it assumed and had no way to verify.

The questions I run on anything a model wrote:

• Where does this get the current time, and is that source real?
• What does this return when the caller has no permission? An error, or silence?
• Which values here came from the world, and which came from the model's memory of the world?
• What happens on the second run, not the first?

None of those are questions about code quality. They're questions about the boundary between the model and reality, which is exactly where a model is blind by construction.

Writing Got Faster. Verifying Didn't.

The assumption I see people make is that because generation got cheap, checking got cheap with it. It didn't. Writing the market generator took an afternoon. Trusting the market generator took another week, and that week was almost entirely me building the checks that would tell me when it was wrong.

That ratio is the actual shape of the work now. Less typing. The same amount of judgment, concentrated into fewer decisions that matter more.

Both of these came out of building the Stips case study.