We’ve been told for years that AI will write our code. That developers should embrace AI assistants, accept AI-generated PRs, and trust that the machine knows better. The narrative is seductive: why spend hours writing something a machine can generate in seconds?
But there’s a fundamental problem with this thinking—one that becomes obvious when you look at what actually happens when AI tries to write serious code. The answer isn’t AI gets it wrong. The answer is more subtle and more troubling: AI generates code that looks right without being right.
The AI Confidence Problem
When an AI writes code, it produces something that appears correct. The syntax is valid. The logic seems sound. Tests might even pass. But the deeper truth is harder to verify: Did the AI understand what it was building? Does the code actually solve the problem it claims to solve? Can you trust it when the edge cases hit?
This isn’t a capability issue. AI models are remarkably good at pattern matching. They can produce syntactically correct code in nearly any language. The problem is that correct-looking code and correct code are not the same thing—and the difference only emerges when something goes wrong.
pResearch from CodeRabbit analyzing 470 GitHub pull requests found that AI-generated code contains 1.7 times more issues than human-written code. Logic errors appear 1.75 times more frequently, and security vulnerabilities increase by 1.57 times. The LinearB 2026 study of 8.1 million pull requests found that production incidents surged 23.5% year-over-year at organizations using AI coding assistants.
66% of developers now deal with “AI solutions that are almost right, but not quite” according to Stack Overflow’s 2025 survey. This is the real cost—not obvious failures, but subtle bugs that slip through.
What Happens When AI Meets Strict Contracts
I ran an experiment recently. I took a real task—a simple order total calculation with discounts and tax—and asked a capable AI assistant to write it in Prove, an intent-first programming language where you must declare your intent before writing code.
The results were revealing. The AI failed six times before producing something close to correct, and each failure exposed a different category of bugs that Prove was specifically designed to prevent:
- Missing intent declarations — The AI used generic function keywords instead of declaring what the function actually does (pure transformation, validation, IO operation)
- Wrong control flow — The AI reached for `if` guards instead of contract-based validation
- Documentation drift — The AI wrote documentation that didn’t match what the code actually did
- Silent assumptions — The AI skipped verification on dependencies, creating gaps in the trust chain
Every one of these is a real production bug in any codebase. Not just AI-generated code—human code too. The difference is that Prove treats all of these as compilation errors. The AI couldn’t cheat its way past them.
The Deeper Problem: Understanding vs. Pattern Matching
Here’s what strikes me most about the AI failures: they didn’t happen because the AI cant pattern match them. They happened because understanding and pattern matching are fundamentally different operations.
When a human writes code, they’re making a series of decisions:
- What does this function need to guarantee?
- What can go wrong, and where should that be caught?
- What does the next person need to know?
When an AI generates code, it’s doing something else entirely—matching patterns from its training data against the prompt. It can produce something that looks like it made those decisions, but it can’t actually make them. It doesn’t know why a contract matters. It doesn’t understand why “validate the inputs” should be a compile-time check rather than a runtime guard.
This is why AI-generated code tends to work in happy-path scenarios and fail on edge cases. The patterns it learned from training data were happy paths. Edge cases require understanding, not matching.
Why Human Authorship Matters
We don’t write code just to get machines to execute it. We write code to communicate intent to other humans—maintainers, reviewers, future-you. The code is as much a document of reasoning as it is a set of instructions.
When code is generated by AI, that reasoning is absent. Not just “not visible”—actually absent. The AI didn’t think about why the code should work a certain way. It just produced something that matches what similar code looked like.
This matters because:
- Maintenance: Someone will need to understand this code in six months. AI-generated code often looks like it was written by someone who didn’t understand the problem.
- Trust: Can you ship code you didn’t write? Not in the sense of “who’s responsible”—in the sense of “do you trust this actually works?”
- Learning: Developers grow by writing code. When AI writes the hard parts, developers don’t learn.
The Alternative: Intent-First Design
[Prove](https://prove.botwork.se) represents a different path. Rather than trying to detect AI-generated code and reject it, they build languages where **the act of writing code correctly requires understanding**.
In Prove, you can’t just write a function. You must declare:
- What kind of operation it is (verb system)
- What it guarantees (contracts)
- Why it works (explain blocks)
This isn’t about being hostile to AI. It’s about building languages where correctness and understanding are inseparable. If you can write correct Prove code, you had to understand what you were writing. If you didn’t understand it, the code won’t compile.
Prove is also naturally resistant to AI code generation—not through filters, but through design. Generating semantically correct Prove requires genuine understanding, not pattern matching.
What We’re Building
At Botwork, we believe in human authorship. Not because we’re luddites—but because code written by humans who understand what they’re building is:
- More reliable
- More maintainable
- More trustworthy
AI can be a tool for exploration, for scaffolding, for handling the boilerplate. But the core logic—the parts that matter—should come from human minds that understand the problem.
This is why we’re building Prove. Not to reject AI, but to create a space where human understanding is mandatory. Where the compiler enforces that someone actually thought about what the code should do.
If you can write correct code, you understood the problem. That’s what matters.
Leave a Reply