It was just a simple extension. Not even 500 lines of code. And still, the AI couldn't do it.
The AI couldn't build an extension that colors a button based on one condition.
It couldn't because the scenario wasn't defined from the start. I had a list of followers, which I downloaded and handed over. In principle nothing was missing, so the AI should have gotten it right the first time.
All I cared about was the people from the last week or so. I wanted to unfollow everyone who hadn't followed me back. I had to iterate on the code with the AI. The first fix was requiring the following list as well. That didn't work either. The AI's proposed solution was to export the data again, covering the whole period. That is, going from a 7-day export to a 5-year one. I noticed the followers list didn't reach nearly as far back as the following one. So I thought of only analyzing from the day set by the oldest account in the followers list. The AI agreed, but it didn't notice that the same problem existed at the other end (it had the information needed to reach that conclusion).
Then it built it, and it worked. I didn't touch a single variable. I didn't even read the code. This is a very simple extension, and that's what got me thinking... that the AI failed to make it.
And it's true: it did what I asked. What I asked just wasn't enough. You could argue that's my fault, that I didn't describe the requirements correctly. But that's exactly the point: I couldn't have described them "correctly." I could never have known in advance that the followers list started later and ended earlier. The AI's solution? Export the full list. Go from 7 days to 5 years. The AI usually proposes these brute-force solutions. Which do work, as far as that goes. In an ideal world they're maybe the right ones. But in the real world there are constraints on money, time and resources, and they're usually inflexible. Those constraints usually create problems. In this case it was a timestamp that stopped too soon. Believing this class of problem can be removed is, I think, naive. There's also this: the problems that constraints create are usually unforeseen. Did it say anywhere that the following list isn't limited by time? Maybe, but neither I nor the AI knew about it.
And there's the key point: if they can't be foreseen, they can't be accounted for in the plan.
Which means you could never have handed an agent a plan covering every error. You could then argue that the agent can discover the errors itself. Which might have been possible if I'd had it test the extension for real. There's a limit there, because often that isn't possible at all. In this case it would have required access that risked getting me banned.
But even if it were possible, the next question is whether the agent eventually sees what's going on. I think they currently have very good problem-solving capabilities. The agent did propose a solution, even if I was the one who found the problem: export the entire period. I considered that a bad solution because it would have meant far more waiting. Maybe I could have told it during planning to prioritize speed. Though that's ambiguous. The agent could reasonably conclude that this time there was no way around exporting everything. Which, by the way, I was the one who had to do. Time and resource constraints are real and often inflexible (in whether they have to be met), but at the same time they're ironically somewhat ambiguous and context-dependent. In this case I didn't want to export all the data because it would take hours, so I somehow "sensed" there had to be another way to use what I had. That turned out to be adding another file, and then a from and an until condition.
So the question is, how could the agent have gotten there? If my requirements had been "make it accept a followers file," it wouldn't have thought to add another file, and the only solution would genuinely have been exporting the whole period.
Hard, highly detailed requirements block creative solutions (or not so creative ones) that break the mold the original plan defined.
So we can go to the other extreme: just tell it what I want. But in my experience AI is very bad at intuitive solutions that actually work. It makes very good UI, but the solutions it proposes tend to... not really solve the problem. Or at least not in a way that's comfortable and intuitive to use. I told it the problem first, and it suggested a page that dumps the information. It's clear to anyone who thinks about the problem for a minute that this wouldn't help at all. I don't want a dashboard. But if the AI's solution is no good, then I have to propose a solution and have it implement that. And there we're back to the previous dilemma.
The problem, then, is that there are variables that act as obstacles in developing a solution (in software, though really in anything). These variables, though very real, are usually hard to describe. They're hard because the way to judge whether something is reasonable is through a kind of intuition, or "common sense." What makes this difficult is that there isn't really a correct answer. It's not "wrong" to export the entire period of data, but... I'd really rather not.
There'd be more to work through, but my premature conclusions would be... In theory an AI agent can, at this point, build any software to solve any problem autonomously, given enough time and money (tokens). But in practice, it can't solve almost any problem autonomously. And the reason is that problems, beyond requiring solutions that are actually useful (which AI doesn't usually manage at a genuinely useful level, though you could argue that's incompetence on my part), tend to be surrounded by constraints on money and resources, or on other things entirely (moral ones, preferences, etc). These constraints are hard to convey to an agent because, while they aren't optional, they're flexible in a sense. How much time or money is acceptable to spend is often just "common sense."
So what would I take from this? Reasoning and a certain capacity for analysis are still necessary. A solution like building an extension probably wouldn't occur to someone without software experience. It didn't occur to the AI. It might have, by pushing it to produce different solutions, but there's a human judgment involved in knowing how far to go.
But if I wanted to draw a lesson... I don't know. I don't think I get anything actionable, or anything that changes how I work.
I think it partly confirms that developing solutions to a problem is still a task where the human is essential. And yes: this is a simple 500-line extension. The repo took 2 minutes. But this problem only gets bigger as the solutions get more complex.
Because it's not about building software. Software is just the shape we're giving to the attempt to solve the problem in front of us.
It's never about building one specific piece of software. If it were, AI could maybe work with no human anywhere in the process.
The reason we build software is to solve problems. And once we start looking for solutions to problems, inescapable constraints show up that affect those solutions, and whose limit usually can't be written into the plan.