Understanding the code is understanding the problem

I’ve been playing with LLMs for years now, and I still write 90% of the code in my side projects by hand. The models are impressive, but they still have a fundamental problem: they fill in too many gaps.

Something I’ve noticed again and again once I let an LLM loose on a project is that it will start to make assumptions, and bake those assumptions into the code as if they were intentionally considered and decided upon.

I saw this happen in large shared codebases even before LLMs. The larger the team, the less each team member understood the reasoning behind each line of code. In general, when approaching some unfamiliar code, we tend to assume “this code is this way for a reason”. So when a shortcut is taken just to get something working, a year or two later somebody might stumble across the code and assume it should be that way, or that it was the ideal way to do things.

This effect seems to compound with LLMs. In one session the agent decides to make a bunch of assumptions and bake them into the codebase, in the next session it reads that same code and assumes it must be that way for a reason, building on top of the (often mistaken) assumption. The larger and deeper these assumptions are baked into the code, the harder and more risky they become to change - even for an LLM.

So to combat that, the human reviewer is introduced into the loop. Review each line of code and look for those assumptions, prompting the LLM to correct them. But to do this, the human must understand the code.

I’ve long subscribed to the idea that the code is the true specification. It’s the only place where every behaviour and edge-case that can be handled, must be described precisely. But importantly, it’s through writing the code that I often discover edge cases. In fact, in my opinion that may be the most important part of software engineering.

When a product manager says “we need this feature”, there is often something neither they nor I have considered until I start actually writing the code. Then it becomes clear that X can’t be changed because it would affect Y. So we need to go back to the drawing board and consider the feature in a different way, or alter the system. An LLM however will not raise this as an issue and will instead just go ahead and change things, bulldozing ahead while making new assumptions and breaking old assumptions to achieve its goal.

One way around this is to write very precise prompts after looking at the code (while hoping the context window doesn’t lead to the LLM forgetting important details). At that stage though, you’re effectively just writing the code in imprecise language.

Personally, I find that writing the code helps me understand the problem itself. I can look at the code and start to see where the constraints are, and where I’m introducing new ones while I type - giving me a chance to reconsider the problem before going too far. Code is a far more precise language for this than English.

I still find LLMs useful for autocomplete (where I already know what I’m writing), for generating boilerplatey code (which is best avoided where possible anyway), and for simple refactors.

The real challenges of building software are not found in high level explanations of the problem, but in the details.