Do we need to understand our systems?
This post was written entirely without AI. It is not perfect and that is fine. If you would rather read something polished made by AI I recommend that you read something else.
I think it started around February. I had been using copilot for several years. I think it made me a better engineer. It did not necessarily solve all my issues but it acted as an easy way to get preliminary information. If I needed more information like docs I could simply go and find it on the internet. Rewind back to February again. I got introduced to Claude Code and how it could be used to explore requirements, write specifications that could reviewed and then have it apply those changes to a new or existing code base. It was by far the best approach to AI code generation I had seen. It provided a way to limit one of the larger issues that I saw with AI generated code: that it was hard to control and get it to do what you wanted. With specifications I could, to some degree, verify that the AI was heading in the right direction before executing. After that I fully embraced AI code generation both professionally and for making my hobby side projects. I even paid for a personal Claude Pro license, something would never had done previously.
That’s almost half a year back and I am starting to think that the focus on the problem being the control of the AI might have been a red herring all along. I went in to a bit of an existential crisis a couple of days ago, which prompted me to write this post. I tried to do some trad-coding (actually writing the code myself. I know, right?) in my side project. It’s in Rust (btw) which I don’t use at work and all my time the last months has been through spec-driven AI code generation. My realization was that: it was super hard. The project had grown from around ~13000 lines of handwritten code to ~20000 lines of AI written code. The progress was sluggish and I found myself not knowing how to write the function signatures any more which I had been comfortable doing with my eyes closed just some months back. It was shocking to me how much I had forgotten in that short amount of time. Mind you that I was still reviewing both the specifications and the code output from the AI before actually committing it to the project. I knew how the product worked, but I could not reason about the code nor the system itself anymore. It had become closer to a black box to me.
I am leaning more and more towards the perception that if you don’t use it you loose it when it comes to software engineering. As such, the more you delegate to AI the more you will become dependent on it and it will erode your understanding and expertise. I thought that I would be able to still keep some of that understanding and expertise by reviewing the output of the AI but I found it very hard to actually think deeply about the code and internalize it. I think writing the code does a lot for the understanding of the code.
That led me to the title of this post: Do we need to understand our systems? For a short answer, I don’t know. What I do think however is that a lot of organizations are currently leaning into the idea that AI is the superior way of producing software. However, I think it is a worse way of building an understanding and ownership of the software you are delivering, and what happens when the organization asks for accountability from the engineers? “I don’t know why it does that, AI wrote it” is not accountability. Is “Let me ask AI why it does that” accountability? I don’t know and it is up to the individual organizations to decide. For more trivial software or one-offs this might be fine. I think it is likely though that the organization that accept that engineers to delegate their understanding to AI might see some issues in the near future. Not necessarily that slop is being produced, but rather because the organization has little understanding and accountability for the software being produced. Also, if we do see these issues, will we even be able to say with certainty that it was due to AI generated code?
As the final part of this post I will conclude with my current thoughts about AI generated code. I think that using AI to write code will in general reduce our understanding of the systems we produce and erode our expertise to the point of being dependent on AI for producing software. I think this might be fine. However, I also think that the accountability and sense of ownership of the software will likely reduce as a consequence. I think that is a much harder idea for organizations to accept.