Learning systems programming was always an aspiration for me. Understanding more about what the machine was doing when you coded something one way as opposed to another. I always felt a vast gulf between toy examples and real programs. I could grasp the concepts quite well, but did not have a good feel for how to compose them. And when I read code for large systems, it was in the how the developer can composed the pieces that I would begin to feel lost and get discouraged. But that feeling that it as all understandable stayed with me. I retained that desire to 'get' the machine better ... in a tangible way.

A few months back I was debugging a personal project that I was building as a learning exercise. It was a small web server written in a low level language. For a few weeks I had forced myself to research what my simple architecture should look like, how it should be modular and appropriate attention paid to separation of concerns. My discipline held on for quite a while, but as the complexity grew I could feel the modularlity slipping and the architecture straining. Anyway, I was having a horrible time with a memory bug. I could grasp that it was to do with the lifetime of piece of data not being appropriate for its role in the server, but I could get to its root cause. I could have solved it in 5 minutes with an LLM, but I was determined to grasp what I was doing wrong.

I had become reasonably proficient with the debugger by this stage, could make my way around a stack trace and had a degree of comfort with manual memory allocation and management. The purpose of this post is to relay the sense of wonder I felt when grasped the root cause of my pain ... a double free. It had been less than 2 days of effort, but I felt I learned more in that debugging exercise than I had in months of reading books and blog posts. It reinforced a belief I have and I believe is commonly held ... that you must blunder through to obtaining a fix of your bug if you really want to internalise the understanding of the cause and advance your knowledge.

But the more profound thing I felt I had learned is that a debugger can be more than a tool to help you get your code working. It can be an amazing laboratory environment. For those of us who enjoy low level coding, debuggers can feel magical in their ability to let you see what is going inside a machine capable of executing billions of operations a second. And this experience got me thinking. Could I build a tool that helps me leverage this 'debugging as learning' approach? So I sat down and worked through as bunch CS:APP, a US college text on systems programming but with a tilt towards the programming. The process turned really fit my brain. I could retain the concepts so much better if wrote toy programs around them, or extended those of others, and then stepped through them with a debugger.

During this process of iteration and learning I used llvm. I use a MacBook and that's the native debugger. This tool is unbelievable, but boy does it have a learning curve. I feel someone could write a book as long as CS:APP and I could spend a year with it and still not know half of the functionality. It was in this process of learning, of trial and erroring and reading, that I formed the genesis of an idea for a tool and a way of learning that I wanted to share. I've spent a couple of months on it now and those interested can get some more information at https://mercurial-hermes.github.io/systems-thinking-on-apple-silicon/. For other like-minded individuals who have been grappling with similar challenges, this could be something of value.