Learning Machine Learning From First Principles

For a while I could get a model to work without being able to explain why it worked. I knew the shape of the workflow — load the data, split it, fit something from a library, check a metric, adjust a hyperparameter, repeat — and that workflow got results often enough that I stopped questioning it. It’s a comfortable place to live. It’s also, I’ve come to think, a fragile one.

The moment that cracked it open for me was small and a little embarrassing. I was tuning a model that wasn’t converging well, and my instinct was to reach for a bigger model, more data, a different optimizer — the usual grab bag. None of it helped, because the actual problem was something basic about how I’d scaled one of the input features. I didn’t catch it for longer than I’d like to admit, because I didn’t have a strong enough mental model of what the training process was doing underneath the library calls to notice that the symptom pointed there. I could operate the machine. I couldn’t picture its insides.

So I went back to first principles, in the literal sense: linear regression, worked out by hand, then a single-layer perceptron, then backpropagation through a tiny two-layer network, all without a machine learning library in sight. Just arrays and loops and calculus I’d mostly forgotten.

Why start this far back

It’s tempting to see this as a waste of time when the field has moved so far past it. Nobody hand-rolls linear regression in production. But the value isn’t in the artifact — it’s in what building it forces you to confront.

When you implement gradient descent yourself, you can’t avoid the questions that library defaults quietly answer for you: why does the learning rate matter this much, what actually happens when it’s too large, why does normalizing your inputs change the shape of the loss surface, what does it even mean for a loss surface to have a “shape.” A library call absorbs all of that into a single line. Writing it yourself puts every one of those decisions back in your hands, which is uncomfortable in the way that useful learning usually is.

There’s also a more specific reason I think starting with the oldest, simplest models matters: almost everything in modern deep learning is a variation on the same small set of ideas — a linear transformation, a nonlinearity, a loss function, and an optimization procedure that nudges parameters to reduce that loss. A transformer is a much more elaborate arrangement of that pattern, with attention and normalization and residual connections layered on top, but the core loop hasn’t changed. If you don’t have a solid feel for the core loop, the elaboration just reads as complexity for its own sake, and it’s easy to memorize the vocabulary — attention heads, embeddings, layer norm — without any real grip on what’s happening mechanically.

What actually surprised me

A few things I expected to understand already turned out to be shakier than I thought.

The first was how much of “training a model” is really just calculus applied mechanically and repeatedly. I’d always treated backpropagation as a kind of black-box algorithm with a scary name. Working through it by hand — computing partial derivatives layer by layer, watching the chain rule do the same multiplication over and over — made it obvious that it’s mechanical in the most literal sense. The chain rule, applied without cleverness, at scale. Once I’d done it for a two-layer network on paper, the idea that a modern framework does the exact same thing automatically, just for a graph with millions of nodes instead of a handful, stopped feeling mysterious.

The second was how much model behavior comes down to the loss function, not the model architecture. I’d been treating architecture choices as the primary lever, probably because that’s what gets discussed the most. But watching the same small network behave completely differently under mean squared error versus a different loss made it clear that the loss function is doing more of the conceptual work than I’d credited it for — it’s the thing that defines what “good” even means to the optimization process.

The third, and the one that changed how I read papers afterward, was noticing how often “the model learned X” is really shorthand for “the optimization process found parameters that reduce the loss in a way that happens to correlate with X.” Those are not the same claim, and conflating them is an easy way to overstate what a model is doing. This isn’t a novel observation — anyone who’s spent real time in the field will say the same thing — but it’s different to have felt it firsthand by watching your own toy network find a lazy, technically-loss-reducing solution that clearly wasn’t the “real” pattern in the data.

The uncomfortable part

None of this makes me faster at building things. If anything, going back to fundamentals slowed me down for a while, because I started noticing gaps I could previously ignore. I’d catch myself mid-project, reaching for a library function, and stop to ask what it was actually computing before I trusted the output. That’s a slower way to work, at least at first.

But it changed what I trust myself to debug. Before, if a model underperformed, my options were mostly “try a different architecture” or “try more data” — reasonable moves, but blunt ones, closer to superstition than diagnosis. Now I have more places to look: is the loss actually decreasing in a way consistent with the learning rate I chose, are the gradients vanishing or exploding somewhere in the network, is the loss function actually measuring the thing I care about. Having a mechanical model of what’s supposed to be happening gives you somewhere to look when it isn’t.

Where this leaves me

I don’t think everyone needs to implement backpropagation from scratch to use machine learning well — plenty of excellent, thoughtful engineering happens at the library level, and there’s no honor in reinventing tools that already work. But I’d underestimated how much of my own confusion, later on, traced back to skipping this step. Concepts that seemed like they belonged to advanced, specialized subfields — regularization, batch normalization, different optimizer variants — turned out to be much easier to actually understand once I had the basic training loop wired into my hands, not just my vocabulary.

I’m still early in this. There’s a long list of things I’ve implemented by hand exactly once and would struggle to reproduce from memory. But the list is growing, and more importantly, my relationship to the library-level tools has changed — I use them the same way, but I no longer feel like a tourist in my own code.