Micrograd Learning: Making a Tiny Autograd Engine Easier to Verify
Learning extensions to Andrej Karpathy's micrograd, with input-shape fixes, gradient regression tests, a backpropagation walkthrough, and reproducible command-line training.
- Python
- Autograd
- Backpropagation
- Pytest
- Graphviz
- GitHub Actions
Fixes silent input truncation and single-neuron hidden layers, then checks gradients against PyTorch and exercises a seeded training demo with automated tests.
Contents
The Problem
How can the mechanics of backpropagation become easier to inspect and verify? This project builds on Andrej Karpathy’s micrograd, a tiny scalar autograd engine and neural network library in pure Python. The original engine and neural network implementation are Karpathy’s work; this repository adds correctness fixes, tests, a learning walkthrough, and reproducible experiments while preserving the upstream history and MIT attribution.
Why This Approach?
Scalar operations make the computation graph and chain rule visible. For example, computing at gives and a derivative of . Following that expression through the graph connects the forward calculation, backward pass, and gradient descent update without a tensor framework hiding the intermediate steps.
Architecture
The Value engine records scalar operations and performs reverse-mode automatic differentiation. Neuron, Layer, and MLP compose those values into a neural network. A standalone training script fits a small classifier, while notebooks visualize its decision boundary and draw computation graphs with Graphviz.
The engine has no third-party runtime dependencies. PyTorch supplies reference outputs and gradients for tests rather than powering the engine itself.
Implementation
Two fixes address behavior that can otherwise fail quietly or break a valid architecture:
- Neurons reject mismatched input dimensions with a
ValueError, replacing silent truncation throughzip. - Hidden layers pass lists internally so that a one-neuron hidden layer can feed the next layer. Single-output public calls still return a
Value.
Additional tests cover shared operands, branching, ReLU around zero, gradient accumulation, output shapes, and parameter gradient clearing. A walkthrough connects an expression to its graph, gradients, and an SGD update. GitHub Actions runs the suite on Python 3.10 and 3.13.
Experiments and Results
The seeded command-line demo trains an MLP(2, [16, 16, 1]) on 100 noisy moons samples using hinge loss, L2 regularization, and SGD. The README reports that the default seed and 100 steps reduced loss from 0.895844 to 0.010978 and increased training accuracy from 50% to 100%. It also records 23 passing local tests.
These are results reported by the repository, not independently rerun here. The accuracy measures fit on the training data; a held-out evaluation is needed to assess generalization. Reproducing the numerical results also depends on the dependency environment and platform.
Limitations
The engine operates on individual scalars, with no tensors or GPU acceleration. Recursive graph traversal can reach Python’s recursion limit on deep graphs. Training requires clearing parameter gradients and rebuilding the forward graph for each step. Repeating backward() on the same graph reuses intermediate gradients and differs from PyTorch’s retained-graph behavior.