Autograd: Follow Every Path Back to a Parameter

A language model eventually produces one scalar loss from thousands of scalar operations. We want the derivative of that loss with respect to every parameter, without deriving the whole network by hand. Reverse-mode automatic differentiation records what happened in the forward pass, then distributes one incoming derivative backward through each operation.

You need the chain rule, Python objects, and the distinction between a number and a reference to an object. By the end, you should be able to trace a shared computation, explain why traversal order matters, and check an implementation against a numerical derivative. The exercises divide the work into autograd-value (arithmetic and graph construction) and autograd-backward (traversal and nonlinear operations).

Choose an entry point and check readiness

Start here directly if you can trace a derivative through two operations and read (n,d) @ (d,h). You do not need to finish DL v1 first: v1 is an optional conceptual bridge; v2 is the cumulative implementation route. The code is scalar Python throughout, so a tensor-framework installation is not required.

If those prerequisites are uncertain, use these concrete checks before proceeding:

Try without notes	Expected evidence	Review when stuck
Differentiate (2*x+1)^2 at x=1	The local derivative 6 receives a factor 2, giving 12	ML calculus
Predict (4,2) @ (2,3) + (3,)	Shape (4,3); bias is reused for four examples	ML tensors
Explain logits [1000,1001]	Subtract one common maximum before exp; normalize the row	ML classification

In ML's batch formulas, W has shape (input,output) and multiplies on the right. This track's Linear.weight stores (output,input) so each output neuron owns a row. Both compute the same dot products; translate the storage convention, not the learning rule.

The route is graph derivatives → module ownership → token windows → shared embeddings → a bigram baseline → a fixed-window MLP → attention → transformer → capstone. Each stage should earn a behavioral check before adding capacity. The D2L preliminaries offer a longer math/array refresher when the brief ML examples are insufficient.

A node stores a value and a local rule

A Value contains data, a gradient accumulator grad, parent references _prev, and a _backward closure. A leaf is an input or parameter. An operation creates a new node; it does not overwrite its inputs.

For out = a * b, the forward result is a.data * b.data. Once out.grad contains dL/dout, the chain rule sends these contributions:

a.grad += b.data * out.grad
b.grad += a.data * out.grad

The closure captures the input objects and output object. Do not change parameter data between the forward and backward passes: these simple closures read those inputs during backward. Read .data to calculate an operation's numeric forward result inside Value; model code should compose Value operations so the graph remains connected.

Work a graph before writing its traversal

Let a = 2, b = -3, u = a*b, and L = u+a.

Node	Forward value	Incoming derivative	Contributions sent backward
L = u+a	-4	1	u.grad += 1, a.grad += 1
u = a*b	-6	1	a.grad += -3, b.grad += 2
a	2	-2 total	Leaf: stop
b	-3	2 total	Leaf: stop

There are two paths from a to the loss. Replacing += with = loses one path. The negative gradient for a means a small increase in a decreases this particular loss; gradient descent therefore increases a here.

Now predict the derivative of x*x at x=3. It is 6. Both argument positions refer to the same object, so multiplication must add two contributions of 3. _prev can be a set containing x once: the set controls visits, while the operation's closure preserves both argument roles.

Why reverse topological order works

Build an order by visiting every node's parents before appending the node itself. Use object identity to visit each node once. For the graph above, one valid order is [a, b, u, L]; reverse it for backward.

visit(node):
    if node was visited: return
    mark node visited
    for each parent: visit(parent)
    append node

visit(loss)
loss.grad = 1
for node in reverse(order):
    node._backward()

A node's consumers all run before that node, so its accumulated gradient is complete when its closure runs. Recursing backward immediately along each edge is insufficient for shared intermediate nodes: they may propagate an incomplete gradient, or propagate the same contribution repeatedly. The seed 1 is dL/dL, not a learning rate.

Local derivatives and domain choices

Operation	Derivative with respect to input x	Contract
x**n	n*x**(n-1)	Constant numeric exponent; use a valid real domain
tanh(x)	1-tanh(x)**2	Compute forward with math.tanh to avoid overflowing an exponential formula
relu(x)	1 for positive x, otherwise 0	Choose derivative 0 at the kink
exp(x)	exp(x)	Extremely large positive inputs can overflow
log(x)	1/x	Raise ValueError for x <= 0

Implement subtraction and division by composing negation, addition, multiplication, and reciprocal powers. Reverse operators matter: 2-x and 2/x must preserve operand order. We will avoid overflow in probability calculations by changing the formula, not by pretending exp supports arbitrary inputs.

What accumulation does and does not mean

Before an independent training step, zero each parameter's gradient, create a fresh forward graph, backpropagate once, then update parameter data. Fresh intermediate nodes begin with zero gradients.

For intentional accumulation, create separate fresh graphs sharing the same parameter leaves and do not clear the leaf gradients between them. Scale each loss consistently if you want a mean. Calling backward() twice on the same graph is not supported as clean accumulation by this minimal engine: old intermediate gradients remain. Clearing only parameter gradients does not reset that graph.

Check the derivative independently

For f(x)=x*x+x, f'(3)=7. A central difference is:

h = 1e-5
numeric = (f(3 + h) - f(3 - h)) / (2 * h)

Compare it to the autograd result on a fresh graph. Repeat for a shared intermediate such as u=x*x; L=u+u, whose derivative at 3 is 12. Avoid testing a central difference exactly at ReLU's nondifferentiable zero.

Checkpoint. If u=x*x, L=u*u, and x=2, what should reach u before its closure runs?

Check your reasoning

u=4, L=16, and u.grad=8; the two multiplication roles each contribute 4. Then x.grad=8*(2+2)=32. This distinguishes summing paths from merely visiting every node.

The next lesson gives parameters an owner, so a training loop can find and update each leaf once. For a compact implementation of the graph/closure design, see micrograd's scalar engine.