Autograd: Follow Every Path Back to a Parameter

Predict a shared gradient

At x=3, let u=x*x and L=u+u.

Predict u.data, L.data, and x.grad before implementing backward.

Forward creates a graph

u=9, L=18. Both references in u+u point to one object.

A node owns a value, parent references, a gradient accumulator, and a local backward rule. The forward pass creates new operation nodes.

Trace the shared graph

Seed L.grad=1.

L=u+u adds 1 twice: u.grad=2.

u=x*x adds 3*2 twice: x.grad=12.

A parent set deduplicates visits; it must not erase operand roles.

Choose an execution order

Visit parents, then append each node once.

Forward order: x, u, L. Backward order: L, u, x.

A node propagates only after every consumer has contributed. Explain why immediate recursion along each edge can mishandle a shared intermediate.

Separate two kinds of accumulation

Fresh graphs sharing parameter leaves can accumulate leaf gradients.

Repeating backward on the same graph leaves stale intermediate gradients in this engine.

For a new step: clear leaves, build a fresh graph, backpropagate once, then update.

Use an independent oracle

For f(x)=x*x+x, a central difference near x=3 is 7.

Compare with autograd on a fresh graph.

Avoid ReLU at exactly zero; its derivative is not unique there. Use math.tanh for the forward value at large magnitudes.

Reconstruct the engine

Implement arithmetic nodes, then topological traversal, then activations.

Explain all three: why +=, why loss.grad=1, why parameter data stays fixed until backward ends.

Ready when x*x, u+u, and a finite-difference check all agree.