From Logistic Regression to a Neural Network You Can Check

This capstone turns the chain rule into a working two-layer classifier. You will build the neural-network-xor exercise with Python lists, explain why its hidden layer helps, and check gradients before relying on a training curve.

Use the earlier matrix multiplication, sigmoid, binary cross-entropy, and gradient descent lessons. If a matrix shape is unclear, resolve it before coding backpropagation. Our running example has two inputs, two hidden units, and one output; the training exercise normally uses four hidden units.

What XOR proves about a linear boundary

input     label
[0,0]       0
[0,1]       1
[1,0]       1
[1,1]       0

With the prediction rule score >= 0 -> 1, correct logistic classification on raw inputs would require:

b < 0                     for [0,0]
w2 + b >= 0               for [0,1]
w1 + b >= 0               for [1,0]
w1 + w2 + b < 0           for [1,1]

Adding the middle inequalities gives w1+w2+b >= -b > 0, contradicting the last. No single affine boundary classifies all four correctly.

That does not mean every linear classifier is limited to 50% accuracy. Score x1+x2-0.5 correctly labels three of the four points, giving 75%. Minimizing logistic loss on the symmetric dataset and maximizing classification accuracy are different objectives. Also, adding an engineered interaction feature changes the model class: x1+x2-2*x1*x2 already computes XOR on binary inputs.

A hidden layer learns new features rather than requiring us to supply that interaction. This expressiveness argument and the optimization question “will this training run find suitable weights?” must be checked separately.

See a hidden representation before training it

Let s=x1+x2. Two sigmoid units can approximately detect “at least one” and “both”:

h1 = sigmoid(10*(s-0.5))
h2 = sigmoid(10*(s-1.5))
output = sigmoid(10*(h1-h2-0.5))

Input sum	h1, approximately	h2, approximately	Output class
0	0.0067	0.0000003	0
1	0.9933	0.0067	1
2	0.9999997	0.9933	0

These are hand-chosen weights proving that the architecture can express XOR. They are not the exercise's initialization or a claim about what every hidden neuron must learn.

Without a nonlinear hidden activation, the composition collapses:

(X @ W1 + b1) @ W2 + b2
= X @ (W1 @ W2) + b1 @ W2 + b2

It is still an affine score in the original inputs. A final sigmoid changes the score to a probability but does not make its threshold boundary nonlinear.

Keep one shape convention throughout

Rows are examples. Use X @ W1, not W1 @ X. For batch size m, input width d, hidden width h, and output width o:

Object	Shape	Running example
X	(m,d)	(m,2)
W1, b1	(d,h), (h,)	(2,2), (2,)
z1, hidden activations H	(m,h)	(m,2)
W2, b2	(h,o), (o,)	(2,1), (1,)
z2, predictions P, targets Y	(m,o)	(m,1)

The starter names the hidden activations h; here H keeps them distinct from the width. add_bias adds the bias vector to each row. Build and check the list helpers matmul, transpose, and add_bias first.

z1 = add_bias(matmul(X, W1), b1)
H = sigmoid(z1)
z2 = add_bias(matmul(H, W2), b2)
P = sigmoid(z2)
# forward returns (P, H, z1)

For scalar sigmoid, evaluating exp(-z) directly can overflow for very negative z. A stable branch uses 1/(1+exp(-z)) when z>=0, and exp(z)/(1+exp(z)) otherwise. Apply it elementwise to a matrix. The derivative is sigmoid(z)*(1-sigmoid(z)).

Derive a backward pass from its loss

The exercise loss sums output-column losses and averages over examples:

L = -(1/m) * sum_i sum_j [Y[i,j]*log(P[i,j])
                         +(1-Y[i,j])*log(1-P[i,j])]

XOR has one output, but this convention also specifies what the helpers mean with multiple independent binary outputs. Do not silently divide by the number of output columns.

For one target y and probability p=sigmoid(z), combine the two derivatives:

dL/dp = -y/p + (1-y)/(1-p)
dp/dz = p*(1-p)
dL/dz = p-y

Define the unaveraged output errors E2=P-Y, then average each parameter gradient exactly once:

dW2 = H.T @ E2 / m               shape (h,o)
db2 = column_sums(E2) / m        shape (o,)
E1  = (E2 @ W2.T) * H * (1-H)    shape (m,h)
dW1 = X.T @ E1 / m               shape (d,h)
db1 = column_sums(E1) / m        shape (h,)

Here * is elementwise multiplication, whereas @ is matrix multiplication. Each hidden unit receives the sum of errors through all its outgoing weights. Use the same W2 that produced the forward pass; updating W2 before computing E1 differentiates a different computation.

The loss helper clamps probabilities to [1e-15, 1-1e-15] to keep its diagnostic logarithms finite. The simplified gradient describes sigmoid plus unclipped BCE. They agree while probabilities lie inside that interval; at saturation, clipping is a numerical reporting convention, not an exact gradient identity. Perform finite-difference checks away from that boundary. Later DL modules use objectives in logit space to handle extreme values more faithfully.

Work one full gradient step by hand

Choose one example and deliberately simple parameters:

X=[[1,0]], Y=[[1]]
W1=[[0,0],[0,0]], b1=[0,0]
W2=[[1],[-1]], b2=[0]

Forward gives z1=[[0,0]], H=[[0.5,0.5]], z2=[[0]], and P=[[0.5]]. Loss is log(2).

Now trace the gradients:

E2  = [[-0.5]]
dW2 = [[-0.25],[-0.25]]          db2 = [-0.5]
E1  = [[-0.125,0.125]]
dW1 = [[-0.125,0.125],[0,0]]     db1 = [-0.125,0.125]

The second input is zero, so the second row of dW1 is zero. Hidden errors have opposite signs because the outgoing weights have opposite signs. With learning rate 0.1, the first W1 row becomes [0.0125,-0.0125]; subtracting a negative gradient increases a weight.

This diagnostic setup uses zero W1 to make arithmetic visible. For actual training, independently initialize all weight entries in [-0.5,0.5] using the supplied seed and initialize biases to zero. Identical incoming and outgoing parameters make hidden units receive identical updates, wasting their capacity. The exercise's uniform initialization is not Xavier initialization.

Verify before a long training run

For one chosen parameter, make two copies of the current parameters. Perturb only that entry by +epsilon and -epsilon, rerun forward and loss, and compare:

numerical_gradient = (loss_plus-loss_minus)/(2*epsilon)

Use a small non-saturated example and, for example, epsilon=1e-5. Check a W1 entry, a W2 entry, and both bias groups. Compare with an absolute-plus-relative tolerance; near-zero gradients make a purely relative ratio unreliable. Restore the original parameter after each perturbation.

A second useful check is batch duplication: repeat every example and target twice. Because the loss is a mean over examples, the loss and all parameter gradients should remain unchanged. This detects a missing or duplicated /m without needing a known trained model.

Train, then interpret the result

train returns (W1,b1,W2,b2,losses). On each epoch:

1. Forward using current parameters.
2. Append the current, pre-update loss.
3. Compute all gradients without changing parameters.
4. Subtract learning_rate*gradient from each parameter.

The returned parameters include the last update; losses[-1] is from just before it. With zero epochs, return initial parameters and an empty loss list. predict uses p>=0.5 to return a matrix of integer labels.

For the supplied XOR example, try four hidden units, seed 42, 5,000 epochs, and learning rate 1.0. Check all four predictions and loss reduction. This is a small deterministic learning exercise, not evidence that a network will generalize to new tasks or that every seed converges.

Symptom	Check before changing capacity
Forward shape error	Row-example convention and the shared inner dimension
Gradients grow when the batch is duplicated	Exactly one division by batch size
Hidden gradients disagree with finite differences	Old W2, chain-rule factors, and transpose placement
NaN/overflow	Locate the first nonfinite value; inspect sigmoid and log inputs
Loss falls but XOR predictions fail	Thresholded predictions as well as the loss; then seed and learning rate

You now have the bridge to scalar autograd in DL v2: it automates these local derivatives and their accumulation. Understanding the manual shapes and checks remains useful when a larger model fails. For further explanation of XOR representations and backpropagation, see Deep Learning, chapter 6.