XOR: Expressiveness, Gradients, Evidence
Goal: a list-based neural network whose predictions and gradients you can explain.
Prerequisites: dot products, sigmoid, BCE, chain rule.
1 / 9
Goal: a list-based neural network whose predictions and gradients you can explain.
Prerequisites: dot products, sigmoid, BCE, chain rule.
[0,0] -> 0 [0,1] -> 1
[1,0] -> 1 [1,1] -> 0
No affine boundary gets all four right.
But x1+x2-0.5 >= 0 gets three right: 75%, not a universal 50% ceiling.
Let s=x1+x2.
h1 = sigmoid(10*(s-0.5)) # approximately “at least one”
h2 = sigmoid(10*(s-1.5)) # approximately “both”
score = 10*(h1-h2-0.5)
Hand-picked weights prove capacity; a training run still needs checking.
X: (m,d) W1: (d,h) W2: (h,o)
z1 = X @ W1 + b1 (m,h)
H = sigmoid(z1) (m,h)
z2 = H @ W2 + b2 (m,o)
P = sigmoid(z2) (m,o)
Biases broadcast across rows.
The exercise uses list helpers, not NumPy.
BCE derivative with sigmoid: p-y
E2 = P-Y
dW2 = H.T @ E2 / m
db2 = column_sums(E2) / m
E1 = (E2 @ W2.T) * H * (1-H)
dW1 = X.T @ E1 / m
db1 = column_sums(E1) / m
Mean over examples, sum over output columns.
Use old W2 until all gradients are ready.
X=[[1,0]], Y=[[1]]
W1=all zeros, b1=[0,0]
W2=[[1],[-1]], b2=[0]
H=[[0.5,0.5]], P=[[0.5]].
What is dW1's second row? [0,0]: that input is zero.
Its first row is [-0.125,0.125].
Finite difference: perturb one parameter by ±epsilon and recompute loss.
Batch duplication: mean loss and all gradients should stay the same.
Check away from probability-clipping boundaries.
A falling loss alone does not prove backpropagation is correct.
Random weights in [-0.5,0.5], zero biases, reproducible seed.
Each epoch: forward → record loss → compute all gradients → update.
Return four parameter groups plus losses.
Final parameters include the last update; final recorded loss precedes it.
Next: scalar autograd automates local chain-rule propagation.