Classification: Scores, Training, Decisions

• Rows are examples; prerequisites are dot products and gradient descent.
• Binary: sigmoid; exactly one of K classes: softmax.
• Multilabel targets need independent label decisions.

Stable binary probabilities

• z=dot(x,w)+b; default label is 1 when p>=0.5.
• For z>=0, use 1/(1+exp(-z)).
• Otherwise use exp(z)/(1+exp(z)).
• z=-1000 must not overflow; rounded 0/1 endpoints are possible.

One full-batch update

X=[[0],[2]], y=[0,1], w=0, b=0, lr=0.4

• Probabilities [0.5,0.5]; errors [0.5,-0.5].
• dw=-0.5, db=0; updated w=0.2.
• Pre-update loss is log(2).
• Average once; calculate all gradients before updating.

Softmax without overflow

[1000,1001,1002] -> [-2,-1,0]

• Exponents [0.135335,0.367879,1].
• Probabilities [0.090031,0.244728,0.665241].
• Subtract the maximum separately for each row.
• Adding any common finite shift preserves the mathematical probabilities.

Match the exercise shapes

• X:(n,d), W:(d,K), b:(K,).
• Z=X@W+b, P and one-hot Y: (n,K).
• E=P-Y; dW=X.T@E/n; db=column_sums(E)/n.
• Target 1, P=[0.2,0.5,0.3] gives error [0.2,-0.5,0.3].

Threshold changes actions

For p=[0.9,0.6,0.4,0.2], y=[1,0,1,0]:

• At 0.5: TP=1, FP=1, FN=1.
• At 0.3: TP=2, FP=1, FN=0.
• Ranking and probabilities stay fixed.
• Evaluate precision, recall, and error costs on held-out data.

Calibration is empirical

• Predictions near 0.8 should have positive frequency near 0.8.
• 55 positives among 100 such cases suggest overconfidence in that bin.
• Logistic regression does not guarantee calibration.
• A threshold cannot repair probability estimates.

Evidence before convergence

• Test extreme sigmoid logits and shifted softmax logits.
• Duplicate the batch: mean loss and update should remain unchanged.
• Check gradients away from clipping/saturation.
• Measure held-out behavior; training loss alone is insufficient.