Classification: From Scores to Decisions

What you will learn

Given a dot product and a loss derivative, reconstruct a classifier's prediction and one training step. Then separate three questions: which class wins, which action a threshold triggers, and whether predicted probabilities match observed frequencies.

Prerequisites: vector dot products, derivatives, and gradient descent. Rows of X are examples throughout this module. The exercises use Python lists and full-batch gradient descent.

1) Define the target before choosing the output

Binary classification has labels 0 and 1. Multiclass classification chooses exactly one of K classes, indexed 0 through K-1. Multilabel classification may assign several labels to one row; a single softmax would incorrectly make those labels compete for total probability one.

For binary logistic regression, compute the logit z = dot(x, w) + b, then the model's estimated positive-class probability p = sigmoid(z). The default decision is 1 when p >= 0.5, including a tie; equivalently z >= 0. The boundary is dot(x,w)+b = 0. It is linear in the supplied features: adding x1*x2 changes what a linear model can express in the original coordinates.

A logit is not itself a probability. Its magnitude also is not geometric distance: for nonzero w, distance to the default boundary is abs(z)/norm(w).

2) Stable sigmoid at extreme scores

The mathematical formula is 1 / (1 + exp(-z)). Directly computing exp(-z) overflows for a sufficiently negative finite logit, such as -1000. Use equivalent branches that only exponentiate a nonpositive number:

if z >= 0:
    p = 1 / (1 + math.exp(-z))
else:
    ez = math.exp(z)
    p = ez / (1 + ez)

Both branches give 0.5 at zero and agree mathematically with sigmoid. In floating point, large magnitudes may round to exactly 0 or 1; the exercise accepts those endpoints. Its sigmoid also accepts a list, applying this scalar rule to each entry.

3) Derive a binary training step

For one example, binary cross-entropy is:

loss = -y*log(p) - (1-y)*log(1-p)
error = p-y
weight gradient = error*x
bias gradient = error

Combining the loss derivative with dp/dz = p*(1-p) gives the logit derivative p-y. For n examples, sum each parameter's contributions and divide by n once. Compute all gradients from the same predictions before updating any parameter.

Work the batch X=[[0],[2]], y=[0,1], starting with w=[0], b=0, learning rate 0.4:

Row	x	y	z	p	p-y	x*(p-y)
0	0	0	0	0.5	0.5	0
1	2	1	0	0.5	-0.5	-1

Thus dw=(0-1)/2=-0.5, db=(0.5-0.5)/2=0. One update gives w=[0.2], b=0. The recorded pre-update mean loss is log(2), approximately 0.693147. This is the convention of train(X,y,lr,n_epochs): initialize zeros, record loss, compute the full-batch gradient, then update once per epoch. With zero epochs return the zero parameters and an empty history.

Updating after each row is a different algorithm: later rows see changed parameters. Omitting the division by n changes the effective step size when batch size changes. Duplicating every example should leave the mean loss and a full-batch update unchanged.

The exercise's loss helper clips probabilities into [eps,1-eps] to avoid log(0). The p-y training rule differentiates the unclipped sigmoid/BCE objective; it is not the derivative of a hard-clipped reporting function at saturation. Check finite differences on moderate logits. A later implementation can evaluate the loss directly from logits for a more faithful extreme-value objective.

4) Stable softmax and its invariance

For one multiclass row, produce exactly K logits. Softmax divides each exponentiated logit by their total. A common shift does not change it: the factor exp(c) cancels from numerator and denominator. Choose c=-max(logits) so every exponent is at most one.

For logits [1000,1001,1002]:

Stage	Values
Subtract row maximum	[-2,-1,0]
Exponentiate	[0.135335,0.367879,1]
Divide by total 1.503214	[0.090031,0.244728,0.665241]

The winning class is index 2. Adding 5000 to every logit gives the same probabilities. Dividing logits by their maximum does not preserve them. Normalize each row independently; including another example's scores in the denominator couples predictions that should be independent. Tiny probabilities may underflow to zero even when the calculation is otherwise stable.

5) Multiclass shapes and gradients

The exercise uses the following orientation:

Quantity	Shape	Meaning
X	(n,d)	n examples with d features
W	(d,K)	one column per class
b	(K,)	one bias per class
Z = X @ W + b	(n,K)	bias added to each row
P = softmax_batch(Z)	(n,K)	each row sums to one
Y = one_hot_encode(y,K)	(n,K)	one true class per row

For true class 1 and prediction [0.2,0.5,0.3], the example loss is -log(0.5), and its unaveraged logit error is [0.2,-0.5,0.3]. Each class competes through normalization; errors sum to zero. For a mean batch objective:

E  = P-Y
 dW = transpose(X) @ E / n
 db = column_sums(E) / n

Do not divide E by n and then divide parameter gradients by n again. cross_entropy_loss averages over examples; summing over class columns selects the true-class log probability. train_softmax_classifier(..., learning_rate) returns W,b, and predict(X,W,b) returns the index of the largest logit, choosing the first index on ties. There is no separate softmax_gradient function in this starter: calculate P-Y inside training.

6) A decision threshold is a choice of action

Consider fixed probabilities [0.9,0.6,0.4,0.2] and labels [1,0,1,0]:

Threshold, using >=	Predictions	True positives	False positives	False negatives
0.5	[1,1,0,0]	1	1	1
0.3	[1,1,1,0]	2	1	0

The lower threshold catches the additional positive in this sample; probabilities and ranking are unchanged. Precision is TP/(TP+FP) and recall is TP/(TP+FN) when denominators are nonzero. Lowering a threshold cannot reduce the number flagged, but it may add false positives. Choose the operating point using validation data and the task's cost of each error, then evaluate on a separate test set.

If 99 of 100 examples are negative, always predicting zero gets 99% accuracy and zero positive-class recall. F1 is one useful summary of precision/recall, not a universal replacement for every metric or business cost.

7) Calibration must be checked

A probability-shaped output is an estimate, not a calibration guarantee. Among held-out cases assigned probabilities near 0.8, a calibrated model should have a positive fraction near 0.8. If 100 such cases contain only 55 positives, their observed fraction is 0.55: the model is overconfident in that group. A small bin is noisy; one bin also cannot establish calibration everywhere.

Logistic regression can be well calibrated under suitable model/data conditions, but misspecification, regularization, finite data, or distribution change can spoil that property. Reliability diagrams compare predicted and observed frequencies on held-out data. Changing a decision threshold does not change those probabilities or repair their calibration. Calibration fitting needs data distinct from the classifier's training predictions. See scikit-learn's calibration guide.

Practice and diagnose

Implement the binary exercise first and reproduce the one-step table. In the multiclass exercise, verify the shape table, row sums, common-shift invariance, and mean reduction before judging convergence. If a checkpoint answer is wrong, revisit: section 2 for overflow; sections 3–5 for gradients and shapes; section 6 for thresholds; section 7 for calibration. A decreasing training loss alone does not verify those contracts or generalization.