Model Evaluation Metrics

An evaluation must say which population was held out, which predictions were made, and which mistakes matter. Here you will derive five binary classification metrics, then diagnose why a technically correct score can still answer the wrong question.

Establish the evaluation boundary

Training scores help diagnose optimization and fit; they do not estimate performance on unseen examples. Use training data to fit parameters and preprocessing, validation data to choose features, regularization and thresholds, and an untouched test set for the final estimate. Once a test result guides a change, it has participated in selection.

Choose the split according to the intended use. Random splits can be reasonable for independent examples from the same population. Separate all records of one user into the same group when measuring performance on new users. For forecasting, train on earlier information and evaluate later information; shuffling future observations into training changes the question. Fit scaling and vocabulary inside each training partition. Scikit-learn's cross-validation guide explains these distinct split strategies.

For cross-validation, allocate every example to a validation fold exactly once, including a remainder when n is not divisible by k. For each fold, create a fresh model and fresh fitted preprocessing using only the other folds. Average validation scores for selection; the test set stays outside this loop. A spread across folds describes variation in those splits, not automatically a confidence interval or protection against leakage.

Build the confusion dictionary one row at a time

The exercise receives labels, not scores. Both lists are equal-length and nonempty, with values 0 or 1. Treat 1 as the positive class.

y_true = [1, 0, 1, 0, 1, 0]
y_pred = [1, 1, 0, 0, 1, 0]

Index	True	Predicted	Increment
0	1	1	TP
1	0	1	FP
2	1	0	FN
3	0	0	TN
4	1	1	TP
5	0	0	TN

Return {'TP':2, 'TN':2, 'FP':1, 'FN':1} from confusion_matrix, not a nested list. Check that the four counts sum to n. True/false says whether the prediction was correct; positive/negative says which label was predicted.

A useful display places actual class on rows and predicted class on columns: [[TN,FP],[FN,TP]]. This display is a visualization of the dictionary, not this exercise's return type.

Make every denominator answer a question

From those counts:

accuracy  = (TP+TN)/n       = 4/6 = 2/3
precision = TP/(TP+FP)      = 2/3
recall    = TP/(TP+FN)      = 2/3
F1        = 2TP/(2TP+FP+FN) = 4/6 = 2/3

Precision asks how many positive predictions were correct. Recall asks how many actual positives were found. The same number here is a coincidence, not an interchangeable definition. Reclassify index 1 from positive to negative: FP becomes 0 and TN becomes 3. Precision becomes 1, recall remains 2/3, and F1 becomes 4/5.

This pack returns zero precision when no positives were predicted, zero recall when no positives exist in the labels, and zero F1 when precision plus recall is zero. These are API conventions for otherwise undefined ratios. Report the underlying counts so zero cannot silently mean either an observed failure or absence of relevant cases. The metric definitions distinguish these denominators and exceptional cases.

Sweep a threshold without assuming precision is monotonic

To turn scores into labels, choose a rule; use score >= threshold here. Consider examples ordered by decreasing score:

Score	True label
0.9	1
0.8	0
0.7	1
0.4	0

At threshold 0.9, precision is 1 and recall is 1/2. At 0.8, precision falls to 1/2 while recall stays 1/2. At 0.7, precision rises to 2/3 while recall rises to 1. Lowering a threshold on fixed scores never removes a predicted positive, so recall cannot decrease; precision can move either way depending on which examples enter.

A perfect ranking can achieve both precision and recall of 1 at an appropriate threshold. There is no theorem forbidding both. Choose thresholds on validation data using the actual costs or constraints. If one false negative costs 5 units and one false positive costs 1, the three thresholds above cost 5, 6, and 1 respectively. Cost is a stated decision rule, not a universal replacement for other metrics.

Interpret a score against a baseline

If only one of 100 examples is positive, predicting all negative gives 99% accuracy and zero positive recall. The baseline provides context: a model with 98% accuracy could still be useful if it finds the important positive and the additional false alarms are acceptable. F1 summarizes precision and recall but ignores true negatives and does not encode arbitrary error costs.

ROC curves compare recall to false-positive rate across thresholds. ROC AUC measures ranking across positive-negative pairs; it does not establish calibrated probabilities or select an operating threshold. Metrics for regression answer different questions: MAE uses absolute errors, RMSE takes the square root of mean squared errors, and R² compares squared error to a mean-target baseline. Those extensions are outside this binary-label API.

Finish a fresh case and verify independently

For y_true=[1,1,1,0,0], y_pred=[1,0,1,1,0], find the dictionary and all four scores. Then change the threshold so the false positive becomes negative without changing any other label.

Reveal the calculation

Initially TP=2, TN=1, FP=1, FN=1: accuracy is 3/5 and precision, recall, and F1 are each 2/3. After the change TP=2, TN=2, FP=0, FN=1: accuracy is 4/5, precision is 1, recall is 2/3, and F1 is 4/5.

Implement the dictionary first, then derive each score from it. Use a deliberately asymmetric case to catch precision/recall swaps. Test perfect predictions, all-negative predictions, and no actual positives separately. As a transfer task, give two datasets the same confusion counts but different user/time overlap across the split; explain why identical metrics can have different credibility as evidence for generalization.