Feature Engineering: Fit Once, Transform Consistently

What you will learn

Turn numeric and categorical columns into a stable model input. You should be able to derive training statistics, transform a new row without refitting, explain constant-column policies, and preserve the meaning of every one-hot column.

Prerequisites: lists, means, variance, and the distinction between training and held-out data. The two exercises use rectangular Python lists with rows as examples.

1) A fitted transform is part of the model

A fitted scaler is not just a formula: it is a formula plus stored training parameters and column order. A categorical encoder similarly stores its ordered vocabulary. If either changes at prediction time, the model sees coordinates with different meanings.

Split data first. Learn means, ranges, categories, imputations, and feature-selection decisions using the training portion. Apply the saved transform to validation/test rows. Inside cross-validation, fit these steps separately within each training fold. Fitting a scaler to all rows before the split leaks information about held-out distributions even though it uses no target labels. See scikit-learn's preprocessing pitfalls.

The exercises separate min_max_scale / standardize (fit and transform) from min_max_transform / standardize_transform (reuse parameters). For categories, one_hot_encode learns the vocabulary, whereas encode_value(value,categories) reuses it.

2) Derive min-max scaling by column

For a nonconstant training column, let lo=min(column) and hi=max(column):

scaled(x) = (x-lo)/(hi-lo)

Use training rows [[2,10,7],[4,20,7],[6,30,7]]:

Column	Training minimum	Training maximum	Span
0	2	6	4
1	10	30	20
2	7	7	0

The exercise defines a zero-span column's output as 0, including when a later value differs. Thus the training matrix becomes [[0,0,0],[0.5,0.5,0],[1,1,0]]. Save mins=[2,10,7] and maxs=[6,30,7].

Transform a held-out row [8,15,9] with those saved parameters:

[(8-2)/4, (15-10)/20, zero-span policy] = [1.5,0.25,0]

The first coordinate exceeds one because it exceeds the training maximum. This exercise does not clip; only the fitted nonconstant training columns are guaranteed to lie in [0,1]. Refitting on the held-out row would make every column constant and erase its position relative to training data.

The zero-span policy deliberately discards later variation in that coordinate. It is a course contract, not the only possible scaler convention; libraries may use a unit denominator instead. A valid scalar transform fragment is:

span = maxs[j] - mins[j]
value = 0.0 if span == 0 else (row[j] - mins[j]) / span

3) Standardization and the denominator you save

For n training rows, this exercise uses population variance, divided by n:

mean = sum(x)/n
variance = sum((x-mean)^2)/n
scale = sqrt(variance) if variance > 0 else 1
z = (x-mean)/scale

The returned vector is named stds, but it contains these effective scales: a constant column stores 1, not 0. That makes the same transformation valid on future rows without dividing by zero. The /n and unit-scale convention also appears in StandardScaler's documentation.

Use training rows [[1,7],[3,7]]. Means are [2,7], population variances [1,0], and saved stds=[1,1]. Transformed training rows are [[-1,0],[1,0]]. The held-out row [5,9] becomes [3,2] using the same means and scales. Unlike this exercise's min-max rule, the constant training column preserves its later displacement 9-7.

Only nonconstant training columns have unit variance after standardization. Held-out data need not have zero mean or unit variance. Standardization changes location and scale; it does not make a skewed distribution normal or remove outliers.

4) Why scale, and when to question it

In a Euclidean distance, changing one coordinate by 1000 contributes one million to squared distance; changing another by 1 contributes one. Scaling therefore chooses how coordinates trade off. It can also improve gradient-descent conditioning and change the effect of coefficient penalties. It does not automatically make every distance meaningful or guarantee faster training on every problem.

Min-max scaling retains sensitivity to extreme training values. Standardization is also sensitive to outliers through its mean and variance. Pick a representation using domain meaning and held-out performance. A count, a binary flag, and an ordered category need not all use the same transform.

5) One-hot columns are an ordered vocabulary

Integer IDs such as bird=0, cat=1, dog=2 may accidentally imply meaningful distance and order to a numeric model. One-hot encoding instead gives each category its own coordinate.

For training values ["cat","dog","cat","bird","dog"], get_categories returns the sorted unique vocabulary ["bird","cat","dog"]:

Value	bird	cat	dog
cat	0	1	0
dog	0	0	1
cat	0	1	0
bird	1	0	0
dog	0	0	1

The encoded matrix is [[0,1,0],[0,0,1],[0,1,0],[1,0,0],[0,0,1]]. decode([0,1,0],categories) returns "cat" because the 1 occupies category index 1.

At prediction time reuse these three columns. Calling one_hot_encode(["dog","cat"]) would fit a new two-column vocabulary ["cat","dog"]: both width and positions change. The correct transformation is [encode_value(v,categories) for v in new_column].

6) Unseen values require an explicit policy

This exercise defines encode_value("fox",["bird","cat","dog"]) as [0,0,0]. It neither invents a column nor changes the saved vocabulary. A known value has exactly one 1; an unseen value has none. decode returns None for an all-zero row. That row cannot recover the original unseen category: "fox" and "horse" collide.

In a larger system, an explicit unknown category, an error, or retraining may be better. Here the fixed-width all-zero policy is intentional, and we keep all category columns: dropping one as a baseline would make that baseline indistinguishable from unknowns unless handled separately. Missing values need their own declared policy too.

7) Extending the pipeline without leaking information

Mean imputation learns a fill value on observed training values and reuses it; an optional missingness flag retains information that a value was absent. A fully missing column needs an explicit fallback, not a division by zero. The two exercises assume no missing input values and do not implement imputation.

Feature creation can expose relationships a model cannot otherwise express. For [x1,x2], adding x1*x2 lets a linear classifier use an interaction; adding squares changes its boundary in the original coordinates. log1p(x) compresses large nonnegative counts but is not valid for arbitrary negative inputs. Binning replaces continuous detail with chosen intervals and may lose useful signal.

High-cardinality one-hot columns can use substantial memory. Frequency encodings learn counts from training data. Target encodings require greater care: a training row's own target must not determine its encoded feature; use out-of-fold encodings and keep final evaluation data separate. Feature selection using target correlations also belongs inside the training fold. These are extensions, not additional APIs in these exercises.

Practice and diagnose

First reproduce both numeric tables, saving parameters before transforming new rows. Then encode and decode the category table, followed by a held-out subset and an unseen value. Explain why a min-max result above one is valid, why a constant training column has different policies in the two scalers, and why refitting an encoder on a prediction batch is wrong. The checkpoint uses new numbers to test whether those rules transfer.