Tokenization and Batching: Define What the Model May Predict

A model consumes integer IDs, but its objective is a relationship between positions in text. Tokenization defines the IDs; batching defines which token is visible and which token must be predicted. An off-by-one error can teach the model to copy its input while reporting an impressive loss.

The exercise uses characters, not bytes or subwords. You will implement deterministic vocabulary construction, reversible encoding, a contiguous train/validation split, and next-token windows. No neural network is needed for these checks.

Make the mapping reproducible

For the corpus "abaca", sorting its distinct characters gives:

characters: ['a', 'b', 'c']
stoi:       {'a': 0, 'b': 1, 'c': 2}
itos:       {0: 'a', 1: 'b', 2: 'c'}
encoded:    [0, 1, 0, 2, 0]

Character IDs have no magnitude-based meaning: 2 is not “more c” than 1 is “b.” Preserve whitespace and punctuation as characters. Python strings iterate Unicode code points, so this simple tokenizer is not a general grapheme or byte tokenizer.

decode(encode(text)) == text must hold for text entirely inside the vocabulary. This exercise reserves no unknown-token ID: an unknown character or unknown decoded ID raises KeyError. Save and reuse the same mapping with the model; rebuilding it from a different corpus can change what every ID means.

Shift targets exactly once

Let the encoded data above have length N=5 and choose T=3. A training window needs T+1 source tokens: three inputs and their three successors. There are N-T=2 legal start positions.

Start	Input x	Target y	Last prediction
0	[0,1,0]	[1,0,2]	Last visible a predicts c
1	[1,0,2]	[0,2,0]	Last visible c predicts a

Thus x=data[i:i+T] and y=data[i+1:i+T+1]. The last legal start is N-T-1, inclusively. randint(a,b) includes both endpoints in Python's random API, so use rng.randint(0, N-T-1).

Each returned batch has shape (B,T) for both inputs and targets. Draw starts independently, with replacement. Repeated or overlapping windows are allowed. N=T+1 is valid and has only start 0; N=T has no complete example. For positive batch sizes, require positive T and at least T+1 data items. A zero batch size returns two empty lists.

The MLP language model later uses a different target shape: one next token for an entire context. Here we deliberately prepare one target per position, which the bigram and GPT models consume.

Split before sampling windows

For ten encoded tokens and split_ratio=0.7, the split index is int(10*0.7)=7. Training owns indices 0 through 6, validation owns 7 through 9. Call get_batch independently on the two lists. Do not sample a window across their boundary and then assign it to a split by its start alone.

A contiguous split preserves text order, but it can produce different token distributions in the two segments. That is useful evidence, not a reason to tune on the validation examples. In this closed-corpus lab, the character inventory may be fixed from the supplied corpus before the split; that reveals the alphabet, not transition frequencies. For an unseen-data deployment, define a training-derived or externally fixed vocabulary and an explicit unknown-character policy.

The split function is simple slicing, with 0 <= split_ratio <= 1 as its input contract. It does not guarantee either side is long enough for your chosen context length. Check that before sampling.

Determinism requires controlling consumption

Pass a random.Random(seed) instance into get_batch. Two newly created RNGs with the same seed and identical arguments should generate identical batches. Two successive calls using the same RNG usually differ because its state advances. Resetting the seed before every batch would repeatedly show the model the same examples.

Use a separate RNG or fixed windows for evaluation. Otherwise running an evaluation can consume the training RNG's state and change the later training sequence, making comparisons harder to interpret.

Predict the failure before implementing

Checkpoint 1. N=8, T=4: how many legal starts, and what is the largest?

Check your reasoning

four starts, 0 through 3. Start 4 supplies four inputs but only three targets.

Checkpoint 2. A batch satisfies x == y and the loss falls rapidly. Is that a successful next-token model?

Check your reasoning

no. The model is being trained to reproduce the current token. Assert the shift relation using a source sequence with distinct IDs so accidental equality cannot hide the mistake.

Test the exact minimum-length corpus, both endpoint starts, unknown symbols, and a round trip containing spaces and non-ASCII characters. Then move to embeddings, where these IDs select shared trainable vectors.