For abaca: vocabulary [a,b,c], IDs [0,1,0,2,0].
Sorting makes the mapping reproducible. ID 2 does not encode more similarity or magnitude than ID 1.
Decode must invert encode exactly for known characters, including whitespace.
No unknown token is reserved; unknown symbols raise KeyError.
Keep the mapping with the model.
Data length 5, context length 3.
How many legal starts? What are the input and target at the final start?
Each example needs four source tokens.
Legal starts: 0 and 1.
At start 1: input [1,0,2], target [0,2,0].
Inclusive upper bound: N-T-1. N=T+1 has exactly one window.
Sequence models: inputs (B,T), targets (B,T) shifted one position.
Fixed-window MLP: contexts (B,T), targets (B,).
The last target of a shifted window is the MLP target for that full context.
Split the stream into two contiguous lists; sample each independently.
A training window must not cross into validation.
A valid split ratio alone does not make each segment long enough for T.
Two fresh RNGs with the same seed reproduce a batch.
Successive calls on one RNG advance its state.
Resetting before every batch repeats examples; evaluation should use separate RNG state.
Use distinct source IDs to expose an unshifted target bug.
Check both endpoint starts, minimum data length, exact output shapes, and text round trips.
Fast falling copy-task loss is not evidence of next-token learning.