A (V,D) table owns V*D scalar parameters.
IDs [2,0,2] return shape (3,D) but do not create new parameters.
What must remain identical when token 2 occurs twice?
Return references to table Value objects.
Copying a list can preserve those references; Value(old.data) cannot.
The latter produces plausible outputs with no gradient path to the table.
Token rows: E[0]=[1,0], E[2]=[-1,1].
Positions: [0.1,0.2], [0.3,-0.1], [-0.2,0.4].
Combine for IDs [2,0,2].
[-0.9, 1.2]
[ 1.3,-0.1]
[-1.2, 1.4]
Elementwise addition preserves width D. Concatenation would produce width 2D.
Diagnostic loss: sum every combined output scalar.
Gradients: E[0]=[1,1], E[1]=[0,0], E[2]=[2,2].
Each used position row gets [1,1]. Explain the factor of two.
Unmasked attention without position signals reorders outputs when inputs reorder: permutation equivariance.
It does not leave the whole output unchanged.
A fixed causal mask changes the symmetry; explicit positions still supply an address.
Random lookup rows do not begin with semantic geometry.
Learning depends on the objective. Learned absolute positions do not automatically extend beyond Tmax.
A concatenating MLP gets order from its input slots.
For V=5, D=4, Tmax=8: 52 table parameters.
Test shared scalar identity, repeated-ID gradient accumulation, unused-row zeros, and matching widths.
Only lookup paths are covered by the sparse-gradient claim.