Tensors: Reconstruct Shapes and Indices

A tensor in these exercises is a scalar or a rectangular nested Python list. There is no NumPy dependency, automatic GPU execution, or automatic differentiation yet. You will reconstruct element positions yourself; the same shape reasoning later applies to array frameworks.

Prerequisites: nested lists, loops/recursion, matrix multiplication, and the difference between a list reference and a copy. Use linear algebra to review dot products. Read this module before a shape-heavy classifier if needed; its late catalog position is not a prerequisite restriction.

A shape describes positions, not meanings

5 has shape (), [1,2,3] has (3,), [[1,2,3]] has (1,3), and [[1],[2],[3]] has (3,1). Equal element counts do not make these shapes interchangeable. The rank is the number of dimensions; the element count is their product (one for a scalar).

For a list of training examples, (n,d) means n rows of d features. A sequence may use (T,D) or a batch (B,T,D). Shape alone cannot tell whether the first axis is a batch or time: name each axis in the API. Ragged inputs such as [[1,2],[3]] are outside these exercises.

Reshape preserves traversal order

Flatten by visiting nested lists left to right. For A=[[1,2,3],[4,5,6]], this gives [1,2,3,4,5,6]. Reshape to (3,2) consumes two scalars per new row:

[[1,2],
 [3,4],
 [5,6]]

This is different from transpose, which maps B[j][i]=A[i][j] and produces [[1,4],[2,5],[3,6]]. A reshape changes grouping; a transpose changes index order.

This task uses explicit positive sizes, plus () for a scalar. It does not implement NumPy's inferred -1 dimensions or shared-memory views. A mismatched element count raises ValueError. create_zeros((2,3)) must allocate separate rows: [[0]*3]*2 aliases a single row, so assigning [0][0] also changes [1][0].

Reduction removes the axis being summed

For A=[[1,2,3],[4,5,6]]:

Operation	Which index varies inside the sum?	Result
axis 0	row i, while column j stays fixed	[5,7,9]
axis 1	column j, while row i stays fixed	[6,15]

The exercise's sum_along_axis handles a nonempty 2D matrix and axis 0 or 1. Its result removes that axis; it does not keep a size-one dimension. General higher-rank reductions are an extension.

Matrix multiplication contracts an inner axis rather than pairing every scalar elementwise. For A=[[1,2],[3,4]], B=[[5],[6]], A@B=[[17],[39]], shape (2,1). (m,n)@(n,p) sums n products for each of m*p output positions. elementwise_add, in the tensor-operations exercise, requires identical shapes; broadcasting belongs to the separate exercise.

Broadcasting starts with shape compatibility

Align dimensions from the right. Paired dimensions must be equal or one must be 1; a missing leading dimension acts as 1. This is the rule in NumPy's broadcasting documentation, but you will implement the index mapping with lists.

A:      (2,1,3)
B:        (4,1)   -> padded (1,4,1)
result: (2,4,3)

The first output dimension comes from A, the middle from B, and the last from A. (2,3) and (2,) are incompatible because the last pair is 3 versus 2; a vector is aligned to columns, not automatically to rows. (1,3) and (3,) are compatible and still produce (1,3): padding an axis of size 1 must not accidentally remove the only row.

Map one output coordinate back to each input

After finding the output shape, enumerate each output coordinate. Drop any leading coordinates corresponding to missing input axes. For every real input dimension of size 1, use input index 0; otherwise use the matching output index. Read the two scalars, apply the operation, and store a fresh output scalar.

For the shapes above, output coordinate (1,2,2) reads:

Input	Padded shape	Read coordinate
A	(2,1,3)	A[1][0][2]
B	(1,4,1)	B[2][0] after dropping the padded axis

If A is [[[1,2,3]],[[4,5,6]]] and B is [[10],[20],[30],[40]], that sum is 6+30=36. This rule also handles scalars: shape () needs no indexing, so the scalar participates at every output coordinate. There is no need to create repeated copies of whole input lists.

Complete a shape and value trace

Let A be [[1],[2]] and b be [10,20,30]. Derive the output shape and the source indices for output (1,2), then write the product at that coordinate.

Check the broadcast trace

Shapes (2,1) and (3,) align as (2,1) and (1,3), yielding (2,3). Output (1,2) reads A[1][0]=2 and b[2]=30, so the product is 60. The full result is [[10,20,30],[20,40,60]].

Independent transfer to neural-network shapes

Implement scalar reshape, matrix transpose/reduction, and compatible matmul before the general broadcast mapper. Then test (1,3)+(3,), a scalar times a rank-three tensor, and incompatible shapes. Compute expected values by source indices rather than by duplicating your implementation's recursive logic.

Finally predict X:(5,2) @ W:(2,3) + b:(3,): result (5,3), with b's three feature coordinates reused for all five examples. In the XOR capstone, those dimensions connect data, hidden activations, and gradients. Later scalar-autograd Linear stores one weight row per output instead of the ML matrix convention's column per output; translate the stored orientation before composing layers. A transpose is not a new learning rule.