Language Expansion: Variables, Statements, Functions

v2 turns expressions into programs with mutable bindings, blocks, branches, while loops, and top-level named functions. This module's coding task is the expanded lexer; later modules implement the other stages. Carry the tiny scanner's cursor and maximal-munch invariants forward while replacing its token contract with program-lexer's provided kinds.

New language boundary

let and var both make mutable bindings in this course. const requires an initializer and rejects reassignment in the analyzer and VM. Declarations introduce a binding; assignment updates a previously declared one. Functions are top-level in v2; nested functions and closure values begin in v3.

The lexer adds var/const/while/true/false/nil and strings. v2 does not accept the tiny lexer's standalone dot token: member access arrives with v5. A minus remains its own token; integer literals contain digits only.

Worked reconstruction

For source let text = "a\n";, distinguish two levels of representation. In the actual language source the quoted characters are a, backslash, n; the STRING token's value contains a followed by one newline character. The source cursor remains on its original line because the escape occupies two source characters. A literal unescaped newline inside quotes is an error.

The escaped input "say: \"hi\"" produces a string value containing quote characters. Scanning looks for an unescaped closing quote, so stopping at the first quote byte after a backslash would truncate it. Support exactly the four listed escapes (\", \\, \n, \t); an unlisted \q must fail.

A feature crosses several stages

For const total = 2; total = 3;, lexing emits valid tokens and parsing builds two statements. The analyzer identifies assignment to a const; the program never reaches compilation through the capstone. For if (0) { ... }, runtime truthiness treats zero as true: only false and nil are falsey in v2–v5. v6 instead demands boolean conditions statically. Keep these version rules separate from familiar Go or JavaScript conventions.

Debugging lab

A scanner increments its source line counter when it decodes the two-character escape backslash-n inside a string. The next token is now reported on the wrong line. Explain which representation was confused.

Compare your diagnosis after attempting the lab

Decoded string content was confused with source text. Line/column track characters consumed from source, while the token value holds interpreted escapes. Only an actual source newline advances the cursor line.

Transfer task

Make a three-row comparison of let name="x";, let name="x;, and let name="x\q";. Predict tokens or the earliest lexer failure, then separately explain why a valid token sequence does not prove legal assignment.

Carry forward

Carry decoded STRING values and exact token positions into program parsing. Retain the old expression regressions where token contracts agree, and add explicit version-boundary tests where they differ.

Build program-lexer. Keep a before/after trace of one failing test. Use the checkpoint to diagnose specific skills; passing it does not replace implementing and testing the artifact.