Lexing: From Text to Tokens

Trace maximal-munch scanning and keyword boundaries.
Maintain source cursor positions through comments and newlines.
Separate the tiny lexer from later string-capable lexers.

1 / 6

Readiness

Recognize the expression milestone and one-based source positions.

2 / 6

Worked reconstruction

For the exact source let x=12;\n//n\nx!=3, where \n means an actual newline:

Token	Start
LET `let`	1:1
IDENT `x`	1:5
TokenAssign `=`	1:6
NUMBER `12`	1:7
SEMICOLON `;`	1:9
IDENT `x`	3:1
TokenNotEqual `!=`	3:2
NUMBER `3`	3:4
EOF	3:5

The comment produces no tokens but still consumes line 2. A leading minus is an operator token followed by a number, so -12 is not one signed-number token. Tokenizing first makes subtraction and unary negation the parser's decision.

3 / 6

Diagnose before implementing

A scanner reads returnValue==7 as RETURN, IDENT(Value), TokenAssign, TokenAssign, NUMBER. Identify two independent defects and specify the correct token boundaries.

4 / 6

Independent transfer

Construct three inputs ending immediately after /, after // note, and after a newline. Predict each EOF position before running. Then change one character to an unsupported symbol and confirm the scanner makes progress by returning an error.

5 / 6

Carry forward

Carry deterministic tokens with start positions into expression parsing. Keep token boundary tests separate from AST tests; a parser cannot recover a token that the scanner split incorrectly.

6 / 6