Lexing: From Text to Tokens

The tiny-lexer produces tokens for the first milestone. It recognizes integer literals, identifiers, the keywords let/fn/if/else/return, the listed operators, and punctuation including .. It does not recognize strings, booleans as keywords, &&, or arrays. Later lexers extend this contract.

Cursor invariant

Keep a byte index plus a one-based (line,column). Capture a token's start before consuming its characters. Newline advances the line and resets column to 1; a tab consumes one source column in this ASCII exercise. EOF describes the cursor after the final character, not the start of the last token.

Read a full identifier before testing keyword membership: ifx is one identifier, not keyword if followed by x. For =, look ahead for another =; for /, look ahead for / before emitting division. A comment ends at newline or EOF. Consume the comment text without emitting tokens, then update the newline through the usual cursor rule.

Worked reconstruction

For the exact source let x=12;\n//n\nx!=3, where \n means an actual newline:

Token	Start
LET let	1:1
IDENT x	1:5
TokenAssign =	1:6
NUMBER 12	1:7
SEMICOLON ;	1:9
IDENT x	3:1
TokenNotEqual !=	3:2
NUMBER 3	3:4
EOF	3:5

The comment produces no tokens but still consumes line 2. A leading minus is an operator token followed by a number, so -12 is not one signed-number token. Tokenizing first makes subtraction and unary negation the parser's decision.

From trace to implementation

Start with identifiers/numbers and EOF, then operators with lookahead, then comments/position updates. Every loop iteration must consume input, emit EOF, or return an error; otherwise an unexpected character can cause an infinite loop. Unexpected characters return an error rather than disappearing.

Debugging lab

A scanner reads returnValue==7 as RETURN, IDENT(Value), TokenAssign, TokenAssign, NUMBER. Identify two independent defects and specify the correct token boundaries.

Compare your diagnosis after attempting the lab

Scan the complete identifier returnValue before keyword lookup. Prefer the legal two-character == token to two = tokens. The correct boundaries are IDENT(returnValue), TokenEqual, NUMBER(7), EOF.

Transfer task

Construct three inputs ending immediately after /, after // note, and after a newline. Predict each EOF position before running. Then change one character to an unsupported symbol and confirm the scanner makes progress by returning an error.

Carry forward

Carry deterministic tokens with start positions into expression parsing. Keep token boundary tests separate from AST tests; a parser cannot recover a token that the scanner split incorrectly.

Build tiny-lexer. Keep a before/after trace of one failing test. Use the checkpoint to diagnose specific skills; passing it does not replace implementing and testing the artifact.