Lexing: From source text to parseable stream

Lexing is where every parser bug starts or disappears. The v2 lexer must produce a stable token stream with reliable positions.

Why lexers must be deterministic

• parser correctness depends on one token for each syntactic unit
• ambiguous scanning (like = vs ==) must always prefer the maximal token
• token positions are how we make downstream diagnostics actionable

A deterministic lexer is easier to test and harder for parser regressions to hide.

Character-level flow

1. initialize (line=1, col=1, i=0)
2. skip whitespace and comments
3. scan one token:
   • identifiers/keywords
   • numbers
   • strings
   • operators/punctuation
   • unexpected character => error
4. append token with lexeme and start position
5. emit explicit EOF with current position

Core token classes

Identifiers and keywords

Regex-like class:

• start: letter or _
• continue: letter, digit, _

After scan, map identifier lexeme to keyword tokens (let, fn, if, ...).

Numbers

• decimal integer only: [0-9]+
• no floats or signs here

String literals

• double-quoted form: "..."
• supports escapes: \", \\, \n, \t
• no newlines allowed inside strings
• Lexeme stores unescaped content (without wrapper quotes)

Operators/punctuation

• single and double char operators:
  • = vs ==
  • ! vs !=
  • < vs <=
  • > vs >=
• punctuation: ( ) { } , ;

Longest-match first is non-negotiable for every two-character operator.

Comments and whitespace

• skip spaces, tabs, carriage returns
• increment line/reset col on newline
• line comments: // through end of line

Positioning contract

• Line and Col are 1-indexed.
• Line/Col stored on token is the start position of that token.
• EOF token should point at the position after the last character.
• when scanning consumes newline, line++ and col = 1.

Error model

Treat these as immediate lexer failures:

• unknown character
• unterminated string
• invalid escape sequence

Return errors with location context so tests can show exactly where scanning diverged.

Deep practice checkpoints (almost solved)

1) Longest match

Input: a==b and x<=y

Expected:

• tokens include == and <=, not two separate singles.

2) Comment boundaries

Input: let x = 1; // comment\nlet y = 2;

Expected:

• comment text is not emitted
• second statement starts at line 2, column 1.

3) Positioning around tokens

Input: ` let x = 1; `

Expected:

• let at line 2, col 3
• 1 at line 2, col 9
• EOF at line 2, col 11

4) String escapes

Input: "a\\n\\t\""

Expected:

• one token TokenString
• lexeme is actual a\n\t\" string content

5) Bad escapes

Input: "\x"

Expected:

• lexer error: invalid escape

6) Unterminated string

Input: "abc\n

Expected:

• lexer error: unterminated string

7) Unknown character

Input: let x = 1 $

Expected:

• error at $ with location

8) Keyword recognition edge

Input: let var const fn true false nil if else

Expected:

• every recognized word maps to its keyword kind, not identifier.

9) Number scanning

Input: 12a

Expected:

• number token 12, then identifier token a.

10) Punctuation set expectations

Input: ( ) { } , ;

Expected:

• all punctuation tokens emitted in order with each start column tracked.