Lexing

Build reliable token stream

• deterministic char-level scanner
• skip whitespace and // comments
• emit all keywords/operators/punctuation explicitly

Longest-match rule

• check multi-char operators before single-char ones
• ==, !=, <=, >=

Lexemes and positions

• identifiers/keywords: [A-Za-z_] [A-Za-z0-9_]*
• numbers: [0-9]+
• strings: quoted with escapes and unescaped lexeme

Position contract

• 1-based line/column
• token Line/Col = token start
• EOF at location after final character

Error behavior

• unknown char
• invalid escape
• unterminated string

Practical parser-facing notes

• keyword mapping occurs after identifier scan
• parser depends on exact operator split (= vs ==, ! vs !=)