Program Lexer v3

Implement a lexer for the v3 language (arrays, maps, nested functions).

Function signature

func Lex(src string) ([]Token, error)

Token kinds

Keywords

• let, var, const
• fn, if, else, while, return
• true, false, nil

Identifiers

• /[A-Za-z_][A-Za-z0-9_]*/

Numbers

• /[0-9]+/ (decimal integers)

Strings

• Double-quoted: "..."
• Escapes: \", \\, \n, \t
• Newlines are not allowed inside a string
• Token Lexeme should be the unescaped string value (no quotes)

Operators

• + - * /
• = ==
• ! !=
• < <=
• > >=

Punctuation

• ( ) { } [ ] , ; :

End of file

• EOF

Whitespace and comments

• Skip spaces, tabs, and newlines.
• Line comments start with // and run to the end of the line.

Positions

Each token must record line/column where the token starts.

• Lines start at 1.
• Columns start at 1.
• EOF uses position after the last character.

Errors

Return a non-nil error for:

• unexpected characters
• unterminated strings
• invalid escape sequences

Notes

• Prefer readability over cleverness.
• deterministic tokenization keeps later parser behavior stable.

Before implementation

Use the module’s worked reconstruction and independent debugging lab to predict an intermediate artifact first. Preserve one failing counterexample while implementing the stated API. The lesson’s transfer task uses a fresh case after the guided trace.