Program Lexer v5

Implement a lexer for the v5 language (modules, imports, exports, member access).

Function signature

func Lex(src string) ([]Token, error)

Token kinds

Keywords

• let, var, const
• fn, if, else, while, for, return
• break, continue
• module, import, export, as
• true, false, nil

Identifiers

• /[A-Za-z_][A-Za-z0-9_]*/

Numbers

• /[0-9]+/ (decimal integers)

Strings

• Double-quoted: "..."
• Escapes: \", \\, \n, \t
• Newlines are not allowed inside a string
• Token Lexeme should be the unescaped string value (no quotes)

Operators

• + - * /
• = ==
• ! !=
• < <=
• > >=
• && ||

Punctuation

• ( ) { } [ ] , ; : .

End of file

• EOF

Whitespace and comments

• Skip spaces, tabs, and newlines.
• Line comments start with // and run to end of line.

Positions

Each token must record the line and column where it starts.

• Lines start at 1.
• Columns start at 1.
• The EOF token uses the line/column after the last character.

Errors

Return a non-nil error for:

• unexpected characters
• unterminated strings
• invalid escape sequences

Notes

• Prefer readability over cleverness.