Program Lexer v5

Implement a lexer for the v5 language (modules, imports, exports, member access).

Function signature

func Lex(src string) ([]Token, error)

Token kinds

Keywords

• let, var, const
• fn, if, else, while, for, return
• break, continue
• module, import, export, as
• true, false, nil

Identifiers

• /[A-Za-z_][A-Za-z0-9_]*/

Numbers

• /[0-9]+/ (decimal integers)

Strings

• Double-quoted: "..."
• Escapes: \", \\, \n, \t
• Newlines are not allowed inside a string
• Token Lexeme should be the unescaped string value (no quotes)

Operators

• + - * /
• = ==
• ! !=
• < <=
• > >=
• && ||

Punctuation

• ( ) { } [ ] , ; : .

End of file

• EOF

Whitespace and comments

• Skip spaces, tabs, and newlines.
• Line comments start with // and run to the end of the line.

Positions

Each token must record the line and column where the token starts.

• Lines start at 1.
• Columns start at 1.
• The EOF token uses the line/column after the last character.

Errors

Return a non-nil error for:

• unexpected characters
• unterminated strings
• invalid escape sequences

Notes

• Prefer readability over cleverness.
• This version introduces module keywords and member-access punctuation.

Before implementation

Use the module’s worked reconstruction and independent debugging lab to predict an intermediate artifact first. Preserve one failing counterexample while implementing the stated API. The lesson’s transfer task uses a fresh case after the guided trace.