Program Lexer v6

Implement a lexer for the v6 language (modules + type annotations).

Function signature

func Lex(src string) ([]Token, error)

New token

• -> should be tokenized as ARROW.

Rest of tokens are v5-compatible

Keywords

• let, var, const
• fn, if, else, while, for, return
• break, continue
• module, import, export, as
• true, false, nil

Identifiers

• /[A-Za-z_][A-Za-z0-9_]*/

Numbers

• /[0-9]+/ (decimal integers)

Strings

• Double-quoted: "..."
• Escapes: \", \\, \n, \t
• Newlines are not allowed inside a string
• Token Lexeme should be the unescaped string value (no quotes)

Operators

• + - * /
• = ==
• ! !=
• < <=
• > >=
• && ||
• ->

Punctuation

• ( ) { } [ ] , ; : .

End of file

• EOF

Whitespace and comments

• Skip spaces, tabs, and newlines.
• Line comments start with // and run to the end of the line.

Positions

Each token must record line/column where token starts.

• Lines and columns are 1-based.
• EOF position is the cursor after the last character.

Errors

Return a non-nil error for:

• unexpected characters
• unterminated strings
• invalid escapes

Notes

• Type names (number, bool, string, array, map) are lexed as identifiers.
• - is still minus by default; only -> gets ARROW.

Before implementation

Use the module’s worked reconstruction and independent debugging lab to predict an intermediate artifact first. Preserve one failing counterexample while implementing the stated API. The lesson’s transfer task uses a fresh case after the guided trace.