Capstone: full pipeline wiring from source to VM

Why this capstone is the real project boundary

By now we have many pieces:

• lexer and parser
• semantic checks
• code generator
• bytecode serializer/deserializer (BC1)
• VM executor

This module is where those pieces stop being independent exercises and start being a single reproducible compiler toolchain.

The engineering point is: each stage has a strict interface and a failure policy.

1) One clear ownership model

Compile owns source transformation. Run owns execution.

• Compile must never execute user code
• Run must never read parser/AST internals

That split enables:

• independent testing
• replacing serializer or VM without touching analyzer contracts
• deterministic replay from BC1

Think of it as two teams:

1. a front-end team that only emits bytecode
2. a runtime team that only consumes bytecode

CompileAndRun glues them only at the artifact boundary.

2) Pipeline contract and data shapes

Canonical shape:

1. Source text
2. Tokens
3. AST
4. Typed/validated program graph
5. Bytecode object
6. BC1 text (or any serial form)
7. Decoded bytecode object
8. Runtime result or runtime error

Each edge is typed:

• no hidden “side-channel” information
• no stage reads data from non-neighbor stages
• no skipping of required transforms

If any stage cannot produce its required output, fail immediately and return that diagnostic.

3) Failure ordering (deterministic short-circuit)

When multiple things are wrong, this order must win:

1. Lex error
2. Parse error
3. Semantic/analyzer errors (first one in deterministic order)
4. Compile errors
5. Serialize/encode errors
6. Deserialize/decode errors
7. Runtime errors

Why this order matters:

• users and grader outputs become predictable
• tests can assert exact stage ownership
• you avoid masking parse bugs with runtime noise

Even if analyzer can compute many diagnostics, the public behavior here is a single first error from that stage to keep output stable.

4) Idempotence and round-trip guarantees

For this pack, treat BC1 as a stable format contract:

• compile + encode should produce deterministic text for identical source
• decode must reconstruct an equivalent bytecode object
• encode(decode(encode(program))) should be idempotent in behavior

Important distinction:

• equivalence is behavioral for execution
• not necessarily byte-for-byte canonical formatting unless your encoder is strict

Debugging strategy:

1. compare tokens
2. compare ASTs
3. compare opcode lists
4. compare BC1
5. compare decoded program behavior

Each boundary gets its own test, so faults stay local.

5) Reproducible execution and side-effect boundaries

CompileAndRun is expected to:

• run on decoded bytecode only
• not depend on compile-time-only state like symbol tables after stage handoff
• avoid hidden global state between calls

This lets two runs over same BC1 yield same result, and makes your grader resilient to random map iteration and nondeterministic serialization.

6) Practical checkpoints (almost solved)

1) Front-end first

Source: let x = ;
Expected:

• lexical/parsing stage returns an error first
• no analyzer/compile/runtime output

2) Semantic gate

Source: return 1;
Expected:

• parse succeeds
• analyzer returns invalid return-context error
• no compile, encode, or VM execution

3) BC1 round-trip correctness

Source: let x = 1;
Expected flow:

• compile emits BC1
• decode reconstructs bytecode
• VM executes decoded bytes, not in-memory compiler artifact
• result remains same across both executions

4) Deterministic short-circuit

If analyzer reports [e1, e2, e3], CompileAndRun returns only e1 plus stage location.

5) Separation proof (bad serializer)

If serializer writes malformed BC1:

• decode must fail
• runtime must not see compiler internals directly

6) Deterministic outputs

Running same source twice:

• BC1 text unchanged
• decoded artifact stable
• runtime stack behavior identical

This module is not “big project” fluff; it is discipline.