Go Binary Protocols: Framing, Varints, and Buffer Ownership

A stream delivers bytes, not messages. A read can stop halfway through a length, halfway through a payload, or after several frames. Your decoder must identify one complete frame without losing the bytes needed for a retry.

You should know Go slices, integer shifts, and error returns. After this module, you should be able to derive a varint length byte by byte, classify incomplete and malformed inputs, check bounds without integer overflow, and explain how long a returned payload remains valid.

One frame and one contract

The exercise format is [unsigned-varint length][payload bytes]. Length counts only payload bytes. DecodeFrame(buf,maxLen) decodes the first frame; trailing bytes belong to later frames. On success, consumed counts both the length prefix and payload. On every error, it returns nil,0,error and leaves input unchanged. There is no saved decoder state: retries pass the entire unconsumed buffer again.

The return errors describe different decisions:

Stage	Condition	Result
Length	Prefix can still become a valid uint64	ErrNeedMore
Length	Prefix cannot encode a uint64	ErrMalformedVarint
Limit	Complete length exceeds maxLen	ErrFrameTooLarge
Body	Valid allowed length exceeds bytes available	ErrNeedMore
Done	Whole first frame is present	payload slice and consumed count

A declared size over the limit is rejected even if its payload has not arrived. A partial varint is classified before checking the frame limit. maxLen is a nonnegative int in this exercise.

Derive unsigned varints

Each byte contributes seven low bits; the high bit says another byte follows. The groups are least significant first. At zero-based byte index i, add uint64(b & 0x7f) << (7*i) after checking the boundary rules.

Decode bytes AC 02 (hex):

Index	Byte	Continue bit	Seven data bits as decimal	Contribution	Accumulator
0	AC	1	44	44 << 0 = 44	44
1	02	0	2	2 << 7 = 256	300

Thus 300 bytes require a two-byte prefix, and a complete frame consumes 302 bytes. Converting in the other direction, 300 = 44 + 2*128: encode 44 with continuation (0xAC), followed by the final group 2 (0x02). A one-byte zero prefix describes a complete empty payload and still consumes one byte.

This format permits non-minimal encodings: 80 00 also means zero. Canonical encoding is a separate protocol rule, not an implicit overflow check here.

Why the tenth byte is special

Nine groups cover 63 bits. A uint64 has just one bit left for the tenth group, so byte 10 (index 9) must terminate and can only be 00 or 01.

Prefix	Meaning
Nine 80 bytes, no tenth byte	Incomplete: another byte could terminate it
Nine 80 bytes then 00	Valid non-minimal zero
Nine FF bytes then 01	Valid 2^64-1; still subject to maxLen
Nine 80 bytes then 02	Overflow; malformed
Ten 80 bytes	Malformed now; an eleventh byte cannot repair it

Do not let a shift silently discard overflow bits before validating byte 10. Go's encoding/binary.Uvarint returns a positive count on success, zero for incomplete input, and a negative count for overflow. Inspect its implementation when comparing a helper with the exercise's error mapping: an unterminated ten-byte prefix must be rejected immediately by this frame contract.

Trace an incremental receive buffer

Suppose a sender emits a 130-byte payload with prefix 82 01:

Bytes retained so far	Decoder state	Result and caller action
82	First group is 2, continuation set	Need more; retain that byte
82 01 plus 40 payload bytes	Length 130, prefix 2, body 40	Need more; retain all 42 bytes
82 01 plus 130 payload bytes	Whole frame available	Success; consume 132 bytes

Consuming the two prefix bytes at the middle row would lose the length on the next stateless call. Consuming zero means the caller can append new bytes and retry the same frame.

For 02 'h' 'i' 01 '!', the first call returns "hi", consumed 3. Once the caller has finished using that payload, it can decode buf[3:], which returns "!", consumed 2. A read boundary is never used as a frame boundary.

Bounds checks must remain valid at extreme lengths

Decode into uint64 first. After a complete length value is known, compare value > uint64(maxLen) before converting it to int. That proves conversion is representable on the current architecture.

If the prefix uses n bytes, test length > len(buf)-n. Prefix success already proves n <= len(buf), so subtraction is safe. The tempting check n+length > len(buf) can overflow when length is near the largest int, even though conversion itself was valid. Only calculate the end offset after the remaining-byte check succeeds.

For example, a valid prefix declaring MaxInt bytes with maxLen=MaxInt and no payload must return ErrNeedMore; it must not wrap an addition and panic while slicing. No enormous allocation is necessary to test this boundary.

Zero-copy means shared ownership of backing memory

On success, payload := buf[n:end] shares the backing array. Consider:

buf := []byte{2, 'h', 'i', 1, '!'}
payload, _, _ := DecodeFrame(buf, 10)
// payload currently contains "hi".
buf[1] = 'H'
// payload now contains "Hi": it was never an independent copy.

The reverse also holds: writing a payload byte changes buf. If the receive loop moves trailing bytes to the front with copy, reuses the array for a new read, or returns it to a pool, saved payload slices may change. Process them before reuse, transfer buffer ownership, or explicitly copy the payload before retaining it. Slicing buf = buf[consumed:] alone does not overwrite the old bytes, but later compaction or reuse can. Aliasing does not provide concurrency synchronization, and retaining a tiny slice can keep a large array alive.

The decoder cannot promise both zero-copy and independent long-lived payloads; that choice belongs to its caller.

Measure before choosing an optimization

Using encoding/binary is not inherently slower or allocating. Compare a correct custom parser and a helper-based parser on representative lengths, fragmented prefixes, incomplete bodies, and rejected sizes. Adapt error classification before comparing results.

Allocate test inputs outside the timed loop, use a package-level sink so work remains observable, call b.ReportAllocs(), and run a benchmark repeatedly:

go test -run '^$' -bench DecodeFrame -benchmem -count=5

Use this command in the implementation's test package; the exercise's AllocsPerRun check enforces no decoder allocations on successful input. Report both time and allocations with the input size and Go version. Zero allocations does not prove a parser faster, and a deliberate ownership copy in the caller can be the correct tradeoff.

Practice before the exercise

1. Decode 85 01. What is the payload length? How many total bytes would a complete frame consume?
2. With maxLen=4, what does [05] return? What about [80]?
3. A parser returns an alias then its caller overwrites the receive array. Which component must preserve data needed beyond that overwrite?

Check your reasoning

1. 5 + 128 = 133; with a two-byte prefix, total consumption is 135.
2. [05] is a complete length exceeding the limit: ErrFrameTooLarge. [80] is an incomplete prefix: ErrNeedMore. Both consume zero.
3. The caller must copy or preserve ownership of the backing memory. A successful parse alone supplies no independent storage lifetime.

Now implement DecodeFrame, then take the checkpoint using fresh byte streams. If incomplete and malformed inputs are confused, revisit the tenth-byte table; if a retained payload changes, trace who owns and writes its backing array.