Capstone: a parser that can explain every byte

A parser that prints plausible fields can still be wrong: one lost byte shifts every field that follows it. Build a parser whose cursor you can justify, whose errors stop at the first unknown boundary, and whose diagnostic views reveal only the fields their display policy allows.

This capstone combines bitmaps and field encoding. You should already be able to derive a field's bit position and distinguish ASCII decimal lengths from packed BCD digits. If either step is uncertain, revisit those modules first.

The task is a library exercised by tests, despite its historical iso-parser-cli slug. Build the supplied functions; a terminal program is optional transfer work. The two built-in profiles are synthetic course examples. VisaSpec is a legacy API name for the hex-ASCII bitmap profile, not an authoritative Visa specification.

Begin with the actual interfaces

The supplied MessageSpec contains a bitmap encoding and a field map. FieldSpec contains Number, Name, LengthType, MaxLength, DataType and Encoding. There is no LengthEncoding, MTIEncoding or padding-policy field. This course therefore fixes MTI to four ASCII digits, variable length prefixes to ASCII decimal digits, and odd BCD values to leading-zero padding.

Real libraries expose richer configuration: pyiso8583's field specification separates data encoding, length encoding, length units and padding. Extending a course schema would be a separate API change. Do not write sample code that accesses configuration members absent from the learner's starter.

Parse(hexStr) receives a hex representation of the entire wire message. It hex-decodes once. The resulting wire bytes contain an ASCII MTI, the configured bitmap representation, and the field values. Bitmap encoding is chosen by the profile, never guessed from whether the first bytes happen to be printable.

Work one complete cursor trace

Use this synthetic wire message: MTI 0100, primary bitmap 70 00 00 00 00 00 00 00, DE2=1234567890123456, DE3=000000, DE4=000000001234. The first bitmap byte is 01110000, setting fields 2,3,4.

wire := append([]byte("0100"), []byte{0x70, 0, 0, 0, 0, 0, 0, 0}...)
wire = append(wire, []byte("16" + "1234567890123456" + "000000" + "000000001234")...)
hexInput := hex.EncodeToString(wire)
// Parse(hexInput), not Parse("0100" + printedBitmap + fieldText).

Stage	Wire range, half-open	Value	Next cursor
ASCII MTI	[0,4)	0100	4
Binary primary bitmap	[4,12)	fields 2,3,4	12
DE2 ASCII length	[12,14)	16 → sixteen bytes	14
DE2 value	[14,30)	sixteen PAN digits	30
DE3 fixed value	[30,36)	six bytes	36
DE4 fixed value	[36,48)	twelve minor-unit digits	48

The wire length is 48 bytes; the outer hex string has 96 characters. RawHex for DE2 represents its value bytes only, excluding its two-byte length prefix.

Under the hex-ASCII bitmap profile, replace the eight bitmap bytes with the sixteen ASCII characters 7000000000000000. The bitmap ends at wire offset 20; all field ranges shift by eight, and the wire length becomes 56. The outer hex input now contains 112 characters. The bitmap's first character 7 is wire byte 37 in hex; the two characters 70 decode to bitmap byte 70 in hex. These are two different representation layers.

If bit 1 is set, read the secondary bitmap before any field. Bit 1 remains in the reported bitmap field list as an indicator; it is not parsed as a data field. The primary and secondary together describe fields up to 128 in this course.

Derive length units before slicing

For ASCII and BINARY fields, MaxLength and the length prefix count bytes. For BCD, they count decimal digits. A length of five BCD digits occupies 5/2 + 5%2 = 3 bytes. Computing this as (length+1)/2 can overflow if a caller supplies an extreme fixed-length specification.

Example: a custom LLVAR BCD field holding 12345 is:

30 35 | 01 23 45
 05      five digits, leading zero pad

Consume two prefix bytes and three payload bytes. The result is 12345, not 01234; stripping the last nibble loses the last data digit. Every non-padding nibble must be decimal 0..9, and this course requires the leading pad itself to be zero. 05 | 11 23 45 fails that padding policy.

Before slicing, validate prefix width, decimal digits, configured maximum and remaining bytes. -1 and +1 are not two decimal digits even if a general integer parser accepts them. Failed fields consume nothing and are not published in the result map.

Stop when the next boundary cannot be proved

Maintain this invariant: pos is the first unconsumed wire byte, and every published field has a complete validated framing interval before pos.

Suppose fields 2,3,4 are present, but the profile omits DE3. After DE2, pos=30. The parser cannot infer how many bytes DE3 occupies. Skipping DE3 in the bitmap without consuming bytes mislabels its first bytes as DE4. Instead return the partial message with a non-nil error naming field 3 and offset 30. DE4 must remain absent. A warning alone is not successful parsing.

The same stop rule applies to truncated values, malformed length prefixes, unsupported encodings and invalid BCD. Reject trailing bytes after the last expected field: this API parses one complete message, not a stream. A streaming extension would need framing plus a consumed-byte return value.

Header errors can return nil message. After the header is valid, preserve only completed fields for diagnosis and record the error in Warnings. Callers may inspect that partial message, but err != nil means the parse is incomplete. Never fabricate missing values or call a partial message transaction-valid.

A syntactically complete parse still does not establish a MAC, issuer approval, field business semantics, or whether a duplicate operation should be applied. Those are separate validators using the selected profile and transaction state.

Keep raw data separate from diagnostic views

The in-memory ParsedField retains exact value bytes as RawHex and its decoded representation. PrettyPrint and ToJSON are lossy diagnostic views; neither is a serialization for retransmission or MAC calculation.

Both views apply one policy:

• DE2: mask middle digits, retaining the first six and last four only when its length exceeds ten; shorter values are fully masked. Always redact its RawHex.
• DE52: show [ENCRYPTED PIN DATA], and redact RawHex.
• The explicit ordinary-field allowlist in the statement may display values and raw hex. Amount fields retain their original digits labeled as minor units; no currency exponent is present in this API.
• All remaining fields, including track data, ICC/private containers and unknown fields, show [REDACTED] for both raw and decoded forms.

Masking only a decoded PAN while printing its raw hex reveals the same digits in another representation. JSON must follow the same policy as text. Redaction must not mutate the parsed object: a later authorized byte-level operation may need the original bytes, and display is not a parser transformation.

This is the exercise's explicit display contract, not a compliance certification or permission to put real payment records into logs. Use synthetic fixtures. Likewise, 000000001234 cannot universally become 12.34: the currency's configured exponent is needed to interpret its scale.

Build in stages with decreasing support

1. Header checkpoint: parse the worked binary message's MTI and bitmap only. Show cursor 12 and fields [2,3,4]. Repeat with the hex-ASCII profile.
2. Field checkpoint: implement fixed and variable widths, then the five-digit BCD case. Return both parsed value and bytes consumed from your helper.
3. Integration checkpoint: populate fields only after successful framing. Remove DE3 from the profile and prove that DE4 cannot be accidentally emitted.
4. Display checkpoint: produce text and JSON from one parsed message without changing its raw fields. Search both outputs for sensitive plaintext and hex.
5. Independent implementation: complete the capstone exercise, including fresh malformed cases. Keep diagnostic assertions separate from successful parse assertions.

Transfer and diagnosis

Predict these before revealing the answers:

1. With binary bitmap and only fixed DE3, what is the wire length and outer hex length? DE3 contains six ASCII bytes.
2. A custom field has BCD length 7 and a two-byte prefix. How many bytes does its helper consume? What must the first high nibble be?
3. A parser returns partial fields and nil error after a truncated value. Why is that insufficient even if its Warnings array is nonempty?
4. A JSON formatter masks decoded but retains DE2's RawHex. Has it redacted the PAN?

Check your derivation

1. 4+8+6=18 wire bytes and 36 hex characters.
2. 2+ceil(7/2)=6 bytes; the pad nibble must be zero.
3. Callers using the normal error contract can accept incomplete messages.
4. No. Decoding hex recovers the same digits. Redact both representations.

Design variation: add a transport length prefix outside this message format. Specify whether the prefix counts itself, whether it counts wire bytes or hex characters, and how an incomplete frame differs from an invalid complete frame. Then explain why the current Parse signature cannot report consumption for two concatenated messages. Use the checkpoint to diagnose boundaries before progressing to the advanced exercises.