Module 8: Capstone Parser - Slides
1 / 29
Your phone rings at 3:17 AM. Transactions are failing.
The log shows:
0100F23A448108E180001654321012345678900000000000010000001229143045
123456789012TERMINAL1MERCH12345678901234567MERCHANT NAME AND CITY US840
Questions racing through your mind:
- What fields are present?
- Is the bitmap correct?
- What's the amount?
- Why is it failing?
A good parser turns that into answers in seconds.
Module 1: MTI → "0100" = Auth Request
Module 2: Bitmaps → Which fields are present
Module 3: Encoding → How to read each field
Module 4: Key Fields → PAN, Amount, Track 2, Response Codes
Module 5: Flows → Request/Response matching
Module 6: Security → MAC and PIN protection
Module 7: Visa → Network-specific variations
NOW: Put it ALL together into a production tool.
INPUT OUTPUT
│ ▲
▼ │
┌───────────────────────────────────────────────────┐
│ PIPELINE │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
│ │ INPUT │ │ MESSAGE │ │ FIELD │ │
│ │DETECTION │─▶│ FRAMING │─▶│ EXTRACTION │ │
│ └──────────┘ └──────────┘ └────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐│
│ │ OUTPUT ││
│ │ FORMATTING ││
│ └──────────────────┘│
└───────────────────────────────────────────────────┘
type ParsedMessage struct {
RawHex string // Original input
RawBytes []byte // Decoded bytes
MTI MTI // Parsed message type
Bitmap Bitmap // Field presence
Fields map[int]ParsedField // All data elements
Spec *MessageSpec // Which spec was used
Warnings []string // Non-fatal issues
}
type ParsedField struct {
Number int // DE2, DE4, etc.
RawBytes []byte // Wire format
Decoded string // Human-readable
SubFields map[string]ParsedField // For TLV fields
}
// The key insight: parsing logic should be DATA, not CODE
type MessageSpec struct {
Name string // "Visa BASE I"
BitmapEncoding string // "BINARY" or "HEX_ASCII"
MTIEncoding string // "ASCII" or "BCD"
LengthEncoding string // "ASCII" or "BCD"
Fields map[int]*FieldSpec // Field definitions
}
// Adding a new network = adding a new spec
// NOT changing parser code
var VisaBASE1 = &MessageSpec{
Name: "Visa BASE I",
BitmapEncoding: "HEX_ASCII", // Different from ISO base!
Fields: visaFieldSpecs,
}
type FieldSpec struct {
Number int // 2, 4, 11, etc.
Name string // "Primary Account Number"
LengthType string // "FIXED", "LLVAR", "LLLVAR"
MaxLength int // Maximum allowed length
DataType string // "N", "AN", "ANS", "B", "Z"
Encoding string // "ASCII", "BCD", "BINARY"
TLVFormat string // "SIMPLE", "EMV", "NONE"
}
// Example: DE2 (PAN)
{
Number: 2,
Name: "Primary Account Number",
LengthType: "LLVAR", // 2-byte length prefix
MaxLength: 19, // Max 19 digits
DataType: "N", // Numeric only
Encoding: "ASCII", // ASCII digits
}
func DetectInputFormat(input []byte) InputFormat {
// Is it a hex string?
if allHexChars(input) && len(input) % 2 == 0 {
return FormatHexString
}
// Does it have a length header?
if len(input) >= 2 {
binLen := int(input[0])<<8 | int(input[1])
if binLen == len(input) - 2 {
return FormatBinaryWithLength
}
}
// Raw binary
return FormatBinary
}
// Auto-detect means: NEVER ask "what format is this?"
// Just figure it out.
Binary bitmap (8 bytes):
72 3A 44 81 08 E1 80 00
└─ 8 raw bytes, any value 0x00-0xFF
Hex-ASCII bitmap (16 characters):
"723A448108E18000"
└─ 16 printable hex chars (0-9, A-F)
Detection:
if first 16 bytes are all [0-9A-Fa-f]:
return "HEX_ASCII"
else:
return "BINARY"
Visa uses HEX_ASCII. Most others use BINARY.
func parseMTI(data []byte, encoding string) (MTI, int, error) {
var mti MTI
switch encoding {
case "ASCII":
mti.Raw = string(data[0:4]) // "0100"
return mti, 4, nil // consumed 4 bytes
case "BCD":
// 2 bytes = 4 digits
mti.Raw = fmt.Sprintf("%02X%02X", data[0], data[1])
return mti, 2, nil // consumed 2 bytes
}
// Parse components
mti.Version = int(mti.Raw[0] - '0') // 0 = 1987
mti.Class = int(mti.Raw[1] - '0') // 1 = Auth
mti.Function = int(mti.Raw[2] - '0') // 0 = Request
mti.Origin = int(mti.Raw[3] - '0') // 0 = Acquirer
return mti, consumed, nil
}
func parseBitmap(data []byte, encoding string) (Bitmap, int, error) {
var bitmap Bitmap
var consumed int
// Parse primary (always present)
if encoding == "BINARY" {
bitmap.Primary = data[0:8]
consumed = 8
} else { // HEX_ASCII
bitmap.Primary, _ = hex.DecodeString(string(data[0:16]))
consumed = 16
}
// Check for secondary (bit 1 = 1)
if bitmap.Primary[0] & 0x80 != 0 {
// Parse secondary...
consumed += bitmapSize
}
// Extract field list
bitmap.Fields = extractFields(bitmap)
return bitmap, consumed, nil
}
For field N (1-64):
byteIndex = (N - 1) / 8
bitIndex = 7 - ((N - 1) % 8)
present = bitmap[byteIndex] & (1 << bitIndex) != 0
Example: Is field 11 present?
byteIndex = (11-1) / 8 = 1
bitIndex = 7 - ((11-1) % 8) = 7 - 2 = 5
Check: bitmap[1] & 0x20 (bit 5 = 0b00100000)
For fields 65-128: Same formula on secondary bitmap
// FIXED: Length is constant, read directly
//
// Example: DE4 (Amount) - fixed 12 bytes
spec := FieldSpec{
LengthType: "FIXED",
MaxLength: 12,
}
// Just read MaxLength bytes
value := data[0:12] // "000000010000"
// Decode based on DataType
decoded := string(value) // "000000010000" = $100.00
// LLVAR: 2-byte ASCII length prefix
// LLLVAR: 3-byte ASCII length prefix
func parseVariable(data []byte, prefixLen int) ([]byte, int) {
// Read length prefix
lengthStr := string(data[0:prefixLen])
length, _ := strconv.Atoi(lengthStr)
// Read value
value := data[prefixLen : prefixLen + length]
// Total consumed = prefix + value
return value, prefixLen + length
}
// Example: DE2 (PAN) = "165432101234567890"
// ^^ = 16 (length)
// ^^^^^^^^^^^^^^^^ = value (16 chars)
BCD (Binary Coded Decimal):
Each nibble = one digit (0-9)
2 digits per byte
ASCII "123456" (6 bytes):
31 32 33 34 35 36
BCD "123456" (3 bytes):
12 34 56
For odd-length, left-pad with 0:
"12345" → 01 23 45 (3 bytes)
BCD length prefix:
15 → 0x15 (1 byte, not "15" = 2 ASCII bytes)
═══════════════════════════════════════════════════════
ISO8583 Message Parser - ISO8583:1987 Base
═══════════════════════════════════════════════════════
MTI: 0100
Version: 0 (ISO 8583:1987)
Class: 1 (Authorization)
Function: 0 (Request)
Origin: 0 (Acquirer)
Bitmap: F23A448108E18000
Fields present: [2, 3, 4, 7, 11, 12, 14, 22, 23, 25, 26, 32, 35, 37, 41, 42, 43, 49]
Data Elements:
───────────────────────────────────────────────────────
DE2 Primary Account Number
Raw: 35343332313031323334353637383930
Decoded: 543210******7890 (masked)
DE4 Amount, Transaction
Raw: 303030303030303130303030
Decoded: 100.00
func formatDecodedValue(field ParsedField) string {
switch field.Number {
case 2: // PAN - mask for PCI compliance
return maskPAN(field.Decoded) // "4321****8901"
case 4, 5, 6: // Amounts - add decimal point
return formatAmount(field.Decoded) // "100.00"
case 7: // DateTime - human readable
return formatDateTime(field.Decoded) // "Dec 29, 14:30:45"
case 39: // Response Code - add meaning
return field.Decoded + " (" + responseCodeMeaning(field.Decoded) + ")"
// "51 (Insufficient funds)"
case 49, 50, 51: // Currency - add name
return field.Decoded + " (" + currencyName(field.Decoded) + ")"
// "840 (USD)"
case 52: // PIN Block - NEVER show
return "[ENCRYPTED PIN DATA]"
}
return field.Decoded
}
{
"mti": {
"raw": "0100",
"version": 0,
"class": 1,
"function": 0,
"origin": 0
},
"bitmap": {
"primary": "f23a448108e18000",
"fields": [2, 3, 4, 7, 11, 12, 14, 22, 23, 25, 26, 32, 35, 37, 41, 42, 43, 49]
},
"fields": {
"2": {"name": "Primary Account Number", "decoded": "5432101234567890"},
"4": {"name": "Amount, Transaction", "decoded": "000000010000"},
"39": {"name": "Response Code", "decoded": "00"}
}
}
// Production messages are messy. Your parser MUST:
// 1. Recover from truncation
if pos >= len(data) {
msg.Warnings = append(msg.Warnings,
fmt.Sprintf("truncated at field %d", fieldNum))
break // Return partial result, don't crash
}
// 2. Skip unknown fields
if _, ok := spec.Fields[fieldNum]; !ok {
msg.Warnings = append(msg.Warnings,
fmt.Sprintf("no spec for field %d", fieldNum))
continue
}
// 3. Report what was parsed
msg.BytesParsed = pos
return msg, nil // Not error - partial success
func ValidateMessage(msg *ParsedMessage) []ValidationError {
var errors []ValidationError
// Required fields for 0100
if msg.MTI.Raw == "0100" {
for _, f := range []int{2, 3, 4, 11} {
if _, ok := msg.Fields[f]; !ok {
errors = append(errors, ValidationError{
Field: fmt.Sprintf("DE%d", f),
Message: "required for authorization request",
})
}
}
}
// PAN Luhn check
if pan, ok := msg.Fields[2]; ok {
if !luhnValid(pan.Decoded) {
errors = append(errors, ValidationError{
Field: "DE2",
Message: "PAN fails Luhn check",
})
}
}
return errors
}
For debugging, annotate the raw hex:
30313030 <-- MTI: 0100
723A448108E18000 <-- Bitmap: [2,3,4,7,11,12...]
1654321012345678 <-- DE2: PAN (16 digits)
303030303030 <-- DE3: Processing Code
303030303030303130 <-- DE4: Amount
...
This shows EXACTLY where each field starts/ends.
Invaluable for "why doesn't this parse?" debugging.
func CompareMessages(msg1, msg2 *ParsedMessage) string {
// MTI changed?
if msg1.MTI.Raw != msg2.MTI.Raw {
output += fmt.Sprintf("MTI: %s -> %s\n", msg1.MTI.Raw, msg2.MTI.Raw)
}
// Fields removed?
for num := range msg1.Fields {
if _, ok := msg2.Fields[num]; !ok {
output += fmt.Sprintf("Field %d: REMOVED\n", num)
}
}
// Fields added?
for num := range msg2.Fields {
if _, ok := msg1.Fields[num]; !ok {
output += fmt.Sprintf("Field %d: ADDED\n", num)
}
}
// Fields changed?
for num, f1 := range msg1.Fields {
if f2, ok := msg2.Fields[num]; ok && f1.Decoded != f2.Decoded {
output += fmt.Sprintf("Field %d: %s -> %s\n", num, f1.Decoded, f2.Decoded)
}
}
}
1. Off-by-one in bitmap fields
WRONG: fields[0] (doesn't exist)
RIGHT: fields[1] (first field is 1)
2. BCD vs ASCII length prefixes
"15" = 2 ASCII bytes OR 0x15 = 1 BCD byte
Check spec.LengthEncoding!
3. Binary fields as ASCII
DE52 (PIN) is BINARY, not printable ASCII
Display as hex: fmt.Sprintf("%X", data)
4. Forgetting padding
DE4 = "10000" but fixed length is 12
Must be: "000000010000" (left-padded zeros)
5. Network-specific bitmaps
ISO8583: BINARY bitmap
Visa: HEX-ASCII bitmap
Auto-detect or configure!
# Parse hex string
$ iso-parser 0100F23A...
# Parse from file
$ iso-parser --file message.hex
# Different specs
$ iso-parser --spec visa 0100723A...
$ iso-parser --spec mastercard 0200...
# Different outputs
$ iso-parser --output json 0100...
$ iso-parser --output table 0100...
# With validation
$ iso-parser --validate 0100...
# Compare two messages
$ iso-parser compare --file1 request.hex --file2 response.hex
// For high-throughput parsing:
// 1. Reuse buffers
type ParserPool struct {
parsers sync.Pool
}
// 2. Avoid string allocations
// Keep as []byte until output needed
// 3. Zero-copy field access
func GetFieldBytes(msg *Message, num int) []byte {
return msg.RawBytes[field.Offset:field.Offset+field.Length]
}
// Benchmarks:
// Simple message (10 fields): < 10µs
// Complex message (40 fields): < 50µs
// With validation: < 100µs
┌─────────────────────────────────────────────────────────────────┐
│ PRODUCTION ISO8583 PARSER │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Input: Hex string, binary file, TCP stream │
│ │
│ Detection: Auto-detect format, encoding, network │
│ │
│ Parsing: MTI → Bitmap → Fields (per spec) │
│ │
│ Output: Pretty-print, JSON, table, annotated hex │
│ │
│ Extras: Validation, comparison, TLV expansion │
│ │
├─────────────────────────────────────────────────────────────────┤
│ │
│ When it works, you have a tool that: │
│ • Debugs production issues at 3 AM │
│ • Validates your implementation against test vectors │
│ • Trains new engineers on message structure │
│ • Adapts to any network with new specs │
│ │
└─────────────────────────────────────────────────────────────────┘
1. SPECIFICATION IS KING
The field table is your source of truth.
Everything flows from the spec.
2. DETECT BEFORE ASSUME
Auto-detect format, encoding, network.
Never require the user to tell you.
3. FAIL GRACEFULLY
Parse what you can.
Report what you couldn't.
Never crash.
4. OUTPUT MATTERS
Pretty-printing is user experience.
Make it readable, make it useful.
5. VALIDATE THOROUGHLY
Catch errors before they propagate.
Luhn checks, required fields, format checks.
6. TEST WITH REAL DATA
Production messages are messy.
Your test cases should be too.
Build iso-parser-cli:
1. Parse any ISO8583 message from hex input
2. Auto-detect format (hex/binary, ASCII/BCD bitmap)
3. Extract all fields using specification
4. Pretty-print with field names and decoded values
5. Handle errors gracefully with meaningful messages
6. Support multiple output formats (pretty, JSON)
This is the culmination of everything you've learned.
Every module builds to this moment.
Good luck. Your future self (debugging at 3 AM) will thank you.
MTI Structure: [Version][Class][Function][Origin]
0100 = 1987/Auth/Request/Acquirer
Bitmap Formula: byte = (N-1)/8, bit = 7-((N-1)%8)
Length Types: FIXED = read MaxLength bytes
LLVAR = 2-byte length + data
LLLVAR = 3-byte length + data
Encoding: ASCII = printable characters
BCD = 2 digits per byte
BINARY = raw bytes
Critical Fields: DE2 = PAN (mask it!)
DE4 = Amount (12 digits, cents)
DE11 = STAN (matching key)
DE39 = Response code
DE52 = PIN (never log!)
Validation: Luhn on DE2
Required fields per MTI
Format checks per DataType