Capstone - ISO8583 CLI Parser Tool
Lesson, slides, and applied problem sets.
View SlidesLesson
Module 8: Capstone - Building a Production ISO8583 Parser
The 3 AM Test
It's 3:17 AM. Your phone is buzzing. Transactions are failing. The on-call engineer is staring at a hex dump in the logs:
0100F23A448108E180001654321012345678900000000000010000001229143045
123456789012TERMINAL1MERCH12345678901234567MERCHANT NAME AND CITY US840
They need to know: What fields are present? Is the bitmap correct? Is DE4 properly formatted? Is there a secondary bitmap? What's in DE43?
A good ISO8583 parser turns that impenetrable string into:
Message Type: 0100 (Authorization Request)
Bitmap: F23A448108E18000 (Primary only)
Fields present: 2, 3, 4, 7, 11, 12, 14, 22, 23, 25, 26, 32, 35, 37, 41, 42, 43, 49
DE2 (PAN): 4321012345678900 (LLVAR, 16 digits)
DE3 (Processing Code): 000000
DE4 (Amount): 000000010000 ($100.00 USD)
DE7 (Transmission DateTime): 1229143045
DE11 (STAN): 123456
...
DE43 (Merchant Name): MERCHANT NAME AND CITY US
DE49 (Currency Code): 840 (USD)
That parser is what you're building in this capstone. Not a toy. A tool you'd actually use in production.
Why Build a Parser?
You've learned the pieces:
- Module 1: MTI structure and message anatomy
- Module 2: Bitmap manipulation and field presence
- Module 3: Field encoding (fixed, LLVAR, LLLVAR, BCD, ASCII)
- Module 4: Key data elements (PAN, amounts, dates, Track 2)
- Module 5: Transaction flows and message matching
- Module 6: MAC and PIN block security
- Module 7: Visa-specific adaptations
Now you integrate them. Building a parser forces you to confront every edge case, every encoding quirk, every ambiguity in the specification. You can't fake understanding—the parser either works or it doesn't.
And when it works, you have a tool that:
- Debugs production issues: Parse any message from logs
- Validates implementations: Check your code against test messages
- Learns new networks: Adapt field specs for different processors
- Trains new engineers: Visual understanding of message structure
Part 1: Architecture of a Production Parser
The Pipeline Model
A robust parser is a pipeline:
Raw Input Structured Output
──────── ─────────────────
│ ▲
▼ │
┌─────────────────────────────────────────────────────────────┐
│ PARSER PIPELINE │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────┐ ┌────────────┐ ┌────────────────────┐ │
│ │ INPUT │ │ MESSAGE │ │ FIELD │ │
│ │ DETECTION │──▶│ FRAMING │──▶│ EXTRACTION │ │
│ └────────────┘ └────────────┘ └──────────┬─────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌────────────┐ ┌────────────┐ ┌────────────────────┐ │
│ │ Hex/Binary │ │ MTI + │ │ Per-field decode │ │
│ │ detection │ │ Bitmap │ │ based on spec │ │
│ └────────────┘ └────────────┘ └──────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────┐ │
│ │ OUTPUT │ │
│ │ FORMATTING │ │
│ └────────────────────┘ │
│ │ │
└───────────────────────────────────────────────┼─────────────┘
│
▼
Pretty-printed
Output
Each stage has one job. Failures at any stage produce meaningful errors, not crashes.
The Data Structures
// ParsedMessage is the output of the parser
type ParsedMessage struct {
// Raw data
RawHex string // Original input (for reference)
RawBytes []byte // Decoded bytes
// Message structure
MTI MTI // Parsed MTI
Bitmap Bitmap // Primary (and optional secondary/tertiary)
Fields map[int]ParsedField // Parsed data elements
// Metadata
Spec *MessageSpec // Specification used for parsing
Warnings []string // Non-fatal issues encountered
BytesParsed int // How many bytes were consumed
}
// MTI contains the parsed Message Type Indicator
type MTI struct {
Raw string // "0100"
Version int // 0 = 1987, 1 = 1993, 2 = 2003
Class int // 1 = auth, 2 = financial, 4 = reversal, 8 = network
Function int // 0 = request, 1 = response, 2 = advice...
Origin int // 0 = acquirer, 1 = acquirer repeat...
}
// Bitmap tracks which fields are present
type Bitmap struct {
Primary []byte // 8 bytes (always present)
Secondary []byte // 8 bytes (if bit 1 of primary = 1)
Tertiary []byte // 8 bytes (if bit 1 of secondary = 1, 2003 spec)
Fields []int // List of present field numbers
}
// ParsedField holds a single data element
type ParsedField struct {
Number int // Field number (2-128)
RawBytes []byte // Raw bytes from message
RawHex string // Hex representation
Decoded string // Human-readable value
Spec *FieldSpec // Specification used
// For nested fields (TLV containers like DE48, DE55, DE63)
SubFields map[string]ParsedField
}
// FieldSpec defines how to parse a single field
type FieldSpec struct {
Number int
Name string
LengthType string // "FIXED", "LLVAR", "LLLVAR", "LLLLVAR"
MaxLength int
DataType string // "N", "AN", "ANS", "B", "Z"
Encoding string // "ASCII", "BCD", "BINARY", "HEX"
Description string
// For TLV fields
TLVFormat string // "SIMPLE", "EMV", "NONE"
SubSpecs map[string]*SubFieldSpec // For parsing nested TLV
}
Configuration-Driven Design
The key insight: parsing logic should be data, not code.
// MessageSpec defines how to parse an entire message type
type MessageSpec struct {
Name string // "ISO8583:1987 Base"
BitmapEncoding string // "BINARY" or "HEX_ASCII"
MTIEncoding string // "ASCII" or "BCD"
LengthEncoding string // "ASCII" or "BCD"
Fields map[int]*FieldSpec // Field specifications
}
// You can swap specs at runtime:
var ISO1987Base = &MessageSpec{
Name: "ISO8583:1987 Base",
BitmapEncoding: "BINARY",
MTIEncoding: "ASCII",
LengthEncoding: "ASCII",
Fields: iso1987FieldSpecs,
}
var VisaBASE1 = &MessageSpec{
Name: "Visa BASE I",
BitmapEncoding: "HEX_ASCII", // Visa uses hex-ASCII bitmaps!
MTIEncoding: "ASCII",
LengthEncoding: "ASCII",
Fields: visaFieldSpecs,
}
// Parser uses whichever spec you provide
func NewParser(spec *MessageSpec) *Parser
This separates "how to parse" from "what each network requires." Adding support for a new network means adding a new spec, not changing parser code.
Part 2: The Complete Field Specification
ISO8583:1987 Field Table
This is the canonical reference. Every ISO8583 parser needs this table (or equivalent):
var ISO1987FieldSpecs = map[int]*FieldSpec{
// Note: Field 1 is the secondary bitmap, handled specially
2: {Number: 2, Name: "Primary Account Number", LengthType: "LLVAR", MaxLength: 19, DataType: "N", Encoding: "ASCII"},
3: {Number: 3, Name: "Processing Code", LengthType: "FIXED", MaxLength: 6, DataType: "N", Encoding: "ASCII"},
4: {Number: 4, Name: "Amount, Transaction", LengthType: "FIXED", MaxLength: 12, DataType: "N", Encoding: "ASCII"},
5: {Number: 5, Name: "Amount, Settlement", LengthType: "FIXED", MaxLength: 12, DataType: "N", Encoding: "ASCII"},
6: {Number: 6, Name: "Amount, Cardholder Billing", LengthType: "FIXED", MaxLength: 12, DataType: "N", Encoding: "ASCII"},
7: {Number: 7, Name: "Transmission Date and Time", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
8: {Number: 8, Name: "Amount, Cardholder Billing Fee", LengthType: "FIXED", MaxLength: 8, DataType: "N", Encoding: "ASCII"},
9: {Number: 9, Name: "Conversion Rate, Settlement", LengthType: "FIXED", MaxLength: 8, DataType: "N", Encoding: "ASCII"},
10: {Number: 10, Name: "Conversion Rate, Cardholder Billing", LengthType: "FIXED", MaxLength: 8, DataType: "N", Encoding: "ASCII"},
11: {Number: 11, Name: "System Trace Audit Number", LengthType: "FIXED", MaxLength: 6, DataType: "N", Encoding: "ASCII"},
12: {Number: 12, Name: "Time, Local Transaction", LengthType: "FIXED", MaxLength: 6, DataType: "N", Encoding: "ASCII"},
13: {Number: 13, Name: "Date, Local Transaction", LengthType: "FIXED", MaxLength: 4, DataType: "N", Encoding: "ASCII"},
14: {Number: 14, Name: "Date, Expiration", LengthType: "FIXED", MaxLength: 4, DataType: "N", Encoding: "ASCII"},
15: {Number: 15, Name: "Date, Settlement", LengthType: "FIXED", MaxLength: 4, DataType: "N", Encoding: "ASCII"},
16: {Number: 16, Name: "Date, Conversion", LengthType: "FIXED", MaxLength: 4, DataType: "N", Encoding: "ASCII"},
17: {Number: 17, Name: "Date, Capture", LengthType: "FIXED", MaxLength: 4, DataType: "N", Encoding: "ASCII"},
18: {Number: 18, Name: "Merchant Type", LengthType: "FIXED", MaxLength: 4, DataType: "N", Encoding: "ASCII"},
19: {Number: 19, Name: "Country Code, Acquiring Institution", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
20: {Number: 20, Name: "Country Code, PAN Extended", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
21: {Number: 21, Name: "Country Code, Forwarding Institution", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
22: {Number: 22, Name: "POS Entry Mode", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
23: {Number: 23, Name: "Card Sequence Number", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
24: {Number: 24, Name: "Network International ID", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
25: {Number: 25, Name: "POS Condition Code", LengthType: "FIXED", MaxLength: 2, DataType: "N", Encoding: "ASCII"},
26: {Number: 26, Name: "POS PIN Capture Code", LengthType: "FIXED", MaxLength: 2, DataType: "N", Encoding: "ASCII"},
27: {Number: 27, Name: "Authorization ID Response Length", LengthType: "FIXED", MaxLength: 1, DataType: "N", Encoding: "ASCII"},
28: {Number: 28, Name: "Amount, Transaction Fee", LengthType: "FIXED", MaxLength: 9, DataType: "AN", Encoding: "ASCII"},
29: {Number: 29, Name: "Amount, Settlement Fee", LengthType: "FIXED", MaxLength: 9, DataType: "AN", Encoding: "ASCII"},
30: {Number: 30, Name: "Amount, Transaction Processing Fee", LengthType: "FIXED", MaxLength: 9, DataType: "AN", Encoding: "ASCII"},
31: {Number: 31, Name: "Amount, Settlement Processing Fee", LengthType: "FIXED", MaxLength: 9, DataType: "AN", Encoding: "ASCII"},
32: {Number: 32, Name: "Acquiring Institution ID", LengthType: "LLVAR", MaxLength: 11, DataType: "N", Encoding: "ASCII"},
33: {Number: 33, Name: "Forwarding Institution ID", LengthType: "LLVAR", MaxLength: 11, DataType: "N", Encoding: "ASCII"},
34: {Number: 34, Name: "PAN, Extended", LengthType: "LLVAR", MaxLength: 28, DataType: "ANS", Encoding: "ASCII"},
35: {Number: 35, Name: "Track 2 Data", LengthType: "LLVAR", MaxLength: 37, DataType: "Z", Encoding: "ASCII"},
36: {Number: 36, Name: "Track 3 Data", LengthType: "LLLVAR", MaxLength: 104, DataType: "Z", Encoding: "ASCII"},
37: {Number: 37, Name: "Retrieval Reference Number", LengthType: "FIXED", MaxLength: 12, DataType: "AN", Encoding: "ASCII"},
38: {Number: 38, Name: "Authorization ID Response", LengthType: "FIXED", MaxLength: 6, DataType: "AN", Encoding: "ASCII"},
39: {Number: 39, Name: "Response Code", LengthType: "FIXED", MaxLength: 2, DataType: "AN", Encoding: "ASCII"},
40: {Number: 40, Name: "Service Restriction Code", LengthType: "FIXED", MaxLength: 3, DataType: "AN", Encoding: "ASCII"},
41: {Number: 41, Name: "Card Acceptor Terminal ID", LengthType: "FIXED", MaxLength: 8, DataType: "ANS", Encoding: "ASCII"},
42: {Number: 42, Name: "Card Acceptor ID", LengthType: "FIXED", MaxLength: 15, DataType: "ANS", Encoding: "ASCII"},
43: {Number: 43, Name: "Card Acceptor Name/Location", LengthType: "FIXED", MaxLength: 40, DataType: "ANS", Encoding: "ASCII"},
44: {Number: 44, Name: "Additional Response Data", LengthType: "LLVAR", MaxLength: 25, DataType: "ANS", Encoding: "ASCII"},
45: {Number: 45, Name: "Track 1 Data", LengthType: "LLVAR", MaxLength: 76, DataType: "ANS", Encoding: "ASCII"},
46: {Number: 46, Name: "Additional Data - ISO", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
47: {Number: 47, Name: "Additional Data - National", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
48: {Number: 48, Name: "Additional Data - Private", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII", TLVFormat: "SIMPLE"},
49: {Number: 49, Name: "Currency Code, Transaction", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
50: {Number: 50, Name: "Currency Code, Settlement", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
51: {Number: 51, Name: "Currency Code, Cardholder Billing", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
52: {Number: 52, Name: "PIN Data", LengthType: "FIXED", MaxLength: 8, DataType: "B", Encoding: "BINARY"},
53: {Number: 53, Name: "Security Related Control Info", LengthType: "FIXED", MaxLength: 16, DataType: "N", Encoding: "ASCII"},
54: {Number: 54, Name: "Additional Amounts", LengthType: "LLLVAR", MaxLength: 120, DataType: "ANS", Encoding: "ASCII"},
55: {Number: 55, Name: "ICC Related Data", LengthType: "LLLVAR", MaxLength: 999, DataType: "B", Encoding: "BINARY", TLVFormat: "EMV"},
56: {Number: 56, Name: "Reserved ISO", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
57: {Number: 57, Name: "Reserved National", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
58: {Number: 58, Name: "Reserved National", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
59: {Number: 59, Name: "Reserved National", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
60: {Number: 60, Name: "Reserved Private", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
61: {Number: 61, Name: "Reserved Private", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
62: {Number: 62, Name: "Reserved Private", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII"},
63: {Number: 63, Name: "Reserved Private", LengthType: "LLLVAR", MaxLength: 999, DataType: "ANS", Encoding: "ASCII", TLVFormat: "SIMPLE"},
64: {Number: 64, Name: "MAC", LengthType: "FIXED", MaxLength: 8, DataType: "B", Encoding: "BINARY"},
// Secondary bitmap fields (65-128)
65: {Number: 65, Name: "Extended Bitmap Indicator", LengthType: "FIXED", MaxLength: 1, DataType: "B", Encoding: "BINARY"},
66: {Number: 66, Name: "Settlement Code", LengthType: "FIXED", MaxLength: 1, DataType: "N", Encoding: "ASCII"},
67: {Number: 67, Name: "Extended Payment Code", LengthType: "FIXED", MaxLength: 2, DataType: "N", Encoding: "ASCII"},
68: {Number: 68, Name: "Receiving Institution Country Code", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
69: {Number: 69, Name: "Settlement Institution Country Code", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
70: {Number: 70, Name: "Network Management Info Code", LengthType: "FIXED", MaxLength: 3, DataType: "N", Encoding: "ASCII"},
71: {Number: 71, Name: "Message Number", LengthType: "FIXED", MaxLength: 4, DataType: "N", Encoding: "ASCII"},
72: {Number: 72, Name: "Message Number, Last", LengthType: "FIXED", MaxLength: 4, DataType: "N", Encoding: "ASCII"},
73: {Number: 73, Name: "Date, Action", LengthType: "FIXED", MaxLength: 6, DataType: "N", Encoding: "ASCII"},
74: {Number: 74, Name: "Credits, Number", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
75: {Number: 75, Name: "Credits, Reversal Number", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
76: {Number: 76, Name: "Debits, Number", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
77: {Number: 77, Name: "Debits, Reversal Number", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
78: {Number: 78, Name: "Transfer, Number", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
79: {Number: 79, Name: "Transfer, Reversal Number", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
80: {Number: 80, Name: "Inquiries, Number", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
81: {Number: 81, Name: "Authorizations, Number", LengthType: "FIXED", MaxLength: 10, DataType: "N", Encoding: "ASCII"},
82: {Number: 82, Name: "Credits, Processing Fee Amount", LengthType: "FIXED", MaxLength: 12, DataType: "N", Encoding: "ASCII"},
83: {Number: 83, Name: "Credits, Transaction Fee Amount", LengthType: "FIXED", MaxLength: 12, DataType: "N", Encoding: "ASCII"},
84: {Number: 84, Name: "Debits, Processing Fee Amount", LengthType: "FIXED", MaxLength: 12, DataType: "N", Encoding: "ASCII"},
85: {Number: 85, Name: "Debits, Transaction Fee Amount", LengthType: "FIXED", MaxLength: 12, DataType: "N", Encoding: "ASCII"},
86: {Number: 86, Name: "Credits, Amount", LengthType: "FIXED", MaxLength: 16, DataType: "N", Encoding: "ASCII"},
87: {Number: 87, Name: "Credits, Reversal Amount", LengthType: "FIXED", MaxLength: 16, DataType: "N", Encoding: "ASCII"},
88: {Number: 88, Name: "Debits, Amount", LengthType: "FIXED", MaxLength: 16, DataType: "N", Encoding: "ASCII"},
89: {Number: 89, Name: "Debits, Reversal Amount", LengthType: "FIXED", MaxLength: 16, DataType: "N", Encoding: "ASCII"},
90: {Number: 90, Name: "Original Data Elements", LengthType: "FIXED", MaxLength: 42, DataType: "N", Encoding: "ASCII"},
91: {Number: 91, Name: "File Update Code", LengthType: "FIXED", MaxLength: 1, DataType: "AN", Encoding: "ASCII"},
92: {Number: 92, Name: "File Security Code", LengthType: "FIXED", MaxLength: 2, DataType: "AN", Encoding: "ASCII"},
93: {Number: 93, Name: "Response Indicator", LengthType: "FIXED", MaxLength: 5, DataType: "AN", Encoding: "ASCII"},
94: {Number: 94, Name: "Service Indicator", LengthType: "FIXED", MaxLength: 7, DataType: "AN", Encoding: "ASCII"},
95: {Number: 95, Name: "Replacement Amounts", LengthType: "FIXED", MaxLength: 42, DataType: "AN", Encoding: "ASCII"},
96: {Number: 96, Name: "Message Security Code", LengthType: "FIXED", MaxLength: 8, DataType: "B", Encoding: "BINARY"},
97: {Number: 97, Name: "Amount, Net Settlement", LengthType: "FIXED", MaxLength: 17, DataType: "AN", Encoding: "ASCII"},
98: {Number: 98, Name: "Payee", LengthType: "FIXED", MaxLength: 25, DataType: "ANS", Encoding: "ASCII"},
99: {Number: 99, Name: "Settlement Institution ID", LengthType: "LLVAR", MaxLength: 11, DataType: "N", Encoding: "ASCII"},
100: {Number: 100, Name: "Receiving Institution ID", LengthType: "LLVAR", MaxLength: 11, DataType: "N", Encoding: "ASCII"},
101: {Number: 101, Name: "File Name", LengthType: "LLVAR", MaxLength: 17, DataType: "ANS", Encoding: "ASCII"},
102: {Number: 102, Name: "Account ID 1", LengthType: "LLVAR", MaxLength: 28, DataType: "ANS", Encoding: "ASCII"},
103: {Number: 103, Name: "Account ID 2", LengthType: "LLVAR", MaxLength: 28, DataType: "ANS", Encoding: "ASCII"},
104: {Number: 104, Name: "Transaction Description", LengthType: "LLLVAR", MaxLength: 100, DataType: "ANS", Encoding: "ASCII"},
// 105-111 Reserved for ISO use
// 112-119 Reserved for National use
// 120-127 Reserved for Private use
128: {Number: 128, Name: "MAC", LengthType: "FIXED", MaxLength: 8, DataType: "B", Encoding: "BINARY"},
}
Network-Specific Variations
Different networks override specific fields:
// Visa variations
var VisaFieldOverrides = map[int]*FieldSpec{
48: {Number: 48, Name: "Additional Data - Private (Visa)", LengthType: "LLLVAR", MaxLength: 999,
DataType: "ANS", Encoding: "ASCII", TLVFormat: "SIMPLE",
Description: "Contains Merchant Advice Code, ECI, UCAF"},
62: {Number: 62, Name: "Auth Characteristics Indicator", LengthType: "LLLVAR", MaxLength: 999,
DataType: "ANS", Encoding: "ASCII",
Description: "Card present/not present, auth method"},
63: {Number: 63, Name: "Visa Private Use (DE63)", LengthType: "LLLVAR", MaxLength: 999,
DataType: "ANS", Encoding: "ASCII", TLVFormat: "SIMPLE",
Description: "Transaction ID, CVV2 result, Network ID"},
}
// Mastercard variations
var MastercardFieldOverrides = map[int]*FieldSpec{
48: {Number: 48, Name: "Additional Data (Mastercard)", LengthType: "LLLVAR", MaxLength: 999,
DataType: "ANS", Encoding: "ASCII", TLVFormat: "SIMPLE",
Description: "TCC, additional merchant data"},
61: {Number: 61, Name: "POS Data", LengthType: "LLLVAR", MaxLength: 999,
DataType: "ANS", Encoding: "ASCII",
Description: "POS data code, card data input"},
}
// Apply overrides to base spec
func ApplyOverrides(base *MessageSpec, overrides map[int]*FieldSpec) *MessageSpec {
result := &MessageSpec{
Name: base.Name,
BitmapEncoding: base.BitmapEncoding,
MTIEncoding: base.MTIEncoding,
LengthEncoding: base.LengthEncoding,
Fields: make(map[int]*FieldSpec),
}
// Copy base specs
for k, v := range base.Fields {
result.Fields[k] = v
}
// Apply overrides
for k, v := range overrides {
result.Fields[k] = v
}
return result
}
Part 3: Input Detection - What Am I Parsing?
Real-world parsing starts with a question: What format is this input?
Hex String vs Binary
// DetectInputFormat examines input and determines format
func DetectInputFormat(input []byte) InputFormat {
// Check if it's a hex string (ASCII hex characters)
isHex := true
for _, b := range input {
if !isHexChar(b) {
isHex = false
break
}
}
if isHex && len(input)%2 == 0 {
// Looks like a hex string
// Additional check: does it decode to valid ISO8583?
decoded, err := hex.DecodeString(string(input))
if err == nil && looksLikeISO8583(decoded) {
return FormatHexString
}
}
// Check if it's raw binary
if looksLikeISO8583(input) {
return FormatBinary
}
// Check if it has a length header
if len(input) >= 2 {
// 2-byte binary length header
length := int(input[0])<<8 | int(input[1])
if length == len(input)-2 {
return FormatBinaryWithLength
}
// 4-byte ASCII length header
if len(input) >= 4 {
lengthStr := string(input[0:4])
if length, err := strconv.Atoi(lengthStr); err == nil {
if length == len(input)-4 {
return FormatASCIIWithLength
}
}
}
}
return FormatUnknown
}
func isHexChar(b byte) bool {
return (b >= '0' && b <= '9') ||
(b >= 'a' && b <= 'f') ||
(b >= 'A' && b <= 'F')
}
func looksLikeISO8583(data []byte) bool {
if len(data) < 12 { // MTI (4) + bitmap (8) minimum
return false
}
// Check MTI format
mti := string(data[0:4])
if !isValidMTI(mti) {
// Maybe BCD MTI?
if len(data) >= 2 && isValidBCDMTI(data[0:2]) {
return true
}
return false
}
return true
}
Bitmap Encoding Detection
// DetectBitmapEncoding determines if bitmap is binary or hex-ASCII
func DetectBitmapEncoding(data []byte, mtiLen int) string {
if len(data) < mtiLen+8 {
return "UNKNOWN"
}
bitmapStart := data[mtiLen:]
// If first 16 bytes are all valid hex characters, it's hex-ASCII
if len(bitmapStart) >= 16 {
allHex := true
for i := 0; i < 16; i++ {
if !isHexChar(bitmapStart[i]) {
allHex = false
break
}
}
if allHex {
return "HEX_ASCII"
}
}
// Otherwise assume binary
return "BINARY"
}
The Auto-Detect Parser
// AutoParse attempts to parse with automatic format detection
func AutoParse(input []byte) (*ParsedMessage, error) {
format := DetectInputFormat(input)
var data []byte
switch format {
case FormatHexString:
var err error
data, err = hex.DecodeString(string(input))
if err != nil {
return nil, fmt.Errorf("hex decode: %w", err)
}
case FormatBinaryWithLength:
data = input[2:] // Skip 2-byte length
case FormatASCIIWithLength:
data = input[4:] // Skip 4-byte length
case FormatBinary:
data = input
default:
return nil, errors.New("could not determine input format")
}
// Detect bitmap encoding
bitmapEnc := DetectBitmapEncoding(data, 4)
// Select appropriate spec
var spec *MessageSpec
if bitmapEnc == "HEX_ASCII" {
spec = VisaBASE1Spec // Visa uses hex-ASCII
} else {
spec = ISO1987BaseSpec
}
return ParseWithSpec(data, spec)
}
Part 4: The Parsing Engine
Stage 1: MTI Extraction
func (p *Parser) parseMTI(data []byte) (MTI, int, error) {
var mti MTI
var consumed int
switch p.spec.MTIEncoding {
case "ASCII":
if len(data) < 4 {
return mti, 0, errors.New("insufficient data for ASCII MTI")
}
mti.Raw = string(data[0:4])
consumed = 4
case "BCD":
if len(data) < 2 {
return mti, 0, errors.New("insufficient data for BCD MTI")
}
mti.Raw = fmt.Sprintf("%02X%02X", data[0], data[1])
consumed = 2
default:
return mti, 0, fmt.Errorf("unknown MTI encoding: %s", p.spec.MTIEncoding)
}
// Parse components
if len(mti.Raw) != 4 {
return mti, 0, fmt.Errorf("invalid MTI length: %s", mti.Raw)
}
for i, c := range mti.Raw {
if c < '0' || c > '9' {
return mti, 0, fmt.Errorf("invalid MTI character at position %d: %c", i, c)
}
}
mti.Version = int(mti.Raw[0] - '0')
mti.Class = int(mti.Raw[1] - '0')
mti.Function = int(mti.Raw[2] - '0')
mti.Origin = int(mti.Raw[3] - '0')
return mti, consumed, nil
}
Stage 2: Bitmap Parsing
func (p *Parser) parseBitmap(data []byte) (Bitmap, int, error) {
var bitmap Bitmap
var consumed int
switch p.spec.BitmapEncoding {
case "BINARY":
if len(data) < 8 {
return bitmap, 0, errors.New("insufficient data for binary bitmap")
}
bitmap.Primary = make([]byte, 8)
copy(bitmap.Primary, data[0:8])
consumed = 8
case "HEX_ASCII":
if len(data) < 16 {
return bitmap, 0, errors.New("insufficient data for hex-ASCII bitmap")
}
var err error
bitmap.Primary, err = hex.DecodeString(string(data[0:16]))
if err != nil {
return bitmap, 0, fmt.Errorf("invalid hex bitmap: %w", err)
}
consumed = 16
default:
return bitmap, 0, fmt.Errorf("unknown bitmap encoding: %s", p.spec.BitmapEncoding)
}
// Check for secondary bitmap (bit 1 = 1)
if bitmap.Primary[0]&0x80 != 0 {
switch p.spec.BitmapEncoding {
case "BINARY":
if len(data) < consumed+8 {
return bitmap, 0, errors.New("insufficient data for secondary bitmap")
}
bitmap.Secondary = make([]byte, 8)
copy(bitmap.Secondary, data[consumed:consumed+8])
consumed += 8
case "HEX_ASCII":
if len(data) < consumed+16 {
return bitmap, 0, errors.New("insufficient data for secondary bitmap")
}
var err error
bitmap.Secondary, err = hex.DecodeString(string(data[consumed : consumed+16]))
if err != nil {
return bitmap, 0, fmt.Errorf("invalid hex secondary bitmap: %w", err)
}
consumed += 16
}
// Check for tertiary (bit 1 of secondary)
if bitmap.Secondary[0]&0x80 != 0 {
// Parse tertiary (rare, 2003 spec)
switch p.spec.BitmapEncoding {
case "BINARY":
if len(data) < consumed+8 {
return bitmap, 0, errors.New("insufficient data for tertiary bitmap")
}
bitmap.Tertiary = make([]byte, 8)
copy(bitmap.Tertiary, data[consumed:consumed+8])
consumed += 8
case "HEX_ASCII":
if len(data) < consumed+16 {
return bitmap, 0, errors.New("insufficient data for tertiary bitmap")
}
var err error
bitmap.Tertiary, err = hex.DecodeString(string(data[consumed : consumed+16]))
if err != nil {
return bitmap, 0, err
}
consumed += 16
}
}
}
// Extract field list
bitmap.Fields = extractFieldList(&bitmap)
return bitmap, consumed, nil
}
func extractFieldList(bitmap *Bitmap) []int {
var fields []int
// Primary bitmap: fields 1-64
for i := 0; i < 64; i++ {
byteIdx := i / 8
bitIdx := 7 - (i % 8)
if bitmap.Primary[byteIdx]&(1<<bitIdx) != 0 {
fields = append(fields, i+1)
}
}
// Secondary bitmap: fields 65-128
if bitmap.Secondary != nil {
for i := 0; i < 64; i++ {
byteIdx := i / 8
bitIdx := 7 - (i % 8)
if bitmap.Secondary[byteIdx]&(1<<bitIdx) != 0 {
fields = append(fields, i+65)
}
}
}
// Tertiary bitmap: fields 129-192
if bitmap.Tertiary != nil {
for i := 0; i < 64; i++ {
byteIdx := i / 8
bitIdx := 7 - (i % 8)
if bitmap.Tertiary[byteIdx]&(1<<bitIdx) != 0 {
fields = append(fields, i+129)
}
}
}
return fields
}
Stage 3: Field Extraction
func (p *Parser) parseFields(data []byte, bitmap Bitmap) (map[int]ParsedField, int, []string, error) {
fields := make(map[int]ParsedField)
warnings := []string{}
pos := 0
for _, fieldNum := range bitmap.Fields {
// Skip field 1 (secondary bitmap, already parsed)
if fieldNum == 1 {
continue
}
spec, ok := p.spec.Fields[fieldNum]
if !ok {
warnings = append(warnings, fmt.Sprintf("no spec for field %d, skipping", fieldNum))
continue
}
field, consumed, err := p.parseField(data[pos:], spec)
if err != nil {
return fields, pos, warnings, fmt.Errorf("field %d: %w", fieldNum, err)
}
fields[fieldNum] = field
pos += consumed
}
return fields, pos, warnings, nil
}
func (p *Parser) parseField(data []byte, spec *FieldSpec) (ParsedField, int, error) {
var field ParsedField
field.Number = spec.Number
field.Spec = spec
var length int
var consumed int
// Determine length
switch spec.LengthType {
case "FIXED":
length = spec.MaxLength
consumed = 0
case "LLVAR":
if len(data) < 2 {
return field, 0, errors.New("insufficient data for LLVAR length")
}
l, err := p.parseLengthPrefix(data[0:2])
if err != nil {
return field, 0, fmt.Errorf("LLVAR length: %w", err)
}
length = l
consumed = 2
case "LLLVAR":
if len(data) < 3 {
return field, 0, errors.New("insufficient data for LLLVAR length")
}
l, err := p.parseLengthPrefix(data[0:3])
if err != nil {
return field, 0, fmt.Errorf("LLLVAR length: %w", err)
}
length = l
consumed = 3
case "LLLLVAR":
if len(data) < 4 {
return field, 0, errors.New("insufficient data for LLLLVAR length")
}
l, err := p.parseLengthPrefix(data[0:4])
if err != nil {
return field, 0, fmt.Errorf("LLLLVAR length: %w", err)
}
length = l
consumed = 4
default:
return field, 0, fmt.Errorf("unknown length type: %s", spec.LengthType)
}
// Validate length
if length > spec.MaxLength {
return field, 0, fmt.Errorf("length %d exceeds max %d", length, spec.MaxLength)
}
// Calculate actual byte length (BCD encoding uses half)
byteLength := length
if spec.Encoding == "BCD" {
byteLength = (length + 1) / 2
}
if len(data) < consumed+byteLength {
return field, 0, fmt.Errorf("insufficient data: need %d bytes, have %d",
consumed+byteLength, len(data))
}
// Extract raw bytes
field.RawBytes = make([]byte, byteLength)
copy(field.RawBytes, data[consumed:consumed+byteLength])
field.RawHex = hex.EncodeToString(field.RawBytes)
consumed += byteLength
// Decode value
decoded, err := p.decodeFieldValue(field.RawBytes, spec, length)
if err != nil {
return field, consumed, fmt.Errorf("decode: %w", err)
}
field.Decoded = decoded
// Parse TLV subfields if applicable
if spec.TLVFormat != "" && spec.TLVFormat != "NONE" {
subfields, err := p.parseTLV(field.RawBytes, spec.TLVFormat)
if err == nil {
field.SubFields = subfields
}
// Don't fail on TLV parse error - still return main field
}
return field, consumed, nil
}
func (p *Parser) parseLengthPrefix(data []byte) (int, error) {
switch p.spec.LengthEncoding {
case "ASCII":
lengthStr := string(data)
return strconv.Atoi(lengthStr)
case "BCD":
// Convert BCD bytes to integer
result := 0
for _, b := range data {
result = result*100 + int(b>>4)*10 + int(b&0x0F)
}
return result, nil
default:
return strconv.Atoi(string(data)) // Default to ASCII
}
}
func (p *Parser) decodeFieldValue(data []byte, spec *FieldSpec, declaredLen int) (string, error) {
switch spec.Encoding {
case "ASCII":
return string(data), nil
case "BCD":
result := ""
for _, b := range data {
result += fmt.Sprintf("%02X", b)
}
// Trim to declared length (remove padding)
if len(result) > declaredLen {
result = result[:declaredLen]
}
return result, nil
case "BINARY", "HEX":
return hex.EncodeToString(data), nil
default:
return string(data), nil
}
}
Part 5: Pretty-Printing Strategies
A parser is only as useful as its output. Good pretty-printing is an art.
The Basic Format
func (msg *ParsedMessage) PrettyPrint() string {
var sb strings.Builder
// Header
sb.WriteString(fmt.Sprintf("═══════════════════════════════════════════════════════\n"))
sb.WriteString(fmt.Sprintf(" ISO8583 Message Parser - %s\n", msg.Spec.Name))
sb.WriteString(fmt.Sprintf("═══════════════════════════════════════════════════════\n\n"))
// MTI
sb.WriteString(fmt.Sprintf("MTI: %s\n", msg.MTI.Raw))
sb.WriteString(fmt.Sprintf(" Version: %d (%s)\n", msg.MTI.Version, mtiVersionName(msg.MTI.Version)))
sb.WriteString(fmt.Sprintf(" Class: %d (%s)\n", msg.MTI.Class, mtiClassName(msg.MTI.Class)))
sb.WriteString(fmt.Sprintf(" Function: %d (%s)\n", msg.MTI.Function, mtiFunctionName(msg.MTI.Function)))
sb.WriteString(fmt.Sprintf(" Origin: %d (%s)\n", msg.MTI.Origin, mtiOriginName(msg.MTI.Origin)))
sb.WriteString("\n")
// Bitmap
sb.WriteString(fmt.Sprintf("Bitmap: %s", hex.EncodeToString(msg.Bitmap.Primary)))
if msg.Bitmap.Secondary != nil {
sb.WriteString(hex.EncodeToString(msg.Bitmap.Secondary))
}
sb.WriteString("\n")
sb.WriteString(fmt.Sprintf("Fields present: %v\n", msg.Bitmap.Fields))
sb.WriteString("\n")
// Fields
sb.WriteString("Data Elements:\n")
sb.WriteString("───────────────────────────────────────────────────────\n")
// Sort fields by number
fieldNums := make([]int, 0, len(msg.Fields))
for num := range msg.Fields {
fieldNums = append(fieldNums, num)
}
sort.Ints(fieldNums)
for _, num := range fieldNums {
field := msg.Fields[num]
sb.WriteString(formatField(field))
}
// Warnings
if len(msg.Warnings) > 0 {
sb.WriteString("\nWarnings:\n")
for _, w := range msg.Warnings {
sb.WriteString(fmt.Sprintf(" - %s\n", w))
}
}
return sb.String()
}
func formatField(field ParsedField) string {
var sb strings.Builder
// Field header
sb.WriteString(fmt.Sprintf("DE%-3d %-35s\n", field.Number, field.Spec.Name))
// Raw value
sb.WriteString(fmt.Sprintf(" Raw: %s\n", truncate(field.RawHex, 60)))
// Decoded value with special formatting
decoded := formatDecodedValue(field)
sb.WriteString(fmt.Sprintf(" Decoded: %s\n", decoded))
// Subfields if present
if len(field.SubFields) > 0 {
sb.WriteString(" Subfields:\n")
for tag, sub := range field.SubFields {
sb.WriteString(fmt.Sprintf(" [%s] %s\n", tag, sub.Decoded))
}
}
sb.WriteString("\n")
return sb.String()
}
func formatDecodedValue(field ParsedField) string {
switch field.Number {
case 2: // PAN - mask it
return maskPAN(field.Decoded)
case 4, 5, 6: // Amounts - format with decimals
return formatAmount(field.Decoded)
case 7: // Transmission DateTime
return formatDateTime(field.Decoded, "MMDDhhmmss")
case 12: // Local Time
return formatTime(field.Decoded)
case 13, 14, 15, 16, 17: // Dates
return formatDate(field.Decoded)
case 35: // Track 2 - mask
return maskTrack2(field.Decoded)
case 39: // Response Code - add meaning
return fmt.Sprintf("%s (%s)", field.Decoded, responseCodeMeaning(field.Decoded))
case 49, 50, 51: // Currency codes
return fmt.Sprintf("%s (%s)", field.Decoded, currencyName(field.Decoded))
case 52: // PIN Block - never show
return "[ENCRYPTED PIN DATA]"
default:
return field.Decoded
}
}
func maskPAN(pan string) string {
if len(pan) <= 8 {
return pan
}
// Show first 6, last 4
return pan[:6] + strings.Repeat("*", len(pan)-10) + pan[len(pan)-4:]
}
func maskTrack2(track2 string) string {
if len(track2) < 20 {
return track2
}
// Show separator and last 8
idx := strings.Index(track2, "=")
if idx == -1 {
idx = strings.Index(track2, "D")
}
if idx > 0 && idx < len(track2)-8 {
return track2[:6] + strings.Repeat("*", idx-6) + track2[idx:idx+5] + "***"
}
return track2[:6] + "****" + track2[len(track2)-4:]
}
func formatAmount(amount string) string {
// Remove leading zeros and format
trimmed := strings.TrimLeft(amount, "0")
if trimmed == "" {
return "0.00"
}
// Insert decimal point (last 2 digits are cents)
if len(trimmed) <= 2 {
return "0." + fmt.Sprintf("%02s", trimmed)
}
return trimmed[:len(trimmed)-2] + "." + trimmed[len(trimmed)-2:]
}
func responseCodeMeaning(code string) string {
meanings := map[string]string{
"00": "Approved",
"01": "Refer to card issuer",
"03": "Invalid merchant",
"04": "Capture card",
"05": "Do not honor",
"12": "Invalid transaction",
"13": "Invalid amount",
"14": "Invalid card number",
"30": "Format error",
"41": "Lost card",
"43": "Stolen card",
"51": "Insufficient funds",
"54": "Expired card",
"55": "Incorrect PIN",
"57": "Transaction not permitted",
"58": "Terminal not permitted",
"59": "Suspected fraud",
"61": "Exceeds withdrawal limit",
"65": "Exceeds frequency limit",
"91": "Issuer unavailable",
"94": "Duplicate transmission",
"96": "System malfunction",
}
if meaning, ok := meanings[code]; ok {
return meaning
}
return "Unknown"
}
func currencyName(code string) string {
currencies := map[string]string{
"840": "USD",
"978": "EUR",
"826": "GBP",
"392": "JPY",
"124": "CAD",
"036": "AUD",
"756": "CHF",
"156": "CNY",
"356": "INR",
"682": "SAR",
"784": "AED",
}
if name, ok := currencies[code]; ok {
return name
}
return "Unknown"
}
JSON Output Format
For machine processing:
func (msg *ParsedMessage) ToJSON() ([]byte, error) {
output := map[string]interface{}{
"mti": map[string]interface{}{
"raw": msg.MTI.Raw,
"version": msg.MTI.Version,
"class": msg.MTI.Class,
"function": msg.MTI.Function,
"origin": msg.MTI.Origin,
},
"bitmap": map[string]interface{}{
"primary": hex.EncodeToString(msg.Bitmap.Primary),
"secondary": optionalHex(msg.Bitmap.Secondary),
"tertiary": optionalHex(msg.Bitmap.Tertiary),
"fields": msg.Bitmap.Fields,
},
"fields": map[string]interface{}{},
}
fields := output["fields"].(map[string]interface{})
for num, field := range msg.Fields {
fields[fmt.Sprintf("%d", num)] = map[string]interface{}{
"name": field.Spec.Name,
"raw": field.RawHex,
"decoded": field.Decoded,
"subfields": field.SubFields,
}
}
if len(msg.Warnings) > 0 {
output["warnings"] = msg.Warnings
}
return json.MarshalIndent(output, "", " ")
}
Compact Table Format
For log analysis:
func (msg *ParsedMessage) ToTable() string {
var sb strings.Builder
sb.WriteString(fmt.Sprintf("MTI=%s | ", msg.MTI.Raw))
sb.WriteString(fmt.Sprintf("BMP=%s | ", hex.EncodeToString(msg.Bitmap.Primary)[:8]))
// Key fields in compact form
if f, ok := msg.Fields[2]; ok {
sb.WriteString(fmt.Sprintf("PAN=%s | ", maskPAN(f.Decoded)))
}
if f, ok := msg.Fields[4]; ok {
sb.WriteString(fmt.Sprintf("AMT=%s | ", formatAmount(f.Decoded)))
}
if f, ok := msg.Fields[11]; ok {
sb.WriteString(fmt.Sprintf("STAN=%s | ", f.Decoded))
}
if f, ok := msg.Fields[39]; ok {
sb.WriteString(fmt.Sprintf("RC=%s", f.Decoded))
}
return sb.String()
}
Part 6: Handling Malformed Messages
Production messages are messy. Your parser must handle:
Truncated Messages
func (p *Parser) ParseWithRecovery(data []byte) (*ParsedMessage, error) {
msg := &ParsedMessage{
RawBytes: data,
RawHex: hex.EncodeToString(data),
Fields: make(map[int]ParsedField),
}
// Try to parse MTI
mti, mtiLen, err := p.parseMTI(data)
if err != nil {
return nil, fmt.Errorf("MTI parse failed: %w", err)
}
msg.MTI = mti
// Try to parse bitmap
if len(data) < mtiLen+8 {
msg.Warnings = append(msg.Warnings, "truncated before bitmap complete")
return msg, nil // Return partial result
}
bitmap, bmpLen, err := p.parseBitmap(data[mtiLen:])
if err != nil {
msg.Warnings = append(msg.Warnings, fmt.Sprintf("bitmap parse: %v", err))
return msg, nil
}
msg.Bitmap = bitmap
// Parse fields with recovery
pos := mtiLen + bmpLen
for _, fieldNum := range bitmap.Fields {
if fieldNum == 1 {
continue
}
if pos >= len(data) {
msg.Warnings = append(msg.Warnings,
fmt.Sprintf("truncated at field %d", fieldNum))
break
}
spec, ok := p.spec.Fields[fieldNum]
if !ok {
msg.Warnings = append(msg.Warnings,
fmt.Sprintf("no spec for field %d", fieldNum))
continue
}
field, consumed, err := p.parseField(data[pos:], spec)
if err != nil {
msg.Warnings = append(msg.Warnings,
fmt.Sprintf("field %d: %v", fieldNum, err))
// Try to estimate size and skip
if spec.LengthType == "FIXED" {
pos += spec.MaxLength
} else {
// Can't skip variable field safely
break
}
continue
}
msg.Fields[fieldNum] = field
pos += consumed
}
msg.BytesParsed = pos
return msg, nil
}
Unknown Fields
// When bitmap indicates a field but no spec exists
func (p *Parser) parseUnknownField(data []byte, fieldNum int) (ParsedField, int, error) {
field := ParsedField{
Number: fieldNum,
Spec: &FieldSpec{
Number: fieldNum,
Name: fmt.Sprintf("Unknown Field %d", fieldNum),
},
}
// Try to detect if it's variable length
// Heuristic: if first 2-3 bytes look like length, assume LLVAR/LLLVAR
if len(data) >= 2 {
llLen, err := strconv.Atoi(string(data[0:2]))
if err == nil && llLen > 0 && llLen < 100 && llLen+2 <= len(data) {
// Looks like LLVAR
field.RawBytes = data[2 : 2+llLen]
field.RawHex = hex.EncodeToString(field.RawBytes)
field.Decoded = string(field.RawBytes)
return field, 2 + llLen, nil
}
}
if len(data) >= 3 {
lllLen, err := strconv.Atoi(string(data[0:3]))
if err == nil && lllLen > 0 && lllLen < 1000 && lllLen+3 <= len(data) {
// Looks like LLLVAR
field.RawBytes = data[3 : 3+lllLen]
field.RawHex = hex.EncodeToString(field.RawBytes)
field.Decoded = string(field.RawBytes)
return field, 3 + lllLen, nil
}
}
// Can't determine format
return field, 0, fmt.Errorf("cannot determine format for unknown field %d", fieldNum)
}
Encoding Mismatches
// Detect when BCD vs ASCII assumption is wrong
func detectEncodingMismatch(data []byte, expectedLen int, spec *FieldSpec) error {
if spec.DataType == "N" { // Numeric
// Check if all bytes are printable ASCII digits
allASCIIDigits := true
for _, b := range data {
if b < '0' || b > '9' {
allASCIIDigits = false
break
}
}
// Check if it looks like BCD
allValidBCD := true
for _, b := range data {
if (b>>4) > 9 || (b&0x0F) > 9 {
allValidBCD = false
break
}
}
if spec.Encoding == "ASCII" && !allASCIIDigits && allValidBCD {
return fmt.Errorf("expected ASCII encoding but data looks like BCD")
}
if spec.Encoding == "BCD" && !allValidBCD && allASCIIDigits {
return fmt.Errorf("expected BCD encoding but data looks like ASCII")
}
}
return nil
}
Part 7: Production Debugging Techniques
Message Comparison
When debugging, you often compare two messages:
func CompareMessages(msg1, msg2 *ParsedMessage) string {
var sb strings.Builder
// MTI comparison
if msg1.MTI.Raw != msg2.MTI.Raw {
sb.WriteString(fmt.Sprintf("MTI: %s -> %s\n", msg1.MTI.Raw, msg2.MTI.Raw))
}
// Bitmap comparison
if hex.EncodeToString(msg1.Bitmap.Primary) != hex.EncodeToString(msg2.Bitmap.Primary) {
sb.WriteString(fmt.Sprintf("Bitmap: %s -> %s\n",
hex.EncodeToString(msg1.Bitmap.Primary),
hex.EncodeToString(msg2.Bitmap.Primary)))
}
// Find fields in msg1 but not msg2
for num := range msg1.Fields {
if _, ok := msg2.Fields[num]; !ok {
sb.WriteString(fmt.Sprintf("Field %d: REMOVED\n", num))
}
}
// Find fields in msg2 but not msg1
for num := range msg2.Fields {
if _, ok := msg1.Fields[num]; !ok {
sb.WriteString(fmt.Sprintf("Field %d: ADDED = %s\n", num, msg2.Fields[num].Decoded))
}
}
// Compare common fields
for num, f1 := range msg1.Fields {
if f2, ok := msg2.Fields[num]; ok {
if f1.Decoded != f2.Decoded {
sb.WriteString(fmt.Sprintf("Field %d: %s -> %s\n", num, f1.Decoded, f2.Decoded))
}
}
}
if sb.Len() == 0 {
return "Messages are identical"
}
return sb.String()
}
Hex Dump with Annotations
func (msg *ParsedMessage) AnnotatedHexDump() string {
var sb strings.Builder
data := msg.RawBytes
pos := 0
// MTI
mtiEnd := 4
if msg.Spec.MTIEncoding == "BCD" {
mtiEnd = 2
}
sb.WriteString(fmt.Sprintf("%s <-- MTI: %s\n",
hex.EncodeToString(data[pos:mtiEnd]), msg.MTI.Raw))
pos = mtiEnd
// Bitmap
bmpLen := 8
if msg.Spec.BitmapEncoding == "HEX_ASCII" {
bmpLen = 16
}
if msg.Bitmap.Secondary != nil {
bmpLen *= 2
}
sb.WriteString(fmt.Sprintf("%s <-- Bitmap: fields %v\n",
hex.EncodeToString(data[pos:pos+bmpLen]), msg.Bitmap.Fields[:min(10, len(msg.Bitmap.Fields))]))
pos += bmpLen
// Fields (simplified - show first few)
fieldNums := make([]int, 0, len(msg.Fields))
for num := range msg.Fields {
fieldNums = append(fieldNums, num)
}
sort.Ints(fieldNums)
for _, num := range fieldNums {
field := msg.Fields[num]
if field.Spec.LengthType != "FIXED" {
// Show length prefix
prefixLen := 2
if field.Spec.LengthType == "LLLVAR" {
prefixLen = 3
}
sb.WriteString(fmt.Sprintf("%s <-- DE%d length prefix\n",
hex.EncodeToString(data[pos:pos+prefixLen]), num))
pos += prefixLen
}
sb.WriteString(fmt.Sprintf("%s <-- DE%d: %s\n",
truncate(hex.EncodeToString(field.RawBytes), 30), num,
truncate(field.Decoded, 40)))
pos += len(field.RawBytes)
if pos > 200 { // Don't dump too much
sb.WriteString("... (truncated)\n")
break
}
}
return sb.String()
}
Field Validation
func ValidateMessage(msg *ParsedMessage) []ValidationError {
var errors []ValidationError
// MTI validation
if msg.MTI.Class == 0 || msg.MTI.Class == 3 || msg.MTI.Class == 5 ||
msg.MTI.Class == 6 || msg.MTI.Class == 7 || msg.MTI.Class == 9 {
errors = append(errors, ValidationError{
Field: "MTI",
Message: fmt.Sprintf("invalid message class: %d", msg.MTI.Class),
})
}
// Required fields for common message types
switch msg.MTI.Raw {
case "0100": // Auth request
requiredFields := []int{2, 3, 4, 11}
for _, f := range requiredFields {
if _, ok := msg.Fields[f]; !ok {
errors = append(errors, ValidationError{
Field: fmt.Sprintf("DE%d", f),
Message: "required for 0100 authorization request",
})
}
}
case "0110": // Auth response
requiredFields := []int{11, 39}
for _, f := range requiredFields {
if _, ok := msg.Fields[f]; !ok {
errors = append(errors, ValidationError{
Field: fmt.Sprintf("DE%d", f),
Message: "required for 0110 authorization response",
})
}
}
case "0400": // Reversal
if _, ok := msg.Fields[90]; !ok {
errors = append(errors, ValidationError{
Field: "DE90",
Message: "required for 0400 reversal (original data elements)",
})
}
}
// PAN validation (Luhn check)
if pan, ok := msg.Fields[2]; ok {
if !luhnValid(pan.Decoded) {
errors = append(errors, ValidationError{
Field: "DE2",
Message: "PAN fails Luhn check",
Value: maskPAN(pan.Decoded),
})
}
}
// Amount validation
if amt, ok := msg.Fields[4]; ok {
if amt.Decoded == "000000000000" {
errors = append(errors, ValidationError{
Field: "DE4",
Message: "zero amount",
})
}
}
// Expiry validation
if exp, ok := msg.Fields[14]; ok {
if len(exp.Decoded) == 4 {
yy, _ := strconv.Atoi(exp.Decoded[:2])
mm, _ := strconv.Atoi(exp.Decoded[2:])
if mm < 1 || mm > 12 {
errors = append(errors, ValidationError{
Field: "DE14",
Message: fmt.Sprintf("invalid expiry month: %d", mm),
})
}
// Check if expired (rough check)
now := time.Now()
currentYY := now.Year() % 100
if yy < currentYY || (yy == currentYY && mm < int(now.Month())) {
errors = append(errors, ValidationError{
Field: "DE14",
Message: "card appears expired",
Value: exp.Decoded,
})
}
}
}
return errors
}
type ValidationError struct {
Field string
Message string
Value string
}
Part 8: The CLI Interface
Command-Line Argument Parsing
func main() {
// Subcommands
parseCmd := flag.NewFlagSet("parse", flag.ExitOnError)
parseFile := parseCmd.String("file", "", "Input file (hex or binary)")
parseHex := parseCmd.String("hex", "", "Hex string to parse")
parseSpec := parseCmd.String("spec", "iso8583", "Specification (iso8583, visa, mastercard)")
parseOutput := parseCmd.String("output", "pretty", "Output format (pretty, json, table)")
parseValidate := parseCmd.Bool("validate", false, "Run validation checks")
compareCmd := flag.NewFlagSet("compare", flag.ExitOnError)
compareFile1 := compareCmd.String("file1", "", "First message file")
compareFile2 := compareCmd.String("file2", "", "Second message file")
if len(os.Args) < 2 {
printUsage()
os.Exit(1)
}
switch os.Args[1] {
case "parse":
parseCmd.Parse(os.Args[2:])
handleParse(*parseFile, *parseHex, *parseSpec, *parseOutput, *parseValidate)
case "compare":
compareCmd.Parse(os.Args[2:])
handleCompare(*compareFile1, *compareFile2)
case "specs":
listSpecs()
case "fields":
listFields(os.Args[2:])
default:
// Assume it's a hex string
handleParse("", os.Args[1], "iso8583", "pretty", false)
}
}
func handleParse(file, hexStr, specName, outputFmt string, validate bool) {
var data []byte
var err error
if file != "" {
data, err = os.ReadFile(file)
if err != nil {
fmt.Fprintf(os.Stderr, "Error reading file: %v\n", err)
os.Exit(1)
}
} else if hexStr != "" {
data = []byte(hexStr)
} else {
// Read from stdin
data, err = io.ReadAll(os.Stdin)
if err != nil {
fmt.Fprintf(os.Stderr, "Error reading stdin: %v\n", err)
os.Exit(1)
}
}
// Clean input (remove whitespace, newlines)
data = cleanInput(data)
// Get spec
spec := getSpec(specName)
// Parse
parser := NewParser(spec)
msg, err := parser.Parse(data)
if err != nil {
fmt.Fprintf(os.Stderr, "Parse error: %v\n", err)
os.Exit(1)
}
// Validate if requested
if validate {
errors := ValidateMessage(msg)
if len(errors) > 0 {
fmt.Fprintf(os.Stderr, "\nValidation errors:\n")
for _, e := range errors {
fmt.Fprintf(os.Stderr, " %s: %s\n", e.Field, e.Message)
}
}
}
// Output
switch outputFmt {
case "json":
jsonBytes, _ := msg.ToJSON()
fmt.Println(string(jsonBytes))
case "table":
fmt.Println(msg.ToTable())
case "hex":
fmt.Println(msg.AnnotatedHexDump())
default:
fmt.Println(msg.PrettyPrint())
}
}
func cleanInput(data []byte) []byte {
// Remove whitespace, newlines, etc.
var clean []byte
for _, b := range data {
if b != ' ' && b != '\n' && b != '\r' && b != '\t' {
clean = append(clean, b)
}
}
return clean
}
Interactive Mode
For debugging sessions:
func interactiveMode() {
scanner := bufio.NewScanner(os.Stdin)
parser := NewParser(ISO1987BaseSpec)
fmt.Println("ISO8583 Parser Interactive Mode")
fmt.Println("Commands: parse <hex>, spec <name>, field <num>, exit")
fmt.Println()
for {
fmt.Print("> ")
if !scanner.Scan() {
break
}
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
parts := strings.SplitN(line, " ", 2)
cmd := parts[0]
switch cmd {
case "exit", "quit":
return
case "spec":
if len(parts) < 2 {
fmt.Println("Usage: spec <name>")
continue
}
parser = NewParser(getSpec(parts[1]))
fmt.Printf("Switched to %s spec\n", parts[1])
case "field":
if len(parts) < 2 {
fmt.Println("Usage: field <number>")
continue
}
num, _ := strconv.Atoi(parts[1])
if spec, ok := parser.spec.Fields[num]; ok {
fmt.Printf("DE%d: %s\n", num, spec.Name)
fmt.Printf(" Length: %s (max %d)\n", spec.LengthType, spec.MaxLength)
fmt.Printf(" Type: %s, Encoding: %s\n", spec.DataType, spec.Encoding)
if spec.Description != "" {
fmt.Printf(" %s\n", spec.Description)
}
} else {
fmt.Printf("Unknown field: %d\n", num)
}
case "parse":
if len(parts) < 2 {
fmt.Println("Usage: parse <hex>")
continue
}
msg, err := parser.Parse([]byte(parts[1]))
if err != nil {
fmt.Printf("Error: %v\n", err)
} else {
fmt.Println(msg.PrettyPrint())
}
default:
// Assume it's hex to parse
msg, err := parser.Parse([]byte(line))
if err != nil {
fmt.Printf("Error: %v\n", err)
} else {
fmt.Println(msg.PrettyPrint())
}
}
}
}
Part 9: Performance Considerations
Allocation Reduction
// Reuse buffers for high-throughput parsing
type ParserPool struct {
parsers sync.Pool
}
func NewParserPool(spec *MessageSpec) *ParserPool {
return &ParserPool{
parsers: sync.Pool{
New: func() interface{} {
return &Parser{
spec: spec,
buf: make([]byte, 4096), // Reusable buffer
}
},
},
}
}
func (p *ParserPool) Parse(data []byte) (*ParsedMessage, error) {
parser := p.parsers.Get().(*Parser)
defer p.parsers.Put(parser)
return parser.Parse(data)
}
Zero-Copy Field Access
// For performance-critical paths, avoid copying
type ZeroCopyField struct {
Offset int
Length int
}
func (msg *ParsedMessage) GetFieldZeroCopy(num int) ([]byte, bool) {
if f, ok := msg.Fields[num]; ok {
// Return slice of original buffer
return msg.RawBytes[f.Offset:f.Offset+f.Length], true
}
return nil, false
}
Benchmarks
Always benchmark your parser:
func BenchmarkParseAuth(b *testing.B) {
parser := NewParser(ISO1987BaseSpec)
data, _ := hex.DecodeString(testAuthMessage)
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := parser.Parse(data)
if err != nil {
b.Fatal(err)
}
}
}
// Typical targets:
// - Simple message (10 fields): < 10µs
// - Complex message (40 fields): < 50µs
// - With validation: < 100µs
Part 10: Common Parsing Pitfalls
Pitfall 1: Off-by-One in Bitmap Field Numbers
// WRONG: Zero-indexed thinking
if bitmap[0] & 0x80 != 0 {
fields = append(fields, 0) // Field 0 doesn't exist!
}
// RIGHT: ISO8583 is 1-indexed
if bitmap[0] & 0x80 != 0 {
fields = append(fields, 1) // Field 1 = secondary bitmap indicator
}
Pitfall 2: BCD Length Prefixes Treated as ASCII
// WRONG: Assuming ASCII
lengthStr := string(data[0:2])
length, _ := strconv.Atoi(lengthStr)
// RIGHT: Check spec
if spec.LengthEncoding == "BCD" {
length = int(data[0])*100 + int(data[1])
} else {
lengthStr := string(data[0:2])
length, _ = strconv.Atoi(lengthStr)
}
Pitfall 3: Binary Fields Treated as ASCII
// WRONG: DE52 (PIN block) as string
pinBlock := string(data[0:8]) // Garbage!
// RIGHT: Keep as binary
pinBlock := make([]byte, 8)
copy(pinBlock, data[0:8])
// Display as hex
fmt.Printf("PIN Block: %X\n", pinBlock)
Pitfall 4: Fixed-Length Field Padding Not Handled
// WRONG: Return as-is
terminalID := string(data[0:8]) // "TERM001 " (with trailing space)
// RIGHT: Handle padding based on data type
if spec.DataType == "N" {
terminalID = strings.TrimLeft(string(data[0:8]), "0")
} else {
terminalID = strings.TrimRight(string(data[0:8]), " ")
}
Pitfall 5: Network-Specific Variations Ignored
// WRONG: One spec fits all
msg, _ := parser.Parse(data) // Uses ISO8583 base spec
// RIGHT: Detect and adapt
bitmapEnc := DetectBitmapEncoding(data)
if bitmapEnc == "HEX_ASCII" {
// Likely Visa - use Visa spec
parser = NewParser(VisaSpec)
}
msg, _ := parser.Parse(data)
Pitfall 6: Partial Parse Not Surfaced
// WRONG: Fail silently
func Parse(data []byte) (*Message, error) {
// ... parse until error ...
return msg, nil // Returns partial result without warning
}
// RIGHT: Report what was and wasn't parsed
func Parse(data []byte) (*ParsedMessage, error) {
// ... parse with recovery ...
msg.BytesParsed = pos
msg.Warnings = append(msg.Warnings,
fmt.Sprintf("parsed %d of %d bytes", pos, len(data)))
return msg, nil
}
Real-World Case Study: The Phantom Field
A processor's parser was returning incorrect amounts for certain transactions.
The Symptom:
Expected DE4: 000000005000 ($50.00)
Parsed DE4: 120143561234 (garbage)
The Investigation:
Step 1: Hex dump
0100723A...1654321012345678901200000000500012014356
^^─────────────────┘
What's this?
Step 2: Count field positions
MTI: 0100 (4 bytes)
Bitmap: 723A... (8 bytes)
DE2: 16 5432101234567890 (LLVAR: 2 + 16 = 18 bytes)
DE3: 000000 (6 bytes)
DE4: should start here...
Step 3: The discovery
After DE3, the bytes are: 12 00 00 00 00 50 00
^^
This is NOT DE4!
The bitmap indicates field 12 is present.
But DE4 is field 4... DE12 is field 12.
Wait - the bitmap says fields 2, 3, 4, 7, 11, 12... are present.
The parser read fields in order: 2, 3, 4...
But the message has fields in order: 2, 3, 12! (DE4 is missing)
Root Cause: The bitmap indicated DE4 was present (bit 4 = 1), but the actual message didn't contain DE4. The message producer had a bug - they set the bitmap bit but didn't include the field.
The Lesson: Your parser can't fix bad messages, but it should detect them. Validate message integrity:
func ValidateBitmapIntegrity(msg *ParsedMessage) error {
expectedLength := estimateMessageLength(msg.Bitmap.Fields, msg.Spec)
actualLength := msg.BytesParsed
if actualLength < expectedLength - 50 { // Allow some tolerance
return fmt.Errorf("message shorter than expected: parsed %d bytes, expected ~%d",
actualLength, expectedLength)
}
if actualLength > expectedLength + 50 {
return fmt.Errorf("message longer than expected: parsed %d bytes, expected ~%d",
actualLength, expectedLength)
}
return nil
}
Summary
Building a production ISO8583 parser teaches you:
- Specification is king: The field table is your source of truth
- Detect before assume: Auto-detect input format and encoding
- Fail gracefully: Parse what you can, report what you can't
- Output matters: Pretty-printing is user experience
- Validate thoroughly: Catch errors before they propagate
- Network variations are real: Visa != Mastercard != base ISO8583
- Performance has a cost: Optimize where it matters, not everywhere
Your parser will be used in:
- Production debugging at 3 AM
- Message validation during certification
- Training new engineers
- Forensic analysis of failed transactions
Build it well. Build it robust. Build it readable.
What You're Building
The iso-parser-cli problem asks you to implement:
- Parse any ISO8583 message from hex input
- Auto-detect format (hex string vs binary, ASCII vs BCD bitmap)
- Extract all fields using the specification
- Pretty-print output with field names and decoded values
- Handle errors gracefully with meaningful messages
- Support multiple output formats (pretty, JSON, table)
This is the culmination of everything you've learned. Every module builds to this moment.
Good luck. Your future self (debugging at 3 AM) will thank you.