Module 2: Bitmaps & Data Element Presence

Slides & Visual Reference

Slide 1: The Problem Bitmaps Solve

ISO8583 has 128 possible fields. No message uses all of them.
How do you know which fields are present?

Option A: Label each field
┌───────────────────────────────────────────────────────────────────┐
│  "Field02:5412345678901234,Field03:000000,Field04:000000010000"   │
│                                                                   │
│  7+ bytes overhead per field × 20 fields = 140+ bytes wasted      │
└───────────────────────────────────────────────────────────────────┘

Option B: Bitmap (ISO8583's solution)
┌───────────────────────────────────────────────────────────────────┐
│  8 bytes → indicates presence of up to 64 fields                  │
│  72 3A 44 81 08 E1 80 00 → fields 2,3,4,7,11,12,13,15,...        │
└───────────────────────────────────────────────────────────────────┘

8 bytes vs 140 bytes. That's why ISO8583 uses bitmaps.

Slide 2: Primary Bitmap Position

Every ISO8583 message has this structure:

┌──────────┬─────────────────────────┬────────────────────────────────┐
│   MTI    │    PRIMARY BITMAP       │        DATA ELEMENTS           │
│ 4 chars  │      8 bytes            │      (variable length)         │
│          │     (always)            │     (parsed in order)          │
└──────────┴─────────────────────────┴────────────────────────────────┘
     │              │                          │
     │              │                          └─► Only fields with
     │              │                              bits set in bitmap
     │              │
     │              └─► 64 bits = 64 possible fields (1-64)
     │
     └─► Already parsed (Module 1)

The bitmap is ALWAYS present, ALWAYS 8 bytes (primary).

Slide 3: Bit Numbering Convention

ISO8583 uses 1-BASED numbering with MSB-FIRST ordering

Byte 0: 72 (hex) = 0111 0010 (binary)

        ┌─ Bit 1 (MSB)
        │
        ▼
        0  1  1  1  0  0  1  0
        │  │  │  │  │  │  │  │
Bit #:  1  2  3  4  5  6  7  8

Fields present: 2, 3, 4, 7

CRITICAL: Bit 1 is LEFTMOST, not rightmost!
CRITICAL: Field numbers start at 1, not 0!

Slide 4: The Complete Formula

To check if field N is present:

┌─────────────────────────────────────────────────────────────┐
│                                                             │
│   byteIndex  = (N - 1) / 8                                  │
│   bitPosition = 7 - ((N - 1) % 8)                           │
│   isPresent  = (bitmap[byteIndex] >> bitPosition) & 1       │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Example: Is field 11 present?

  byteIndex   = (11 - 1) / 8 = 10 / 8 = 1
  bitPosition = 7 - ((11 - 1) % 8) = 7 - 2 = 5

  Check: bit 5 of byte 1

  Byte 1: 3A = 0011 1010
                   │
                   └─ position 5 (counting from right, 0-indexed)

  Value = 1 → Field 11 IS present ✓

Slide 4b: Off-by-One Sanity Table

If these are wrong, every later lookup will be wrong.

Slide 5: Full Bitmap Decode Example

Bitmap: 72 3A 44 81 08 E1 80 00

Byte 0: 72 = 0111 0010     Fields:  2,  3,  4,  7
Byte 1: 3A = 0011 1010     Fields: 11, 12, 13, 15
Byte 2: 44 = 0100 0100     Fields: 18, 22
Byte 3: 81 = 1000 0001     Fields: 25, 32
Byte 4: 08 = 0000 1000     Fields: 37
Byte 5: E1 = 1110 0001     Fields: 41, 42, 43, 48
Byte 6: 80 = 1000 0000     Fields: 49
Byte 7: 00 = 0000 0000     Fields: (none)

Total present: 2, 3, 4, 7, 11, 12, 13, 15, 18, 22, 25, 32, 37, 41, 42, 43, 48, 49

Typical authorization request fields!

Slide 6: Bit 1 - The Secondary Bitmap Indicator

┌──────────────────────────────────────────────────────────────────┐
│  Bit 1 = 0:  Primary bitmap only (fields 1-64)                   │
│                                                                  │
│  ┌──────────┬────────────────────┬───────────────────┐           │
│  │   MTI    │  Primary Bitmap    │   Data Elements   │           │
│  │          │     8 bytes        │                   │           │
│  └──────────┴────────────────────┴───────────────────┘           │
├──────────────────────────────────────────────────────────────────┤
│  Bit 1 = 1:  Primary + Secondary bitmap (fields 1-128)           │
│                                                                  │
│  ┌──────────┬────────────────┬────────────────────┬───────────┐  │
│  │   MTI    │ Primary Bitmap │  Secondary Bitmap  │   Data    │  │
│  │          │ 8 bytes,bit1=1 │  8 bytes (= DE1)   │  Elements │  │
│  └──────────┴────────────────┴────────────────────┴───────────┘  │
└──────────────────────────────────────────────────────────────────┘

IMPORTANT: The secondary bitmap IS Field 1 (DE1)
           There is no separate "Field 1 data"

Slide 7: Secondary Bitmap Structure

When Bit 1 = 1:

Primary Bitmap          Secondary Bitmap
┌────────────────┐      ┌────────────────┐
│ F2 34 00 01... │      │ 80 00 00 00... │
│ (Bit 1 = 1)    │      │                │
└────────────────┘      └────────────────┘
     │                       │
     │                       └─► Covers fields 65-128
     │                           Bit 1 here = field 65
     │                           Bit 2 here = field 66
     │
     └─► Covers fields 1-64 (as before)
         Bit 1 = 1 means "secondary follows"
         Bit 2 = field 2
         ...

Fields 65-128 only possible when primary bit 1 = 1

Slide 8: Tertiary Bitmap (2003 Spec)

ISO 8583:2003 extends to fields 129-192

Pattern continues:
┌─────────────────────────────────────────────────────────────────┐
│  Bit 1 of Primary (field 1)    → Secondary bitmap follows       │
│  Bit 1 of Secondary (field 65) → Tertiary bitmap follows        │
└─────────────────────────────────────────────────────────────────┘

Full structure:
┌──────┬───────────┬───────────────┬───────────────┬──────────────┐
│ MTI  │  Primary  │   Secondary   │   Tertiary    │    Data      │
│      │  8 bytes  │   8 bytes     │   8 bytes     │   Elements   │
│      │ bit1 = 1  │  bit1 = 1     │  (= DE65)     │              │
└──────┴───────────┴───────────────┴───────────────┴──────────────┘

24 bytes of bitmap = 192 possible fields

In practice: Tertiary bitmaps are rare. Most systems use 1987 spec.

Slide 9: Binary vs Hex-ASCII Encoding

The SAME bitmap can be transmitted two ways:

Binary (8 bytes):
┌──────────────────────────────────────────────────────────┐
│  72  3A  44  81  08  E1  80  00                          │
│  (raw byte values)                                       │
└──────────────────────────────────────────────────────────┘

Hex-ASCII (16 characters = 16 bytes):
┌──────────────────────────────────────────────────────────┐
│  "723A448108E18000"                                      │
│  37 32 33 41 34 34 38 31 30 38 45 31 38 30 30 30        │
│  '7''2''3''A''4''4''8''1''0''8''E''1''8''0''0''0'       │
└──────────────────────────────────────────────────────────┘

HOW TO DETECT:
- 8 bytes → Binary
- 16 bytes with values 0x30-0x39, 0x41-0x46 → Hex-ASCII

When parsing fails: CHECK ENCODING FIRST

Slide 9b: Encoding Guardrails

When unsure of encoding:

1. Validate byte/char length first.
2. Reject odd-length hex candidate immediately.
3. If text mode is possible, decode only on strict hex validation.
4. Confirm secondary/tertiary indicators against consumed bytes before field parse.

Slide 9c: Bitmap Parsing State Machine

// ParseBitmaps reads only what the control bits demand.
func ParseBitmaps(input []byte, isHex bool, support2003 bool) (primary, secondary, tertiary []byte, consumed int, err error) {
    const (
        BIN_LEN = 8
        HEX_LEN = 16
    )

    baseLen := BIN_LEN
    if isHex { baseLen = HEX_LEN }
    if len(input) < baseLen {
        return nil, nil, nil, 0, ErrBitmapTooShort
    }

    // Decode primary bitmap from input[:baseLen]
    primary = append([]byte(nil), input[:baseLen]...)
    consumed = baseLen

    // If primary bit 1 is set, decode secondary
    if IsFieldPresent(primary, 1) {
        if len(input) < consumed+baseLen {
            return nil, nil, nil, 0, ErrBitmapTooShort
        }
        secondary = append([]byte(nil), input[consumed:consumed+baseLen]...)
        consumed += baseLen

        // If secondary bit 1 is set, decode tertiary (2003 mode)
        if support2003 && IsFieldPresent(secondary, 1) {
            if len(input) < consumed+baseLen {
                return nil, nil, nil, 0, ErrBitmapTooShort
            }
            tertiary = append([]byte(nil), input[consumed:consumed+baseLen]...)
            consumed += baseLen
        }
    }

    return primary, secondary, tertiary, consumed, nil
}

The rule is simple:

• control bits decide how many bitmap bytes exist;
• field bytes are never decoded until all required bitmap bytes are validated.

Slide 10: Go Implementation - Check Field

// IsFieldPresent checks if field N is present in bitmap
func IsFieldPresent(bitmap []byte, field int) bool {
    if field < 1 || field > len(bitmap)*8 {
        return false
    }

    byteIndex   := (field - 1) / 8
    bitPosition := 7 - ((field - 1) % 8)

    return (bitmap[byteIndex] >> bitPosition) & 1 == 1
}

// Example usage:
bitmap := []byte{0x72, 0x3A, 0x44, 0x81, 0x08, 0xE1, 0x80, 0x00}

IsFieldPresent(bitmap, 2)  // true  (PAN)
IsFieldPresent(bitmap, 5)  // false
IsFieldPresent(bitmap, 11) // true  (STAN)

Slide 10b: Canonical Presence Round-Trip

// fieldsFromBitmap and bitmapFromFields should round-trip.
func fieldsFromBitmap(bitmap []byte) []int {
    out := make([]int, 0)
    for field := 1; field <= len(bitmap)*8; field++ {
        if IsFieldPresent(bitmap, field) {
            out = append(out, field)
        }
    }
    return out
}

func bitmapRoundTripInvariant(fields []int) bool {
    a := buildBitmapFromFields(fields)
    b := buildBitmapFromFields(fieldsFromBitmap(a))
    return bytes.Equal(a, b)
}

// Property-based test idea:
// generate random field lists -> invariant must hold

Slide 11: Go Implementation - Get All Fields

// GetPresentFields returns all field numbers with bit set
func GetPresentFields(bitmap []byte) []int {
    var fields []int

    for field := 1; field <= len(bitmap)*8; field++ {
        byteIndex   := (field - 1) / 8
        bitPosition := 7 - ((field - 1) % 8)

        if (bitmap[byteIndex] >> bitPosition) & 1 == 1 {
            fields = append(fields, field)
        }
    }

    return fields
}

// Example:
bitmap := []byte{0x72, 0x3A}  // First 16 bits only
fields := GetPresentFields(bitmap)
// Result: []int{2, 3, 4, 7, 11, 12, 13, 15}

Slide 12: Go Implementation - Build Bitmap

// BuildBitmap creates bitmap from field list
func BuildBitmap(fields []int) []byte {
    // Find max field to determine bitmap size
    maxField := 0
    for _, f := range fields {
        if f > maxField {
            maxField = f
        }
    }

    size := 8  // Primary
    if maxField > 64 {
        size = 16  // + Secondary
    }

    bitmap := make([]byte, size)

    for _, field := range fields {
        byteIndex   := (field - 1) / 8
        bitPosition := 7 - ((field - 1) % 8)
        bitmap[byteIndex] |= (1 << bitPosition)
    }

    // Set bit 1 if secondary exists
    if size > 8 {
        bitmap[0] |= 0x80
    }

    return bitmap
}

Slide 13: Common Pitfalls

┌──────────────────────────────────────────────────────────────────┐
│  1. OFF-BY-ONE ERROR                                             │
│     WRONG: byteIndex = field / 8                                 │
│     RIGHT: byteIndex = (field - 1) / 8                           │
├──────────────────────────────────────────────────────────────────┤
│  2. BIT ORDER CONFUSION                                          │
│     WRONG: Bit 1 is rightmost (LSB)                              │
│     RIGHT: Bit 1 is leftmost (MSB)                               │
├──────────────────────────────────────────────────────────────────┤
│  3. FORGETTING BIT 1 WHEN BUILDING                               │
│     WRONG: Just set field 70's bit                               │
│     RIGHT: Set bit 1 FIRST, then field 70                        │
├──────────────────────────────────────────────────────────────────┤
│  4. ENCODING MISMATCH                                            │
│     WRONG: Assume 8 bytes                                        │
│     RIGHT: Check if 16 hex-ASCII chars                           │
├──────────────────────────────────────────────────────────────────┤
│  5. BITMAP/DATA MISMATCH                                         │
│     If bitmap says 10 fields but data has 11:                    │
│     Every field after the extra is misaligned → garbage          │
└──────────────────────────────────────────────────────────────────┘

Slide 14: Debugging Bitmaps

PRINT THE BITMAP IN BINARY:

func PrintBitmap(bitmap []byte) {
    for i, b := range bitmap {
        for bit := 7; bit >= 0; bit-- {
            field := i*8 + (7 - bit) + 1
            if (b >> bit) & 1 == 1 {
                fmt.Printf("Field %2d: PRESENT\n", field)
            }
        }
    }
}

Output:
Field  2: PRESENT
Field  3: PRESENT
Field  4: PRESENT
Field  7: PRESENT
Field 11: PRESENT
...

Compare expected vs actual fields to find mismatches!

Slide 14b: Debugging Triage Checklist

1) Parse stage
   - Confirm expected encoding: binary(8/16 bytes) vs hex-ASCII(16/32 chars)
   - Validate length and allowed chars before field parsing

2) Indicator stage
   - If primary bit 1 = 1 -> secondary must be present
   - If secondary bit 1 = 1 -> check 2003 tertiary support contract

3) Presence stage
   - Compare bitmap-derived fields vs expected message fields
   - Any unexpected set/missing bit is a protocol mismatch

4) Alignment stage
   - Parse first few fields in lockstep and verify cursor movement
   - One wrong length => all subsequent fields misaligned

5) Recovery
   - Log MTI + bitmap bytes + expected/actual field diff
   - Keep it reproducible before changing decode logic

Slide 15: Performance Optimization

For high-throughput systems (100K+ TPS):

BASIC: Array indexing
    bitmap[byteIndex] >> bitPosition & 1
    ~2ns per check

FAST: Use uint64 for primary bitmap
    bitmap := binary.BigEndian.Uint64(data)
    (bitmap >> (64 - field)) & 1
    ~0.5ns per check

FASTEST: Precomputed masks
    var masks [65]uint64  // masks[n] checks field n
    bitmap & masks[field] != 0
    ~0.3ns per check

For most applications: Basic is fine.
For network switches: Every nanosecond matters.

Slide 16: Quick Reference Card

┌────────────────────────────────────────────────────────────────┐
│  BITMAP QUICK REFERENCE                                        │
├────────────────────────────────────────────────────────────────┤
│  Primary Bitmap:     8 bytes, fields 1-64                      │
│  Secondary Bitmap:   8 bytes, fields 65-128 (when bit1=1)      │
│  Tertiary Bitmap:    8 bytes, fields 129-192 (2003 spec)       │
├────────────────────────────────────────────────────────────────┤
│  Bit Numbering:      1-based, MSB-first, big-endian            │
│                                                                │
│  Field N present?    byteIdx = (N-1)/8                         │
│                      bitPos = 7 - ((N-1)%8)                    │
│                      check: (bitmap[byteIdx] >> bitPos) & 1    │
├────────────────────────────────────────────────────────────────┤
│  Set field N:        bitmap[byteIdx] |= (1 << bitPos)          │
│  Clear field N:      bitmap[byteIdx] &= ^(1 << bitPos)         │
├────────────────────────────────────────────────────────────────┤
│  Bit 1 = 1:          Secondary bitmap follows                  │
│  Bit 65 = 1:         Tertiary bitmap follows (2003 only)      │
├────────────────────────────────────────────────────────────────┤
│  Common fields:                                                │
│    2=PAN, 3=ProcCode, 4=Amount, 11=STAN, 39=RespCode        │
│    64=MAC, 70=NetMgmt, 95=Replacement, 128=Private use        │
└────────────────────────────────────────────────────────────────┘

Slide 17: Key Takeaways

1. BITMAP = TABLE OF CONTENTS
   └─► Tells you which fields are present before parsing data

2. 8 BYTES = 64 FIELDS
   └─► Compact: 8 bytes covers 64 fields vs 100+ bytes of labels

3. BIT 1 IS SPECIAL
   └─► When set, secondary bitmap follows (for fields 65-128)

4. 1-BASED, MSB-FIRST
   └─► Field 1 = leftmost bit of first byte
   └─► Not 0-based! This causes bugs.

5. THE FORMULA
   └─► byteIndex = (field-1)/8
   └─► bitPosition = 7 - ((field-1)%8)

6. WATCH FOR ENCODING
   └─► Binary (8 bytes) vs Hex-ASCII (16 chars)
   └─► When parsing fails, check this first!