# How ipsw Handles LZFSE Compression and Extracts PBZX Streams: A Technical Deep Dive

> Explore how ipsw handles LZFSE compression and extracts PBZX streams. Discover its Go FSE decoder and concurrent pipeline for efficient data processing. Learn the technical details.

- Repository: [blacktop/ipsw](https://github.com/blacktop/ipsw)
- Tags: deep-dive
- Published: 2026-02-26

---

**The `ipsw` tool detects LZFSE-compressed data by scanning for the `bvx2` magic header and decompresses it using a pure-Go FSE decoder, while PBZX streams are extracted via a concurrent pipeline that inflates XZ-compressed chunks and reassembles them in order using a min-heap.**

Apple's firmware update formats rely on proprietary compression schemes that standard open-source tools cannot process. The `blacktop/ipsw` repository provides complete Go implementations for both **LZFSE compression**—found in kernelcaches, IMG3 payloads, and device trees—and **PBZX stream extraction**—the container format used in iOS OTA updates. This article examines the detection logic, decoder architecture, and concurrent extraction pipelines implemented in the source code.

## Detecting LZFSE Compression via the bvx2 Magic Header

### How ipsw Identifies LZFSE Blocks

`ipsw` detects LZFSE compression by checking the first four bytes of a payload for the ASCII string **`bvx2`**. This check appears in [`pkg/img3/img3.go`](https://github.com/blacktop/ipsw/blob/main/pkg/img3/img3.go) within the `DecryptData` function, where decrypted payloads are scanned before decompression.

```go
// Detect LZFSE-compressed payload (magic "bvx2")
if len(decrypted) >= 4 && bytes.Contains(decrypted[:4], []byte("bvx2")) {
    decompressed, err := lzfse.NewDecoder(decrypted).DecodeBuffer()
    // … error handling …
}

```

*Source:* [img3.go lines 294-300](https://github.com/blacktop/ipsw/blob/master/pkg/img3/img3.go#L294-L300)

The same detection pattern appears in [`pkg/ftab/ftab.go`](https://github.com/blacktop/ipsw/blob/main/pkg/ftab/ftab.go) and [`pkg/devicetree/devicetree.go`](https://github.com/blacktop/ipsw/blob/main/pkg/devicetree/devicetree.go) for other firmware components.

### The Decoder Architecture in pkg/lzfse

The **LZFSE decoder** is implemented as a pure-Go package in `pkg/lzfse`. The entry point `NewDecoder` initializes a scratch buffer and a growing output buffer, while `DecodeBuffer` orchestrates the decompression.

```go
// NewDecoder creates a decoder with a scratch buffer.
func NewDecoder(data []byte) *Decoder {
    var dst bytes.Buffer
    dst.Grow(4 * len(data))
    scratch := make([]byte, 2*binary.Size(compressedBlockHeaderV1{}))
    return &Decoder{
        src: bytes.NewReader(append(data, scratch...)),
        dst: dst,
    }
}

// DecodeBuffer runs the full decode pipeline.
func (s *Decoder) DecodeBuffer() ([]byte, error) {
    if err := s.decode(); err != nil { return nil, err }
    return s.dst.Bytes(), nil
}

```

*Source:* [lzfse.go lines 27-46](https://github.com/blacktop/ipsw/blob/master/pkg/lzfse/lzfse.go#L27-L46)

The core decompression logic resides in `decodeV1`, `decodeV2HeaderSize`, and `decodeLmdExecuteMatch`. These functions perform the following operations:

- Parse the block header to extract magic, raw-bytes count, and literal/match tables
- Rebuild the **FSE (Finite State Entropy)** tables for literal and match streams
- Execute sliding-window copy operations to reconstruct the original data

Key constants defining hash sizes, symbol counts, and state structures are centralized in [`pkg/lzfse/types.go`](https://github.com/blacktop/ipsw/blob/main/pkg/lzfse/types.go) to maintain compatibility with Apple's reference implementation.

*Source:* [types.go lines 11-57](https://github.com/blacktop/ipsw/blob/master/pkg/lzfse/types.go#L11-L57)

## Extracting PBZX Streams with Concurrent XZ Decompression

### Detecting PBZX Containers

`ipsw` identifies **PBZX streams** by checking the first four bytes for the ASCII string `"pbzx"`. The detection logic resides in [`internal/magic/magic.go`](https://github.com/blacktop/ipsw/blob/main/internal/magic/magic.go), which provides the `IsPBZX` and `IsPBZXData` helper functions.

```go
func IsPBZX(filePath string) (bool, error) {
    f, err := os.Open(filePath)
    // … read 4 bytes …
    return IsPBZXData(f)
}

func IsPBZXData(r io.Reader) (bool, error) {
    magic := make([]byte, 4)
    _, err := io.ReadFull(r, magic)
    // …
    if string(magic) == "pbzx" { return true, nil }
    return false, nil
}

```

*Source:* [magic.go lines 33-36 and 318-334](https://github.com/blacktop/ipsw/blob/master/internal/magic/magic.go#L33-L36)

The CLI command `ipsw pbzx` (implemented in [`cmd/ipsw/cmd/pbzx.go`](https://github.com/blacktop/ipsw/blob/main/cmd/ipsw/cmd/pbzx.go)) uses `magic.IsPBZX` to validate input files before invoking the extraction pipeline.

### The Three-Stage Pipeline: Read, Inflate, and Ordered Write

The **PBZX extraction engine** is implemented in `pkg/ota/pbzx` as a concurrent pipeline that decompresses XZ-compressed chunks while maintaining output order. The public API is the `Extract` function:

```go
func Extract(ctx context.Context, src io.Reader, dst io.Writer, numWorker int) error

```

*Source:* [extract.go lines 10-21](https://github.com/blacktop/ipsw/blob/master/pkg/ota/pbzx/extract.go#L10-L21)

The pipeline consists of three coordinated stages:

**1. Reader Stage ([`read.go`](https://github.com/blacktop/ipsw/blob/main/read.go))**

The reader parses the PBZX container header and iterates through chunks, each defined by an inflated (uncompressed) size and a deflated (compressed) size.

```go
// Logic from read.go:
// If deflateSize < inflateSize → compressed chunk → inflateCh
// If deflateSize == inflateSize → raw chunk → writeCh

```

*Source:* [read.go lines 13-70](https://github.com/blacktop/ipsw/blob/master/pkg/ota/pbzx/read.go#L13-L70)

**2. Inflate Workers ([`inflate.go`](https://github.com/blacktop/ipsw/blob/main/inflate.go))**

A configurable pool of workers (defaulting to `runtime.NumCPU()`) receives compressed chunks from the `inflateCh` channel. Each worker uses `github.com/xi2/xz` to decompress the XZ payload, validates the output length against the expected inflated size, and forwards the result to the `writeCh`.

```go
// Worker logic from inflate.go:
// xz.NewReader(chunkData) → decompress → validate length → writeCh

```

*Source:* [inflate.go lines 12-40](https://github.com/blacktop/ipsw/blob/master/pkg/ota/pbzx/inflate.go#L12-L40)

**3. Ordered Writer ([`write.go`](https://github.com/blacktop/ipsw/blob/main/write.go))**

Because XZ decompression happens concurrently, chunks may complete out of order. The writer stage uses a min-heap (`_Heap`) keyed by chunk index to buffer out-of-order blocks. It writes chunks to the output stream only when the next expected index arrives, guaranteeing that the final extracted file maintains the correct byte order.

```go
// Heap-based ordering from write.go:
// out-of-order chunks buffered in _Heap
// write when idx == nextExpected

```

*Source:* [write.go lines 10-49](https://github.com/blacktop/ipsw/blob/master/pkg/ota/pbzx/write.go#L10-L49)

The `Extract` function orchestrates these stages by creating buffered channels, launching the reader goroutine, spawning the worker pool, and running the writer. A shared `context.Context` enables cancellation if any stage encounters an error.

## Practical Code Examples

### Decompressing Raw LZFSE Data in Go

The following example demonstrates how to use `ipsw`'s LZFSE decoder to decompress a raw buffer:

```go
package main

import (
	"fmt"
	"os"

	"github.com/blacktop/ipsw/pkg/lzfse"
)

func main() {
	// Load a file containing raw LZFSE-compressed data.
	data, err := os.ReadFile("kernelcache.lzfse")
	if err != nil {
		panic(err)
	}

	// Initialize the decoder with a scratch buffer.
	dec := lzfse.NewDecoder(data)
	
	// Decompress the entire buffer.
	plain, err := dec.DecodeBuffer()
	if err != nil {
		panic(err)
	}

	fmt.Printf("Decompressed %d → %d bytes\n", len(data), len(plain))
	// Use plain (e.g., parse Mach-O, write to disk, etc.)
}

```

*Key API:* `lzfse.NewDecoder(data).DecodeBuffer()` implemented in [`pkg/lzfse/lzfse.go`](https://github.com/blacktop/ipsw/blob/main/pkg/lzfse/lzfse.go).

### Extracting PBZX Streams Programmatically

This example shows how to extract a PBZX container using the concurrent pipeline:

```go
package main

import (
	"bytes"
	"context"
	"fmt"
	"os"
	"runtime"

	"github.com/blacktop/ipsw/pkg/ota/pbzx"
	"github.com/blacktop/ipsw/internal/magic"
)

func main() {
	inPath := "AppleSoftwareUpdate.pbzx"

	// Verify the file is a valid PBZX container.
	ok, err := magic.IsPBZX(inPath)
	if err != nil || !ok {
		panic("not a PBZX file")
	}

	f, err := os.Open(inPath)
	if err != nil {
		panic(err)
	}
	defer f.Close()

	var out bytes.Buffer
	
	// Extract using all available CPU cores.
	err = pbzx.Extract(context.Background(), f, &out, runtime.NumCPU())
	if err != nil {
		panic(err)
	}

	// Write the concatenated output.
	outputPath := inPath + ".extracted"
	err = os.WriteFile(outputPath, out.Bytes(), 0o644)
	if err != nil {
		panic(err)
	}
	
	fmt.Printf("Extracted %d bytes to %s\n", out.Len(), outputPath)
}

```

*Key API:* `pbzx.Extract(ctx, src, dst, workers)` defined in [`pkg/ota/pbzx/extract.go`](https://github.com/blacktop/ipsw/blob/main/pkg/ota/pbzx/extract.go).

## Summary

- **LZFSE Detection**: `ipsw` scans the first four bytes for the `bvx2` magic string in [`pkg/img3/img3.go`](https://github.com/blacktop/ipsw/blob/main/pkg/img3/img3.go) and other firmware parsers.
- **LZFSE Decoding**: A pure-Go implementation in `pkg/lzfse` uses `NewDecoder` and `DecodeBuffer` to rebuild FSE tables and execute sliding-window matches.
- **PBZX Detection**: The [`internal/magic/magic.go`](https://github.com/blacktop/ipsw/blob/main/internal/magic/magic.go) package validates the `pbzx` header before processing.
- **PBZX Extraction**: `pkg/ota/pbzx` implements a concurrent three-stage pipeline (read, inflate via `github.com/xi2/xz`, ordered write) that preserves chunk sequence using a min-heap.

## Frequently Asked Questions

### How does ipsw detect LZFSE-compressed data?

`ipsw` detects LZFSE compression by checking the first four bytes of a payload for the ASCII string `bvx2`. This check appears in [`pkg/img3/img3.go`](https://github.com/blacktop/ipsw/blob/main/pkg/img3/img3.go) within the `DecryptData` function, where decrypted payloads are scanned before decompression. The same detection logic is reused in [`pkg/ftab/ftab.go`](https://github.com/blacktop/ipsw/blob/main/pkg/ftab/ftab.go) and [`pkg/devicetree/devicetree.go`](https://github.com/blacktop/ipsw/blob/main/pkg/devicetree/devicetree.go) for other firmware components.

### What is the architecture of the LZFSE decoder in ipsw?

The LZFSE decoder is implemented as a pure-Go package in `pkg/lzfse`. The entry point `NewDecoder` initializes a scratch buffer and a growing output buffer, while `DecodeBuffer` orchestrates the decompression. Internally, `decodeV1`, `decodeV2HeaderSize`, and `decodeLmdExecuteMatch` parse block headers, rebuild Finite State Entropy (FSE) tables for literal and match streams, and execute sliding-window copy operations to reconstruct the original data.

### How does ipsw handle out-of-order chunks during PBZX extraction?

`ipsw` handles out-of-order decompression using a min-heap in the writer stage ([`pkg/ota/pbzx/write.go`](https://github.com/blacktop/ipsw/blob/main/pkg/ota/pbzx/write.go)). Because XZ decompression happens concurrently across multiple workers, chunks may complete out of sequence. The writer buffers completed chunks in the heap keyed by their original index, writing them to the output stream only when the next expected index arrives. This guarantees that the final extracted file maintains the correct byte order despite parallel processing.

### Can I use ipsw's LZFSE and PBZX packages in my own Go programs?

Yes, both packages are designed as reusable Go modules. For LZFSE, import `github.com/blacktop/ipsw/pkg/lzfse` and use `lzfse.NewDecoder(data).DecodeBuffer()` to decompress buffers. For PBZX, import `github.com/blacktop/ipsw/pkg/ota/pbzx` and call `pbzx.Extract(ctx, src, dst, workers)` with an `io.Reader` and `io.Writer`. The `internal/magic` package provides `magic.IsPBZX` for validation before extraction.