# How Protocol‑Reverse Recovers Custom Protocols: A 4‑Phase Reverse Engineering Methodology

> Learn how protocol-reverse recovers custom binary protocols, Protobuf/gRPC, and WebSocket frames using a 4-phase methodology for efficient reverse engineering.

- Repository: [ZhaoXu/reverse-skill](https://github.com/zhaoxuya520/reverse-skill)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Protocol‑reverse is a systematic skill that recovers proprietary binary protocols, Protobuf/gRPC definitions, WebSocket frames, and PCAP‑driven protocol layouts through four progressive phases: Collection & Triage, Frame Layout Reconstruction, Serialization & Encryption Analysis, and Deliverable Generation.**

The `protocol-reverse` skill in the [zhaoxuya520/reverse‑skill](https://github.com/zhaoxuya520/reverse-skill) repository provides a battle‑tested framework for security researchers and reverse engineers to decode unknown network protocols. Unlike ad‑hoc packet analysis, this methodology ensures reproducible results and complete documentation. This article examines how protocol‑reverse recovers custom protocols by walking through its architecture, key techniques, and concrete code implementations as defined in [[`skills/protocol-reverse/SKILL.md`](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/protocol-reverse/SKILL.md)](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/protocol-reverse/SKILL.md).

## Phase 1: Collection & Triage – Capturing Protocol Samples

The foundation of protocol‑reverse is acquiring high‑quality traffic samples. Analysts gather **PCAP files**, proxy exports, client logs, or raw binaries, then annotate directionality (client→server versus server→client).

### Identifying Framing Cues

During triage, look for structural patterns that reveal message boundaries:

- **Fixed headers** with constant size (typically 2–4 bytes)
- **Magic numbers** serving as protocol identifiers
- **Length fields** indicating payload size
- **TLV (Type‑Length‑Value)** repeating patterns
- **Compression or encryption layers** obscuring cleartext

The skill recommends `tshark` for rapid TCP payload extraction. Per [[`protocol-workflow.md`](https://github.com/zhaoxuya520/reverse-skill/blob/main/protocol-workflow.md)](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/protocol-reverse/references/protocol-workflow.md):

```bash
tshark -r cap.pcap -T fields -e frame.number -e ip.src -e tcp.payload

```

Filter by specific ports when targeting known services:

```bash
tshark -r capture.pcap -Y "tcp.port==4433" -T fields -e tcp.payload | head

```

## Phase 2: Frame Layout Reconstruction

Once samples are collected, protocol‑reverse reconstructs the frame layout by aligning multiple instances of the same message type. This phase separates **invariant bytes** (headers, magic values) from **dynamic fields** (sequence counters, timestamps, payload lengths).

### Critical Analysis Steps

1. **Determine endianness** – big‑endian versus little‑endian length encodings
2. **Calculate length field semantics** – inclusive versus exclusive of header bytes
3. **Locate integrity checks** – CRC, checksum, or HMAC trailing bytes
4. **Map state transitions** – sketch the protocol state machine (Connect → Auth → Ready → Request/Response → Close)

For repeatable analysis, the skill suggests creating **Wireshark custom dissectors** or **Kaitai Struct templates**. A minimal Lua dissector from the workflow documentation:

```lua
protocol = Proto("myproto","My Custom Protocol")
function protocol.dissector(buf,pinfo,tree)
    pinfo.cols.protocol = "MYPROTO"
    local subtree = tree:add(protocol,buf())
    subtree:add(buf(0,4), "Magic")
    subtree:add(buf(4,2), "Length")
    subtree:add(buf(6,4), "Message Type")
end
DissectorTable.get("tcp.port"):add(4433,protocol)

```

## Phase 3: Serialization & Encryption Analysis

Modern protocols rarely use raw binary frames. Protocol‑reverse addresses three common serialization layers: **Protobuf**, **gRPC**, and **custom encryption**.

### Recovering Protobuf Definitions

Protobuf messages encode field numbers as varints, making them detectable via entropy analysis. The skill recommends three recovery approaches:

- **`blackboxprotobuf`** – automated inference of .proto structures from raw binary
- **pbtk** – Protobuf toolkit for definition extraction
- **`protoc --decode_raw`** – manual field number inspection

Install and run blackboxprotobuf:

```bash
pip install blackboxprotobuf
python -c "import blackboxprotobuf, sys; data = open('msg.bin','rb').read(); print(blackboxprotobuf.decode_message(data))"

```

### Handling gRPC Traffic

gRPC layers HTTP/2 headers over protobuf bodies. Recovery requires:
1. HTTP/2 header inspection for method names and routing
2. Protobuf body decoding using tools above

### Decrypting Encrypted Protocols

Encrypted frames exhibit high entropy with no cleartext URLs or structure. Protocol‑reverse routes to companion skills for key material extraction:

- **`ida-reverse`** – static analysis of compiled client binaries
- **`js-reverse`** – debugging JavaScript/WebSocket clients
- **`apk-reverse`** – Android application disassembly

Extract keys from the client binary, then implement decryption before returning to frame parsing.

### Reference Parser Implementation

The workflow file provides a Python skeleton for fixed‑header frames. This demonstrates how protocol‑reverse recovers custom protocols programmatically:

```python
import struct

def parse_frame(buf: bytes):
    # >   big-endian

    # I   4-byte magic

    # H   2-byte length

    # I   4-byte message type

    magic, length, msg_type = struct.unpack_from(">IHI", buf, 0)
    body = buf[10:10 + length]
    return {"magic": magic, "type": msg_type, "body": body}

```

## Phase 4: Deliverable Generation

Protocol recovery is incomplete without documentation. The skill's self‑checklist in [`SKILL.md`](https://github.com/zhaoxuya520/reverse-skill/blob/main/SKILL.md) mandates three deliverables:

| Artifact | Requirement |
|----------|-------------|
| **Message‑type table** | Name, opcode, field definitions for each message type |
| **Reproducible decoder** | Command or script that parses captured traffic |
| **Evidence package** | Raw hex excerpt plus sanitized decoded result |

These artifacts enable peer review, regression testing, and handoff to downstream security activities.

## Routing and Skill Integration

Protocol‑reverse does not operate in isolation. The [[`skills/config/routing.json`](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/config/routing.json)](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/config/routing.json) defines upstream and downstream links:

- **Upstream**: Binary reverse‑engineering skills (`ida-reverse`, `js-reverse`, `apk-reverse`) feed extracted keys and client behavior
- **Downstream**: Penetration‑testing utilities (`pentest-tools`) consume recovered protocols for replay attacks or fuzzing

This routing ensures protocol‑reverse integrates into comprehensive security assessments.

## Protocol Pattern Reference

| Pattern | Indicator | Tool |
|---------|-----------|------|
| Fixed‑length header | Constant initial bytes | Python `struct` module |
| Magic number | Known constants like `0xDEAD` | ImHex binary search |
| TLV structure | Repeating type‑length‑value triples | Custom Wireshark dissector |
| Protobuf encoding | Varint field numbers | `blackboxprotobuf`, `protoc` |
| Encrypted payload | High entropy, no cleartext | Key extraction via `ida-reverse`/`js-reverse` |

## Summary

- **Protocol‑reverse** employs a **four‑phase methodology**: Collection & Triage → Frame Layout Reconstruction → Serialization & Encryption Analysis → Deliverable Generation.

- **Key diagnostic patterns** include fixed headers, magic numbers, TLV structures, and Protobuf varints—each with recommended tooling in the workflow documentation.

- **Critical code resources** reside in [[`skills/protocol-reverse/SKILL.md`](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/protocol-reverse/SKILL.md)](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/protocol-reverse/SKILL.md) (master specification) and [[`skills/protocol-reverse/references/protocol-workflow.md`](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/protocol-reverse/references/protocol-workflow.md)](https://github.com/zhaoxuya520/reverse-skill/blob/main/skills/protocol-reverse/references/protocol-workflow.md) (quick‑reference with Python skeletons and `tshark` commands).

- **Skill routing** connects protocol‑reverse to binary reverse‑engineering for key extraction and penetration testing for exploitation, forming a complete reverse engineering pipeline.

## Frequently Asked Questions

### What file format does protocol‑reverse use for traffic samples?

Protocol‑reverse primarily uses **PCAP files** as the standard input format, captured via tools like Wireshark, tcpdump, or mitmproxy. The workflow also accepts proxy exports, client logs, and raw binary blobs when PCAP is unavailable. All formats ultimately convert to byte sequences for frame analysis.

### How does protocol‑reverse distinguish between Protobuf and custom binary protocols?

Protobuf exhibits specific signatures: **varint‑encoded field numbers** (typically values 1–15 encoded as single bytes `0x08`–`0x78`), **zigzag encoding** for signed integers, and characteristic field delimiters. Custom binary protocols more commonly use **fixed‑width fields**, **magic numbers at fixed offsets**, and **explicit length fields**. The skill applies entropy analysis and structure heuristics before selecting the appropriate decoder.

### Can protocol‑reverse handle encrypted protocols without source code access?

Yes, through **key material extraction from client binaries**. When encryption is detected, protocol‑reverse routes to `ida-reverse`, `js-reverse`, or `apk-reverse` skills to statically analyze the client application. Analysts locate key‑derivation functions, hardcoded keys, or cryptographic constants, then implement decryption before resuming frame parsing. This approach succeeds against proprietary encryption when the key resides in the client rather than using true key exchange.

### What makes protocol‑reverse different from standard Wireshark analysis?

Standard Wireshark analysis is **reactive and manual**—analysts inspect packets without systematic documentation. Protocol‑reverse is **procedural and reproducible**: it mandates specific phases, requires deliverable artifacts, and integrates with a broader skill ecosystem. The framework ensures that protocol knowledge persists beyond individual analysts and supports automated downstream testing.