What Is the Scrcpy Packet Merger and How It Handles Video Streams

The Scrcpy packet merger is a specialized component in app/src/packet_merger.c that intercepts H.264 and H.265 configuration packets and prepends them to the first media frame, ensuring FFmpeg decoders receive the necessary SPS/PPS extradata for correct video decoding.

The packet merger in Genymobile/scrcpy solves a critical synchronization challenge between Android’s MediaCodec encoder and FFmpeg’s downstream decoder. When streaming H.26x video, the encoder transmits configuration data separately from media frames, but the decoder expects these combined into a single packet. This article examines how the packet merger detects, stores, and prepends configuration data while leaving audio streams completely untouched.

Why Scrcpy Needs a Packet Merger for H.26x Video

Android’s hardware encoder sends configuration packets containing Sequence Parameter Set (SPS) and Picture Parameter Set (PPS) NAL units before the first video frame. These packets have no presentation timestamp (pts == AV_NOPTS_VALUE), distinguishing them from regular media packets.

The downstream FFmpeg decoder, however, requires this configuration data to be prefixed to the first media packet of each encoding session. If the config packet is delivered separately, the recorder loses the extradata needed for correct decoding, resulting in corrupted or unplayable video files—particularly after device orientation changes that trigger a new encoder session.

How the Packet Merger Detects Configuration Packets

Identifying Config Packets via Presentation Timestamp

The entry point sc_packet_merger_merge() in app/src/packet_merger.c detects configuration packets by checking the presentation timestamp at line 21:

if (packet->pts == AV_NOPTS_VALUE) {
    // This is a config packet
}

When pts == AV_NOPTS_VALUE, the function treats the packet as configuration data rather than a media frame.

Storing SPS/PPS Data in the Merger Structure

Upon detecting a config packet, the merger frees any previous configuration, allocates a new buffer, and copies the raw bytes into merger->config (lines 24–34 in app/src/packet_merger.c):

// Free previous config if exists
av_freep(&merger->config);

// Allocate and copy new config data
merger->config = av_malloc(packet->size);
memcpy(merger->config, packet->data, packet->size);
merger->config_size = packet->size;

This preserves the configuration data until the first media packet arrives.

Merging Config Packets with Media Frames

When a normal media packet arrives (with a valid PTS) and a config is pending, the merger performs three operations to combine them:

1. Expand the packet using av_grow_packet() to make room for the config data (lines 38–42)
2. Shift the original payload using memmove() to create space at the beginning (line 43)
3. Copy the config to the start using memcpy() (lines 44–45)

// Grow packet to fit config + original data
int ret = av_grow_packet(packet, merger->config_size);
if (ret < 0) {
    return false;
}

// Shift original data to make room for config
memmove(packet->data + merger->config_size, packet->data, packet->size - merger->config_size);

// Prepend config data
memcpy(packet->data, merger->config, merger->config_size);

// Clean up
av_freep(&merger->config);
merger->config_size = 0;

After the merge, the temporary config buffer is freed and the merger is ready for the next configuration packet.

Audio Stream Handling and When the Merger Is Active

The packet merger is only instantiated for H.26x video codecs. The demuxer in app/src/demuxer.c decides whether merging is required based on the codec ID (lines 26–30):

bool must_merge_config_packet = raw_codec_id == SC_CODEC_ID_H264
                             || raw_codec_id == SC_CODEC_ID_H265;

For audio streams (Opus, FLAC, AAC) or any codec that does not use in-band configuration, must_merge_config_packet remains false and the merger is never created. This ensures zero overhead for audio processing and prevents unnecessary memory operations on audio packets.

When merging is needed, the demuxer creates a struct sc_packet_merger at the start of the thread, calls sc_packet_merger_init(), and feeds every received packet to sc_packet_merger_merge() before pushing it to the packet sinks. After the demuxer finishes, it destroys the merger with sc_packet_merger_destroy().

Implementation Details and Source File Locations

The packet merger implementation spans three core files in the Scrcpy codebase:

File	Role
app/src/packet_merger.h	Defines the struct sc_packet_merger and public API (sc_packet_merger_init, sc_packet_merger_merge, sc_packet_merger_destroy)
app/src/packet_merger.c	Implements the merge logic: detecting config packets via pts == AV_NOPTS_VALUE, storing SPS/PPS data, and prepending to media frames
app/src/demuxer.c	Orchestrates the merger lifecycle: conditionally instantiates it for H.264/H.265, calls merge on every packet, and handles cleanup

Initializing the Merger

Inside the demuxer thread, the merger is initialized only when required:

struct sc_packet_merger merger;

if (must_merge_config_packet) {
    sc_packet_merger_init(&merger);
}

Processing the Packet Stream

Each packet flows through the merger before reaching decoders:

bool ok = sc_demuxer_recv_packet(demuxer, packet);
if (!ok) break;

if (must_merge_config_packet) {
    ok = sc_packet_merger_merge(&merger, packet);
    if (!ok) {
        av_packet_unref(packet);
        break;
    }
}

Cleanup

When the stream ends, resources are released:

if (must_merge_config_packet) {
    sc_packet_merger_destroy(&merger);
}

Summary

• The packet merger in app/src/packet_merger.c ensures H.264 and H.265 video streams include necessary configuration data (SPS/PPS NAL units) with the first media frame.
• It detects configuration packets by checking for pts == AV_NOPTS_VALUE and stores them until the first valid media packet arrives.
• The merger prepends configuration data to media frames using av_grow_packet, memmove, and memcpy, then frees the temporary buffer.
• Audio streams (Opus, FLAC, AAC) bypass the merger entirely, as the demuxer in app/src/demuxer.c only instantiates the component for SC_CODEC_ID_H264 and SC_CODEC_ID_H265.

Frequently Asked Questions

Does the packet merger introduce latency in video streaming?

No, the packet merger operates synchronously within the demuxer thread without adding buffering delays. It performs simple memory operations—memcpy, memmove, and av_grow_packet—which execute in microseconds. The component only holds one configuration packet at a time, ensuring minimal memory overhead and zero perceptible latency.

Why does FFmpeg require configuration data to be prepended to video frames?

FFmpeg decoders for H.264 and H.265 require Sequence Parameter Set (SPS) and Picture Parameter Set (PPS) NAL units to initialize the decoding context. Android’s MediaCodec sends these as separate packets with AV_NOPTS_VALUE, but FFmpeg expects them prefixed to the first keyframe. Without this prepended data, the decoder cannot interpret the bitstream, resulting in decoding failures or corrupted output.

Is the packet merger used for all video codecs in Scrcpy?

No, the packet merger is only active for H.264 (SC_CODEC_ID_H264) and H.265 (SC_CODEC_ID_H265) codecs. The demuxer explicitly checks the codec ID before instantiating the merger. Other video codecs that do not use in-band configuration packets, as well as all audio codecs like Opus and FLAC, bypass this component entirely to avoid unnecessary processing overhead.

What happens if a configuration packet is lost during transmission?

If a configuration packet is lost, the packet merger would have no stored SPS/PPS data to prepend to the first media frame. Consequently, the FFmpeg decoder would receive a raw video frame without the necessary initialization parameters, causing it to fail decoding until a new keyframe with embedded configuration data arrives. In practice, Scrcpy’s socket transmission is reliable (TCP/local), making packet loss extremely rare, but the merger’s design assumes reliable ordered delivery.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}