Deduplication Mechanism in BLEReceivePipeline: How Bitchat Prevents Duplicate BLE Packets

The BLEReceivePipeline uses a three‑component deduplication system: a context generator that flags packets for deduplication, a thread‑safe LRU cache of message IDs, and a drop‑logic handler that cancels redundant relays for duplicates in multi‑peer networks.

The BLEReceivePipeline in permissionlesstech/bitchat ensures each Bluetooth Low‑Energy (BLE) packet is processed exactly once. This article explains how the deduplication mechanism works by examining the coordinated roles of BLEReceivePipeline.context(), MessageDeduplicator, and BLEService.dropDuplicatePacketIfNeeded.

Components of the BLEReceivePipeline Deduplication System

BLEReceivePipeline.context(for:localPeerID:)

The pipeline's entry point for deduplication decisions lives in BLEReceivePipeline.swift (lines 13‑33). This static method constructs a packet context that determines whether a packet should be deduplicated.

// Simplified flow from BLEReceivePipeline.swift
let ctx = BLEReceivePipeline.context(for: packet, localPeerID: myPeerID)
// Returns BLEReceivedPacketContext with:
//   - messageID: deterministic hash of sender + timestamp + type + payload digest
//   - shouldDeduplicate: true for most packets, false for fragments and self‑sync replays

The messageID is built from four fields: sender ID, timestamp, packet type, and a SHA‑256 digest of the payload. This ensures two packets with different content produce distinct IDs. The shouldDeduplicate flag is set to false for two edge cases: fragment packets (which must be reassembled before deduplication) and self‑sync replays (packets with TTL = 0 where sender equals local peer).

MessageDeduplicator

The actual duplicate detection happens in MessageDeduplicator.swift (lines 34‑53). This class maintains a thread‑safe LRU cache bounded by both time and count.

// MessageDeduplicator.swift – core interface
final class MessageDeduplicator {
    func isDuplicate(_ messageID: String) -> Bool
    // Checks cache; stores ID with timestamp if new
}

Duplicate detection uses TransportConfig parameters: maxAge (time‑based expiry) and maxCount (capacity limit). When isDuplicate() finds an existing ID, the packet is flagged; otherwise, the ID enters the cache.

BLEService.dropDuplicatePacketIfNeeded

Integration occurs in BLEService.swift (lines 62‑78). After parsing produces a BLEReceivedPacketContext, this method executes the drop logic.

// From BLEService.swift receive path
if context.shouldDeduplicate && deduplicator.isDuplicate(messageID) {
    // Drop duplicate
    if connectedPeerCount > 2 {
        BLEReceivePipeline.shouldCancelScheduledRelayForDuplicate(messageID)
    }
    return
}

A critical optimization activates when the device maintains more than two peer connections. In this case, the pipeline cancels any scheduled relay for the duplicate message, preventing unnecessary network traffic.

Step‑by‑Step Deduplication Flow

1. Packet reception – BLEService receives a serialized BitchatPacket from the BLE stack.

2. Context creation – BLEReceivePipeline.context() builds the messageID and sets shouldDeduplicate based on packet type and source.

3. Duplicate check – BLEService queries the shared MessageDeduplicator instance with the computed ID.

4. Drop or cancel – Duplicates are discarded immediately; for well‑connected nodes, pending relays are cancelled to conserve bandwidth.

Code Examples

Basic Pipeline Usage

// Inside BLEService packet handler
let context = BLEReceivePipeline.context(
    for: receivedPacket,
    localPeerID: localPeerID
)

let isDuplicate = BLEService.dropDuplicatePacketIfNeeded(
    context: context,
    messageID: context.messageID
)

guard !isDuplicate else {
    // Packet already processed—skip
    return
}

// Continue with unique packet processing
processValidPacket(receivedPacket)

Standalone MessageDeduplicator

import Foundation

let dedup = MessageDeduplicator(
    maxAge: 30,    // seconds
    maxCount: 500  // cached IDs
)

func handleUserMessage(_ id: String, content: String) {
    guard !dedup.isDuplicate(id) else {
        print("Discarding duplicate: \(id)")
        return
    }
    print("Processing new message: \(content)")
    // Forward to UI or persistence layer
}

What the Unit Tests Verify

The test suite in BLEReceivePipelineTests.swift validates two boundary conditions:

• context.shouldDeduplicate returns true for normal packets from remote peers.
• context.shouldDeduplicate returns false for fragment packets and self‑sync replays (TTL = 0, sender = local peer).

These tests prevent regressions in the logic that shields the deduplicator from inappropriate inputs.

Summary

• BLEReceivePipeline.context() generates deterministic message IDs and exclusion flags in BLEReceivePipeline.swift.
• MessageDeduplicator provides thread‑safe, bounded caching in MessageDeduplicator.swift.
• BLEService.dropDuplicatePacketIfNeeded integrates the check and cancels redundant relays when connected to 3+ peers.
• The system distinguishes legitimate packets from fragments and self‑sync replays to avoid false positives.

Frequently Asked Questions

How does BLEReceivePipeline generate unique message IDs?

The pipeline concatenates sender ID, timestamp, packet type, and a truncated SHA‑256 hash of the payload. This construction ensures that packets differing only in transit metadata (like BLE advertisement address) still produce distinct IDs if their content varies.

Why are fragment packets excluded from deduplication?

Fragment packets carry partial payload data. Deduplicating them at the BLE layer would prevent proper reassembly of the complete message. The pipeline defers duplicate detection to the reassembly stage by setting shouldDeduplicate = false for all fragment types.

What triggers relay cancellation for duplicates?

When BLEService.dropDuplicatePacketIfNeeded detects a duplicate and connectedPeerCount > 2, it invokes BLEReceivePipeline.shouldCancelScheduledRelayForDuplicate. This prevents the node from redundantly forwarding a message that has already reached sufficient peers.

Can the deduplication cache size be configured?

Yes. MessageDeduplicator initializes with maxAge and maxCount parameters sourced from TransportConfig. The defaults balance memory usage against the expected message velocity in typical mesh deployments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}