# Deduplication Mechanism in BLEReceivePipeline: How Bitchat Prevents Duplicate BLE Packets

> Discover how Bitchat's BLEReceivePipeline prevents duplicate BLE packets with its three-component deduplication system: context generator, LRU cache, and drop-logic handler.

- Repository: [permissionlesstech/bitchat](https://github.com/permissionlesstech/bitchat)
- Tags: internals
- Published: 2026-08-04

---

**The BLEReceivePipeline uses a three‑component deduplication system: a context generator that flags packets for deduplication, a thread‑safe LRU cache of message IDs, and a drop‑logic handler that cancels redundant relays for duplicates in multi‑peer networks.**

The **BLEReceivePipeline** in [permissionlesstech/bitchat](https://github.com/permissionlesstech/bitchat) ensures each Bluetooth Low‑Energy (BLE) packet is processed exactly once. This article explains how the deduplication mechanism works by examining the coordinated roles of `BLEReceivePipeline.context()`, `MessageDeduplicator`, and `BLEService.dropDuplicatePacketIfNeeded`.

## Components of the BLEReceivePipeline Deduplication System

### BLEReceivePipeline.context(for:localPeerID:)

The pipeline's entry point for deduplication decisions lives in [`BLEReceivePipeline.swift`](https://github.com/permissionlesstech/bitchat/blob/main/BLEReceivePipeline.swift) (lines 13‑33). This static method constructs a **packet context** that determines whether a packet should be deduplicated.

```swift
// Simplified flow from BLEReceivePipeline.swift
let ctx = BLEReceivePipeline.context(for: packet, localPeerID: myPeerID)
// Returns BLEReceivedPacketContext with:
//   - messageID: deterministic hash of sender + timestamp + type + payload digest
//   - shouldDeduplicate: true for most packets, false for fragments and self‑sync replays

```

The `messageID` is built from four fields: sender ID, timestamp, packet type, and a **SHA‑256 digest of the payload**. This ensures two packets with different content produce distinct IDs. The `shouldDeduplicate` flag is set to `false` for two edge cases: **fragment packets** (which must be reassembled before deduplication) and **self‑sync replays** (packets with TTL = 0 where sender equals local peer).

### MessageDeduplicator

The actual duplicate detection happens in [`MessageDeduplicator.swift`](https://github.com/permissionlesstech/bitchat/blob/main/MessageDeduplicator.swift) (lines 34‑53). This class maintains a **thread‑safe LRU cache** bounded by both time and count.

```swift
// MessageDeduplicator.swift – core interface
final class MessageDeduplicator {
    func isDuplicate(_ messageID: String) -> Bool
    // Checks cache; stores ID with timestamp if new
}

```

Duplicate detection uses `TransportConfig` parameters: `maxAge` (time‑based expiry) and `maxCount` (capacity limit). When `isDuplicate()` finds an existing ID, the packet is flagged; otherwise, the ID enters the cache.

### BLEService.dropDuplicatePacketIfNeeded

Integration occurs in [`BLEService.swift`](https://github.com/permissionlesstech/bitchat/blob/main/BLEService.swift) (lines 62‑78). After parsing produces a `BLEReceivedPacketContext`, this method executes the drop logic.

```swift
// From BLEService.swift receive path
if context.shouldDeduplicate && deduplicator.isDuplicate(messageID) {
    // Drop duplicate
    if connectedPeerCount > 2 {
        BLEReceivePipeline.shouldCancelScheduledRelayForDuplicate(messageID)
    }
    return
}

```

A critical optimization activates when the device maintains **more than two peer connections**. In this case, the pipeline cancels any **scheduled relay** for the duplicate message, preventing unnecessary network traffic.

## Step‑by‑Step Deduplication Flow

1. **Packet reception** – `BLEService` receives a serialized `BitchatPacket` from the BLE stack.

2. **Context creation** – `BLEReceivePipeline.context()` builds the `messageID` and sets `shouldDeduplicate` based on packet type and source.

3. **Duplicate check** – `BLEService` queries the shared `MessageDeduplicator` instance with the computed ID.

4. **Drop or cancel** – Duplicates are discarded immediately; for well‑connected nodes, pending relays are cancelled to conserve bandwidth.

## Code Examples

### Basic Pipeline Usage

```swift
// Inside BLEService packet handler
let context = BLEReceivePipeline.context(
    for: receivedPacket,
    localPeerID: localPeerID
)

let isDuplicate = BLEService.dropDuplicatePacketIfNeeded(
    context: context,
    messageID: context.messageID
)

guard !isDuplicate else {
    // Packet already processed—skip
    return
}

// Continue with unique packet processing
processValidPacket(receivedPacket)

```

### Standalone MessageDeduplicator

```swift
import Foundation

let dedup = MessageDeduplicator(
    maxAge: 30,    // seconds
    maxCount: 500  // cached IDs
)

func handleUserMessage(_ id: String, content: String) {
    guard !dedup.isDuplicate(id) else {
        print("Discarding duplicate: \(id)")
        return
    }
    print("Processing new message: \(content)")
    // Forward to UI or persistence layer
}

```

## What the Unit Tests Verify

The test suite in [`BLEReceivePipelineTests.swift`](https://github.com/permissionlesstech/bitchat/blob/main/BLEReceivePipelineTests.swift) validates two boundary conditions:

- `context.shouldDeduplicate` returns **true** for normal packets from remote peers.
- `context.shouldDeduplicate` returns **false** for fragment packets and self‑sync replays (TTL = 0, sender = local peer).

These tests prevent regressions in the logic that shields the deduplicator from inappropriate inputs.

## Summary

- **BLEReceivePipeline.context()** generates deterministic message IDs and exclusion flags in [`BLEReceivePipeline.swift`](https://github.com/permissionlesstech/bitchat/blob/main/BLEReceivePipeline.swift).
- **MessageDeduplicator** provides thread‑safe, bounded caching in [`MessageDeduplicator.swift`](https://github.com/permissionlesstech/bitchat/blob/main/MessageDeduplicator.swift).
- **BLEService.dropDuplicatePacketIfNeeded** integrates the check and cancels redundant relays when connected to 3+ peers.
- The system distinguishes legitimate packets from fragments and self‑sync replays to avoid false positives.

## Frequently Asked Questions

### How does BLEReceivePipeline generate unique message IDs?

The pipeline concatenates sender ID, timestamp, packet type, and a truncated SHA‑256 hash of the payload. This construction ensures that packets differing only in transit metadata (like BLE advertisement address) still produce distinct IDs if their content varies.

### Why are fragment packets excluded from deduplication?

Fragment packets carry partial payload data. Deduplicating them at the BLE layer would prevent proper reassembly of the complete message. The pipeline defers duplicate detection to the reassembly stage by setting `shouldDeduplicate = false` for all fragment types.

### What triggers relay cancellation for duplicates?

When `BLEService.dropDuplicatePacketIfNeeded` detects a duplicate and `connectedPeerCount > 2`, it invokes `BLEReceivePipeline.shouldCancelScheduledRelayForDuplicate`. This prevents the node from redundantly forwarding a message that has already reached sufficient peers.

### Can the deduplication cache size be configured?

Yes. `MessageDeduplicator` initializes with `maxAge` and `maxCount` parameters sourced from `TransportConfig`. The defaults balance memory usage against the expected message velocity in typical mesh deployments.