# How to Integrate fabrica-util's Bloom Filter for Efficient Existence Checks in Large Game Datasets

> Integrate fabrica-util's Bloom filter to efficiently check for game data existence. Add and query elements to bypass database trips and speed up your application.

- Repository: [Pantheon/fabrica-util](https://github.com/go-pantheon/fabrica-util)
- Tags: how-to-guide
- Published: 2026-03-02

---

**To integrate fabrica-util's Bloom filter, initialize `bloom.NewInt64Bloom()` with your expected element count and target false-positive probability, then populate it via `Add()` or `MAdd()` and query membership using `Contains()` to eliminate unnecessary database round-trips.**

The go-pantheon/fabrica-util library provides a production-ready, thread-safe Bloom filter implementation specifically designed for high-throughput game backends. By integrating this package, you can perform constant-time existence checks on massive collections of player IDs, item GUIDs, or quest identifiers while consuming only a fraction of the memory required by traditional hash maps.

## How the fabrica-util Bloom Filter Works

The implementation centers on four core components that work together to provide deterministic, concurrent-safe membership testing:

- **Bitmap Storage** ([`bitmap/bitmap.go`](https://github.com/go-pantheon/fabrica-util/blob/main/bitmap/bitmap.go)): A thread-safe bit array that stores the filter bits under a mutex, guaranteeing safe concurrent use across multiple game server goroutines.
- **Int64BloomFilter** ([`bloom/bloom.go`](https://github.com/go-pantheon/fabrica-util/blob/main/bloom/bloom.go)): The primary struct wrapping the bitmap and a slice of hash functions (`[]func(int64) int64`). It exposes `Add`, `MAdd` (batch add), and `Contains` for membership operations.
- **Parameter Estimation** (`estimateParameters`): Located in [`bloom/bloom.go`](https://github.com/go-pantheon/fabrica-util/blob/main/bloom/bloom.go) (lines 72-78), this function calculates the optimal bitmap size `m` and number of hash functions `k` using the classic Bloom filter formulas based on expected element count `n` and target false-positive probability `p`.
- **Hash Function Generation** (`createInt64HashFunctions`): Generates up to eight deterministic 64-bit hash functions using large prime seeds. If more than eight functions are required, a fallback variation is produced to maintain dispersion while keeping the implementation compact.

Because the filter stores only bits, memory consumption scales linearly with `m` (approximately `-n·ln(p)/ln²(2)` bits) rather than with the number of stored elements. For example, according to the `BenchmarkInt64Bloom` tests in [`bloom/bloom_test.go`](https://github.com/go-pantheon/fabrica-util/blob/main/bloom/bloom_test.go), a filter configured for 1 million IDs with a 1% false-positive target occupies roughly 12 MiB.

## When to Use Bloom Filters in Game Servers

Integrate the fabrica-util Bloom filter when you need to minimize memory footprint while maintaining fast rejection capabilities for large datasets:

1. **Static Collection Validation**: Pre-load all known item IDs or quest identifiers at server startup to instantly reject invalid client requests without querying persistent storage.
2. **Database Pre-Filtering**: Check `Contains()` before hitting your database or cache layer. A `false` result guarantees absence, preventing unnecessary round-trips for non-existent entities.
3. **High-Throughput Existence Checks**: Validate player existence during matchmaking or world transfers without acquiring heavy locks on large identity maps.

**Critical Limitation**: A `true` result from `Contains()` indicates only "maybe present." Always verify against your authoritative data store before executing gameplay-critical actions, such as awarding items or completing transactions.

## Implementation Guide: Integrating fabrica-util's Bloom Filter

### Initializing the Filter with Optimal Parameters

Create a new filter using `NewInt64Bloom(n, p)`, where `n` is the expected number of elements and `p` is your acceptable false-positive probability (e.g., `0.01` for 1%). The constructor internally calls `estimateParameters` to configure the underlying `bitmap.Bitmap` size and the number of hash functions generated by `createInt64HashFunctions`.

```go
package main

import (
	"fmt"
	"github.com/go-pantheon/fabrica-util/bloom"
)

func main() {
	// Create a filter for 500,000 player IDs with ≤1% false-positive rate
	bf := bloom.NewInt64Bloom(500_000, 0.01)
	
	fmt.Printf("Bitmap size: %d bits\n", bf.Size())
}

```

### Adding Game Entity IDs

Populate the filter using `Add(int64)` for individual IDs during runtime, or `MAdd([]int64)` for efficient batch loading during server initialization.

```go
// Single additions during runtime
playerIDs := []int64{101, 202, 303, 404, 505}
for _, id := range playerIDs {
	bf.Add(id)
}

// Batch addition during startup (e.g., loading from database)
var bulk []int64
for i := int64(0); i < 1_000_000; i++ {
	bulk = append(bulk, i+1_000_000)
}
bf.MAdd(bulk)

```

### Performing Fast Existence Checks

Query membership using `Contains(int64)`. This operation executes in O(1) time relative to the number of hash functions, not the dataset size.

```go
if bf.Contains(202) {
	fmt.Println("ID 202 might exist – proceed with full lookup.")
}

if !bf.Contains(999) {
	fmt.Println("ID 999 is definitely not in the dataset.")
}

```

### Handling Concurrency in Game Servers

The underlying `bitmap.Bitmap` implementation in [`bitmap/bitmap.go`](https://github.com/go-pantheon/fabrica-util/blob/main/bitmap/bitmap.go) uses a mutex to protect bit operations, making the filter safe for concurrent use across multiple goroutines handling player connections.

```go
var wg sync.WaitGroup
for _, id := range largeIDSlice {
	wg.Add(1)
	go func(i int64) {
		defer wg.Done()
		bf.Add(i) // Safe concurrent write
	}(id)
}
wg.Wait()

```

## Advanced Tuning for Production Workloads

For stricter accuracy requirements, reduce the false-positive probability. Access the filter configuration via `Size()` and `HashFuncs()` to verify the computed parameters.

```go
// 0.1% false-positive probability for 2 million items
bfTuned := bloom.NewInt64Bloom(2_000_000, 0.001)

fmt.Printf("Bitmap size: %d bits, hash functions: %d\n",
	bfTuned.Size(), len(bfTuned.HashFuncs()))

```

You can also inspect the raw bitmap size through the embedded field (`bfTuned.bitmap.Size()`) if you need to monitor memory allocation metrics in your observability stack.

## Summary

- **Initialize** using `bloom.NewInt64Bloom(n, p)` to automatically calculate optimal bitmap size and hash count via `estimateParameters`.
- **Populate** using `Add()` for runtime inserts or `MAdd()` for batch loading large static datasets.
- **Query** using `Contains()` for O(1) membership tests that eliminate unnecessary database lookups when returning `false`.
- **Scale Safely** with the mutex-protected `bitmap.Bitmap` backing store, ensuring thread-safe concurrent access across game server goroutines.
- **Verify Positives** because a `true` result indicates only probabilistic presence—always confirm against your authoritative store before gameplay-critical operations.

## Frequently Asked Questions

### What is the memory overhead of fabrica-util's Bloom filter?

Memory consumption depends on the bitmap size `m`, calculated as approximately `-n·ln(p)/ln²(2)` bits. For 1 million IDs at 1% false-positive probability, the filter occupies roughly 12 MiB according to `BenchmarkInt64Bloom` in [`bloom/bloom_test.go`](https://github.com/go-pantheon/fabrica-util/blob/main/bloom/bloom_test.go). This is significantly smaller than storing the full int64 values in a map.

### Can I use fabrica-util's Bloom filter for string IDs instead of int64?

The current `Int64BloomFilter` implementation in [`bloom/bloom.go`](https://github.com/go-pantheon/fabrica-util/blob/main/bloom/bloom.go) is specialized for `int64` keys using `createInt64HashFunctions`. For string IDs, you would need to hash the strings to int64 values before insertion, or extend the implementation with additional hash function generators. The underlying `bitmap.Bitmap` remains agnostic to key type.

### How does the filter handle concurrent writes during gameplay?

The `bitmap.Bitmap` struct in [`bitmap/bitmap.go`](https://github.com/go-pantheon/fabrica-util/blob/main/bitmap/bitmap.go) protects all bit operations with a mutex, making `Add` and `Contains` calls safe for concurrent use. However, for extremely high-throughput scenarios with millions of operations per second, consider sharding filters by player ID ranges to reduce lock contention.

### What false-positive rate should I target for player ID lookups?

For player ID validation and matchmaking queues, a 1% (0.01) false-positive rate typically provides the best balance between memory savings and database query efficiency. For financial transactions or item grants where certainty is critical, target 0.1% (0.001) or lower, accepting the larger memory footprint (approximately 50% more bits per the `estimateParameters` calculations).