# How ai-memory Enforces Hard Content Limits for Different Observation Types

> Discover how ai-memory enforces hard content limits for observations. Learn about byte, count, and token thresholds to manage your data effectively.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-08-22

---

**ai-memory applies compile-time constant limits to every observation type—truncating, rejecting, or throttling data that exceeds predefined byte, count, or token thresholds.**

The `akitaonrails/ai-memory` repository implements a single-writer SQLite-backed memory system for AI agents. To prevent storage bloat and ensure predictable runtime behavior, the codebase defines **hard content limits for different types of observations**—ranging from work-stream transcripts to wiki admission webhooks—using `pub const` values enforced at ingestion time.

## Work-Stream Transcript Limits

The work-stream crate processes raw telemetry events and file metadata. It guards against unbounded growth using three specific caps defined in [`crates/ai-memory-workstream/src/transcript.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-workstream/src/transcript.rs).

### Truncating Oversized Event Payloads

Raw event content is truncated to **128 KiB** via the `MAX_EVENT_BYTES` constant. The `truncate_utf8` utility ensures valid UTF-8 boundaries are respected.

```rust
// crates/ai-memory-workstream/src/transcript.rs
pub const MAX_EVENT_BYTES: usize = 128 * 1024;

// During ingestion
let content = truncate_utf8(content, MAX_EVENT_BYTES);

```

Any payload exceeding this limit is silently truncated before storage, preventing individual events from consuming excessive disk or memory.

### Limiting File Scan Scope

When scanning directories for transcript files, the system stops after processing **50,000 files**. This is enforced using `Iterator::take` with `MAX_SCAN_FILES`.

```rust
pub const MAX_SCAN_FILES: usize = 50_000;

// In the scanning loop
for bucket in buckets.take(MAX_SCAN_FILES) {
    // process file
}

```

Additionally, native session identifiers attached to work-streams cannot exceed **512 bytes**, enforced by `MAX_NATIVE_SESSION_ID_BYTES`.

## Wiki Admission and Webhook Constraints

The wiki crate ([`crates/ai-memory-wiki/src/admission.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/admission.rs)) governs external admission webhooks that validate or mutate page writes. It imposes hard limits on chain length, response size, timeout duration, and concurrency.

### Capping the Admission Chain Length

An admission chain cannot contain more than **16 webhooks**. Exceeding this returns a hard error during chain construction.

```rust
pub const MAX_ADMISSION_WEBHOOKS: usize = 16;

if webhooks.len() > MAX_ADMISSION_WEBHOOKS {
    return Err(anyhow!(
        "admission chain capped at {MAX_ADMISSION_WEBHOOKS} webhooks, got {}",
        webhooks.len()
    ));
}

```

### Bounding Webhook Response Sizes

Webhook HTTP responses are capped at **1 MiB** (`MAX_RESPONSE_BYTES`). The streaming reader accumulates chunks until the limit is reached, then drops subsequent data.

```rust
pub const MAX_RESPONSE_BYTES: usize = 1024 * 1024;

if bytes.len().saturating_add(chunk.len()) > MAX_RESPONSE_BYTES {
    // Treat as no-op and warn
    return Ok(None);
}

```

Timeouts are similarly bounded to **30,000 milliseconds** (`MAX_WEBHOOK_TIMEOUT_MS`), with any larger values clamped to this maximum.

### Throttling Concurrent Operations

To protect the runtime from admission stampede, the system limits concurrent async admission calls to **256** using a `tokio::sync::Semaphore`.

```rust
pub const MAX_ASYNC_ADMISSION_IN_FLIGHT: usize = 256;

static SEMAPHORE: Semaphore = Semaphore::const_new(MAX_ASYNC_ADMISSION_IN_FLIGHT);

```

Additional calls wait on the semaphore, providing backpressure rather than rejecting requests.

## LLM Consolidation Token Limits

Beyond byte-level limits, the LLM consolidation pipeline enforces token-based boundaries documented in [`docs/install.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/install.md) and applied during prompt construction.

- **Input limit**: `AI_MEMORY_CONSOLIDATION__MAX_INPUT_TOKENS = 6,500`
- **Output limit**: `AI_MEMORY_CONSOLIDATION__MAX_OUTPUT_TOKENS = 1,000`

These limits are checked before calling the LLM provider; requests exceeding them are rejected to prevent API errors and excessive costs.

## Summary

- **Work-stream events** are truncated to 128 KiB using `MAX_EVENT_BYTES` in [`transcript.rs`](https://github.com/akitaonrails/ai-memory/blob/main/transcript.rs).
- **File scanning** halts at 50,000 files via `MAX_SCAN_FILES`.
- **Wiki admission chains** are hard-capped at 16 webhooks by `MAX_ADMISSION_WEBHOOKS`.
- **Webhook responses** cannot exceed 1 MiB (`MAX_RESPONSE_BYTES`), and timeouts are limited to 30 seconds.
- **Concurrent admission** is throttled to 256 in-flight requests.
- **LLM prompts** are gated by token counts (6,500 input / 1,000 output) before reaching the provider.

## Frequently Asked Questions

### What happens when a work-stream event exceeds 128 KiB?

The payload is truncated to the first 128 KiB of valid UTF-8 content. The truncation occurs in [`crates/ai-memory-workstream/src/transcript.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-workstream/src/transcript.rs) using the `truncate_utf8` helper, ensuring the observation is stored without invalid byte sequences.

### Why does ai-memory limit wiki admission chains to 16 webhooks?

The `MAX_ADMISSION_WEBHOOKS` constant prevents combinatorial latency explosion and potential circular validation loops. According to the source in [`crates/ai-memory-wiki/src/admission.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/admission.rs), attempting to register more than 16 hooks returns an immediate error during chain initialization.

### How does ai-memory handle webhook responses larger than 1 MiB?

When the accumulated response bytes reach `MAX_RESPONSE_BYTES` (1 MiB), the receiver stops buffering and treats the remainder as a no-op, logging a warning. This prevents unbounded memory growth when external webhooks return unexpectedly large payloads.

### Are the content limits configurable at runtime?

No. All limits referenced here are `pub const` values defined at compile time in their respective crates (e.g., `ai-memory-workstream`, `ai-memory-wiki`). This design provides deterministic resource guarantees for the single-writer SQLite store and adjacent runtime components.