How to Build Content Moderation Filters with Claude: A Complete Implementation Guide

You can build a production-ready content moderation filter using Claude by encoding moderation policies as plain-text guidelines in prompts and constraining responses to single-token outputs, enabling policy updates without code changes.

The anthropics/claude-cookbooks repository provides a complete reference implementation for content moderation filters using Claude. The approach demonstrated in misc/building_moderation_filter.ipynb leverages prompt engineering to convert the LLM into a binary classifier, allowing you to moderate text against custom guidelines without training custom models or deploying additional infrastructure.

Architecture of a Claude-Based Moderation System

Prompt-Driven Rule Definition

Moderation categories are encoded directly in the prompt as plain-text guidelines. According to the source code in lines 15-28 of misc/building_moderation_filter.ipynb, BLOCK and ALLOW categories are defined using simple descriptive lists that specify prohibited and permitted content types. This approach makes the classification logic entirely transparent and editable.

Dynamic Prompt Templating

The moderate_text helper function (lines 66-78) constructs the final prompt by injecting user-generated text and the chosen guidelines into a multi-line string template. This separation keeps the classification logic flexible while maintaining a consistent prompt structure across different moderation policies.

Single-Token Response Optimization

To ensure deterministic outputs, the implementation constrains Claude's response to a maximum of 10 tokens (max_tokens=10) as shown in lines 82-88. This forces the model to return only "ALLOW" or "BLOCK", eliminating the need for complex output parsing or regex extraction in the standard use case.

Basic Implementation

The following helper function from the cookbook demonstrates the core moderation pattern:

from anthropic import Anthropic

client = Anthropic()
MODEL_NAME = "claude-haiku-4-5"

def moderate_text(user_text: str, guidelines: str) -> str:
    """Return 'ALLOW' or 'BLOCK' based on the supplied guidelines."""
    prompt_template = """
You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines:

{guidelines}

Here is the user-generated text to categorize:
<user_text>{user_text}</user_text>

Based on the guidelines above, classify this text as either ALLOW or BLOCK. Return nothing else.
"""
    prompt = prompt_template.format(user_text=user_text, guidelines=guidelines)
    response = (
        client.messages.create(
            model=MODEL_NAME,
            max_tokens=10,
            messages=[{"role": "user", "content": prompt}],
        )
        .content[0]
        .text
    )
    return response.strip()

You can then apply custom guidelines to content streams:

example_guidelines = """\
BLOCK CATEGORY:
- Promoting violence, illegal activities, or hate speech
- Explicit sexual content
- Harmful misinformation or conspiracy theories

ALLOW CATEGORY:
- All other content that does not violate the above rules
"""

comments = [
    "I love this new feature!",
    "Delete this post now or you'll regret it.",
]

for c in comments:
    print(f"Comment: {c}\nClassification: {moderate_text(c, example_guidelines)}\n")

Advanced Prompting Techniques for Better Accuracy

Chain-of-Thought Reasoning

For borderline cases that require nuanced judgment, the notebook demonstrates Chain-of-Thought (CoT) prompting in lines 40-53. By instructing Claude to reason within <thinking> tags before emitting the final classification inside <output> tags, you improve consistency on ambiguous content while maintaining a parseable output format.

cot_prompt = """\
You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines:

BLOCK CATEGORY:
- ... (same as above)

ALLOW CATEGORY:
- ... (same as above)

First, inside <thinking> tags, reason about any concerning aspects of the post. Then, inside <output> tags, output ALLOW or BLOCK. Return nothing else.

<user_post>{user_post}</user_post>
"""

def moderate_with_cot(user_post: str, guidelines: str) -> str:
    prompt = cot_prompt.format(user_post=user_post, guidelines=guidelines)
    resp = client.messages.create(
        model=MODEL_NAME,
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    ).content[0].text
    # Extract whatever appears between <output> tags

    import re
    m = re.search(r"<output>(.*?)</output>", resp, re.DOTALL)
    return m.group(1).strip() if m else resp.strip()

Few-Shot Examples

Lines 9-24 in the "Improving Performance with Examples" cell show how adding representative BLOCK/ALLOW examples within an <examples> block significantly improves accuracy for domain-specific content without requiring parameter updates.

examples_prompt = """\
You are a content moderation expert ...

BLOCK CATEGORY:
- ...

ALLOW CATEGORY:
- ...

Here are some examples:
<examples>
Text: I'm selling weight loss products, check my link to buy!
Category: BLOCK

Text: Did anyone ride the new RMC raptor trek yet?
Category: ALLOW
</examples>

<user_text>{user_text}</user_text>
"""

def moderate_with_examples(user_text: str, guidelines: str) -> str:
    prompt = examples_prompt.format(user_text=user_text, guidelines=guidelines)
    resp = client.messages.create(
        model=MODEL_NAME,
        max_tokens=10,
        messages=[{"role": "user", "content": prompt}],
    ).content[0].text
    return resp.strip()

Processing Content at Scale

The notebook demonstrates batch moderation patterns suitable for production workloads. Lines 42-45 iterate through comment lists, while lines 6-9 process roller coaster-related titles, calling moderate_text for each entry and collecting classifications. This pattern scales to any list-based data source by wrapping the API call in a simple loop or using Python's map() function for concurrent processing.

Summary

• Prompt-based rules allow policy updates without code changes—simply edit the guideline text passed to the moderate_text function.
• Single-token responses (max_tokens=10) ensure clean ALLOW/BLOCK outputs requiring minimal post-processing.
• Chain-of-Thought reasoning improves accuracy on edge cases by forcing explicit deliberation before classification.
• Few-shot examples align the model with domain-specific nuances without requiring model fine-tuning or retraining.
• Batch processing patterns in misc/building_moderation_filter.ipynb demonstrate production-ready integration for high-volume content streams.

Frequently Asked Questions

How do I update moderation policies without redeploying code?

Because the classification logic lives entirely in the prompt text within misc/building_moderation_filter.ipynb, you can update moderation policies by editing the plain-text guidelines or examples passed as the guidelines parameter. New policies take effect immediately on the next API call without requiring code changes, model retraining, or service restarts.

What is the optimal max_tokens value for moderation responses?

According to the source implementation in lines 82-88, setting max_tokens=10 is sufficient for binary classification tasks. This limit constrains Claude to return only "ALLOW" or "BLOCK" (plus potential whitespace or punctuation), preventing verbose explanations while allowing minor formatting variations that can be standardized with .strip().

Can I implement multi-class moderation beyond ALLOW and BLOCK?

Yes. While the notebook demonstrates binary classification, you can extend the approach by modifying the prompt template in lines 66-78 to request additional categories such as "REVIEW_REQUIRED" or "SPAM". Update the response constraints in the guidelines and adjust max_tokens to accommodate longer category names (e.g., 20-30 tokens for multi-word classifications).

How does Chain-of-Thought prompting affect latency and cost?

Chain-of-Thought reasoning requires higher max_tokens allocations (around 200 tokens) to accommodate the <thinking> section before the <output> tag, as shown in the CoT implementation. This increases both token consumption and response latency compared to the single-token approach. Reserve CoT for content that fails basic moderation or requires nuanced interpretation, and use the standard max_tokens=10 approach for clear-cut cases to optimize costs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}