Building Multi-Document Agents with LlamaIndex and Claude: A Complete Implementation Guide

You can build powerful multi-document agents by integrating LlamaIndex's retrieval infrastructure with Claude's tool-use capabilities, creating autonomous systems that iteratively search across document collections and synthesize answers through a ReAct reasoning loop.

Building multi-document agents with LlamaIndex and Claude enables AI applications to perform complex reasoning across heterogeneous document collections. The anthropics/claude-cookbooks repository demonstrates this pattern through runnable notebooks that combine LlamaIndex's vector indexing with Claude's ability to invoke tools dynamically. This architecture separates the retrieval layer—which handles document chunking and semantic search—from the reasoning layer, where Claude decides which documents to retrieve and how to synthesize the final response.

Architectural Overview

The multi-document agent pattern implemented in the claude-cookbooks repository follows a modular architecture that scales from prototype to production. Understanding these components ensures you build systems that can handle thousands of pages while maintaining accurate source attribution.

Document Ingestion and Vector Storage

The foundation begins with document ingestion using LlamaIndex's SimpleDirectoryReader. This utility loads heterogeneous file types—PDFs, markdown, text files—into a unified document collection. According to the source implementation in third_party/LlamaIndex/Multi_Document_Agents.ipynb, these documents are processed into nodes and stored in a VectorStoreIndex, which enables semantic similarity search across the corpus.

For large-scale deployments, the architecture supports specialized retrievers like MultiVectorRetriever for handling multiple vector representations per document, or RouterQueryEngine for routing queries to specific sub-indexes based on content type.

Claude Integration via Tool-Use API

Claude serves as the reasoning engine through a dedicated wrapper class. The integration uses ClaudeLLM (imported as from llama_index.llms.anthropic import Claude) configured with tool-use capabilities. This setup exposes document search as a callable tool that Claude can invoke during its reasoning process.

The ServiceContext object binds Claude to the index, ensuring all retrieval operations return context that Claude can reference when generating responses.

ReAct Agent Loop

The agent follows the ReAct pattern (Reasoning + Acting) implemented as an outer loop:

1. Thought: Claude analyzes the user query and current context
2. Action: Claude decides whether to call search_documents with a refined sub-query
3. Observation: The system retrieves relevant chunks from the VectorStoreIndex
4. Answer: Claude synthesizes the final response once sufficient information is gathered

This loop terminates when Claude produces a final answer rather than a tool call, or when reaching the maximum turn limit defined in the agent configuration.

Implementation Walkthrough

The following implementation mirrors the production code found in third_party/LlamaIndex/Multi_Document_Agents.ipynb, demonstrating how to wire these components together into a functional agent.

Step 1: Configure the Environment

First, install the required dependencies and set up the document directory:

# Required packages

# pip install llama-index llama-index-llms-anthropic anthropic

import os
os.environ["ANTHROPIC_API_KEY"] = "your-api-key-here"

Step 2: Initialize the Document Index

Load your documents and configure Claude as the underlying LLM:

from llama_index import SimpleDirectoryReader, VectorStoreIndex, ServiceContext
from llama_index.llms.anthropic import Claude

# Load documents from data directory

documents = SimpleDirectoryReader("data/").load_data()

# Configure Claude with tool-use capabilities

claude_llm = Claude(model="claude-3-5-sonnet-20240620")
service_ctx = ServiceContext.from_defaults(llm=claude_llm)

# Build vector index with Claude as the embedding context

index = VectorStoreIndex.from_documents(
    documents, 
    service_context=service_ctx
)

# Create retriever for top-k semantic search

retriever = index.as_retriever(similarity_top_k=5)

Step 3: Implement the ReAct Agent

Construct the agent loop that allows Claude to iteratively refine its search:

def claude_agent(query: str, max_turns: int = 5):
    """
    ReAct-style agent that uses Claude for reasoning and 
    LlamaIndex for document retrieval.
    """
    conversation_history = []
    current_query = query
    
    for turn in range(max_turns):
        # Construct the prompt with tool instructions

        prompt = f"""You are a research assistant with access to the following tool:
- `search_documents(query)`: Returns the most relevant excerpts from the document corpus.

Question: {current_query}

If you need more information to answer accurately, call `search_documents` with a specific sub-query.
Otherwise, provide a comprehensive final answer based on the information available.

Your response:"""

        # Get Claude's response

        response = claude_llm.complete(prompt)
        response_text = str(response)
        
        # Check for tool invocation (simplified parsing)

        if "search_documents(" in response_text:
            # Extract the sub-query from tool call

            start_idx = response_text.find("search_documents(") + len("search_documents(")
            end_idx = response_text.find(")", start_idx)
            sub_query = response_text[start_idx:end_idx].strip('"\'')
            
            # Retrieve relevant documents

            retrieved_docs = retriever.retrieve(sub_query)
            context = "\n".join([doc.text for doc in retrieved_docs])
            
            # Update query with retrieved context for next iteration

            current_query = f"""Original question: {query}
            
            Retrieved context from '{sub_query}':
            {context}
            
            Based on this information, answer the original question."""
            
            conversation_history.append({"turn": turn, "action": "search", "query": sub_query})
        else:
            # Final answer achieved

            return response_text
            
    return "Maximum turns reached. Partial answer: " + response_text

# Execute the agent

answer = claude_agent("What are the primary challenges of LLMs in medical literature review?")
print(answer)

Production Considerations

The snippet above uses simplified string parsing for demonstration. In production implementations—as shown in the full notebook—replace the naive parsing with Claude's official tool-use JSON format, which provides structured tool definitions and reliable argument extraction.

For enhanced performance, consider implementing:

• Memory persistence: Store conversation history using Claude's memory tool to maintain context across sessions
• Query routing: Use RouterQueryEngine to direct questions to specialized sub-indexes (e.g., separating technical documentation from user manuals)
• Hybrid search: Combine vector similarity with keyword matching for better coverage of technical terminology

Key Resources in the Repository

The anthropics/claude-cookbooks repository contains several reference implementations that demonstrate different aspects of this architecture:

• third_party/LlamaIndex/Multi_Document_Agents.ipynb: The primary reference showing end-to-end multi-document agent construction with ReAct reasoning
• third_party/LlamaIndex/ReAct_Agent.ipynb: Focuses specifically on the reasoning-action loop implementation patterns
• third_party/LlamaIndex/Router_Query_Engine.ipynb: Demonstrates query routing strategies for large document collections with heterogeneous content
• third_party/LlamaIndex/Basic_RAG_With_LlamaIndex.ipynb: Introduces the foundational Retrieval-Augmented Generation pattern before adding agentic capabilities

These files are located in the third_party/LlamaIndex/ directory and require the llama-index and anthropic Python packages.

Summary

Building multi-document agents with LlamaIndex and Claude creates systems capable of autonomous research across large text corpora:

• LlamaIndex handles the heavy lifting of document parsing, chunking, and vector storage through SimpleDirectoryReader and VectorStoreIndex
• Claude provides the reasoning layer via tool-use capabilities, deciding when to search and how to synthesize information
• The ReAct loop enables iterative refinement, allowing the agent to drill down into specific documents when initial retrieval is insufficient
• Source files in anthropics/claude-cookbooks provide production-ready templates for implementing this pattern in research assistants and knowledge-base applications

This architecture scales from simple Q&A over a few PDFs to complex investigative agents querying thousands of heterogeneous documents.

Frequently Asked Questions

How does a multi-document agent differ from standard RAG?

Standard Retrieval-Augmented Generation performs a single retrieval pass before generating an answer, which limits accuracy when the initial query doesn't match the document phrasing exactly. A multi-document agent implements an iterative ReAct loop where Claude can reformulate queries and perform multiple retrieval passes until gathering sufficient evidence. As implemented in the claude-cookbooks examples, this allows the system to handle vague initial questions by progressively narrowing the search scope based on intermediate findings.

What document formats are supported for ingestion?

The SimpleDirectoryReader class supports PDFs, Markdown, plain text, and common office formats automatically. According to the source implementation, you can extend this with custom loaders by subclassing LlamaIndex's base reader classes. For production deployments, preprocess documents to handle scanned PDFs (requiring OCR) and ensure consistent encoding for non-English text files before ingestion.

How do I optimize retrieval accuracy across thousands of documents?

For large document collections, implement query routing using the RouterQueryEngine demonstrated in Router_Query_Engine.ipynb. This component classifies incoming questions and routes them to specialized sub-indexes (e.g., legal documents vs. technical specifications). Additionally, configure the similarity_top_k parameter in your retriever to balance between context window limits and coverage—typically 5-10 chunks provides sufficient context without exceeding Claude's input token limits.

Can this architecture maintain context across multiple user sessions?

Yes, by implementing the memory tool pattern shown in the advanced sections of Multi_Document_Agents.ipynb. Rather than re-indexing documents for each conversation, store key insights and user preferences in a persistent memory store that Claude references at the start of new sessions. This approach maintains the agent's understanding of document relationships without repeating expensive embedding operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}