Building RAG Pipelines Using the rag-architect Skill: From Document Chunking to Production Deployment

The rag-architect skill in alirezarezvani/claude-skills provides a Python standard library-based toolkit that designs complete Retrieval-Augmented Generation pipelines through three integrated components: a Chunking Optimizer, Pipeline Designer, and Retrieval Evaluator.

The alirezarezvani/claude-skills repository hosts the rag-architect skill, a POWERFUL-tier engineering package that eliminates architectural guesswork when building RAG systems. Unlike framework-specific tools that lock you into specific vector databases or embedding providers, this skill generates portable, cost-optimized pipeline designs using only JSON inputs and Python's built-in libraries.

Core Components of the rag-architect Skill

The skill encapsulates three tightly-coupled modules located in engineering/rag-architect/. Together, these components handle the full lifecycle from document ingestion to retrieval validation.

Chunking Optimizer (chunking_optimizer.py)

The Chunking Optimizer analyzes document characteristics to recommend optimal segmentation strategies. Implemented in engineering/rag-architect/chunking_optimizer.py, it selects from sentence-based, paragraph-based, semantic-heading-aware, or adaptive chunking approaches based on document type and size constraints. The module includes built-in cost estimation logic that projects processing expenses before implementation.

Pipeline Designer (rag_pipeline_designer.py)

The Pipeline Designer serves as the central orchestration engine. Located at engineering/rag-architect/rag_pipeline_designer.py, this CLI tool accepts JSON-encoded system requirements and outputs complete architectural specifications. The designer maps your constraints—latency requirements, monthly budget, accuracy priorities—to specific component selections including embedding models, vector databases, and retrieval methods.

Retrieval Evaluator (retrieval_evaluator.py)

The Retrieval Evaluator validates pipeline performance against classical information retrieval baselines. Found in engineering/rag-architect/retrieval_evaluator.py, this module constructs a TF-IDF retriever, computes precision@k, recall@k, MRR, and NDCG metrics, and analyzes failure patterns to generate actionable improvement recommendations.

Architectural Data Flow

The rag-architect skill implements a structured flow that separates document processing from query handling and evaluation:

1. Document ingestion flows through the Chunking Optimizer to determine segmentation strategy
2. Embedding generation routes to the selected vector database (Pinecone for large-scale production, Chroma for prototypes)
3. Query processing supports dense, sparse, or hybrid retrieval with optional cross-encoder reranking
4. Evaluation loop compares retrieval results against ground truth using the TF-IDF baseline in retrieval_evaluator.py

The architecture specifically handles trade-offs between latency and accuracy by conditionally adding reranking layers only when accuracy priorities outweigh budget constraints.

Step-by-Step Pipeline Design

Defining System Requirements

Create a JSON configuration file that encodes your operational constraints. The Pipeline Designer expects fields defining document volume, query patterns, latency requirements, and budget ceilings.

{
  "document_types": ["technical", "code"],
  "document_count": 2500000,
  "avg_document_size": 3500,
  "queries_per_day": 8000,
  "query_patterns": ["factual", "analytical"],
  "latency_requirement": "interactive",
  "budget_monthly": 1200,
  "accuracy_priority": 0.85,
  "cost_priority": 0.3,
  "maintenance_complexity": "medium"
}

Running the Pipeline Designer

Execute the designer module as a Python CLI tool, passing your requirements file and output destination:

python -m engineering.rag-architect.rag_pipeline_designer \
    requirements.json \
    -o pipeline_design.json \
    -v

The -v (verbose) flag triggers human-readable output including a cost breakdown, component rationale, and a Mermaid diagram suitable for documentation. The tool writes the complete machine-readable specification to pipeline_design.json.

Interpreting Generated Output

The designer outputs component selections with monthly cost projections. For technical documentation at scale, it typically recommends:

• Chunking Strategy: adaptive_chunking for mixed document types
• Embedding Model: openai-text-embedding-ada-002 when accuracy exceeds 0.8 priority
• Vector Database: Pinecone for managed high-performance workloads exceeding 1M documents
• Retrieval Method: Hybrid search combining dense and sparse signals

The generated Mermaid diagram in the output maps the complete data flow from document corpus through chunking, embedding generation, vector storage, query processing, and optional reranking to final response generation.

Evaluating Retrieval Performance

Executing the TF-IDF Baseline

Validate your corpus quality before investing in vector infrastructure using the standalone evaluator. The tool requires three inputs: a queries file, corpus directory, and ground truth judgments.

python -m engineering.rag-architect.retrieval_evaluator \
    queries.json \
    ./corpus \
    ground_truth.json \
    -o eval_results.json \
    -v

Understanding Metrics and Recommendations

The evaluator in engineering/rag-architect/retrieval_evaluator.py computes standard IR metrics:

• Precision@k and Recall@k for threshold-based performance
• MRR (Mean Reciprocal Rank) for evaluating result ordering quality
• NDCG (Normalized Discounted Cumulative Gain) for graded relevance scenarios

The analysis identifies specific failure modes such as zero-result queries, vocabulary mismatches, and length-based retrieval bias. The module emits prioritized recommendations including query expansion implementation, reranking addition, or chunking strategy adjustments.

Customization Without External Dependencies

Because the rag-architect skill relies exclusively on Python's standard library, you can modify component logic without managing complex dependency trees. Swap the TF-IDF baseline in retrieval_evaluator.py for a BM25 implementation, or extend chunking_optimizer.py with domain-specific heuristics while maintaining the surrounding workflow intact. The modular structure separates strategy selection from execution, allowing granular customization of cost estimation formulas or retrieval metrics.

Summary

• The rag-architect skill provides a complete RAG design toolkit using only Python standard library components
• rag_pipeline_designer.py converts JSON requirements into architecture diagrams with cost estimates and component specifications
• chunking_optimizer.py determines optimal document segmentation strategies based on content type and scale
• retrieval_evaluator.py establishes TF-IDF baselines and computes precision, recall, MRR, and NDCG metrics with automated improvement recommendations
• All modules function without external dependencies, enabling immediate execution in any Python environment

Frequently Asked Questions

What input format does the Pipeline Designer require?

The Pipeline Designer accepts JSON files containing operational parameters including document count, average document size, queries per day, latency requirements, budget constraints, and priority weightings for accuracy versus cost. The schema is validated internally by engineering/rag-architect/rag_pipeline_designer.py before processing.

How does the Chunking Optimizer select between sentence and semantic heading strategies?

The optimizer in engineering/rag-architect/chunking_optimizer.py evaluates document type metadata passed in the requirements JSON. Technical documentation and code repositories trigger semantic-heading-aware chunking to preserve logical boundaries, while narrative content defaults to sentence or paragraph chunking based on average document length and query pattern complexity.

Can the Retrieval Evaluator analyze existing vector database deployments?

Yes. While the retrieval_evaluator.py module includes a built-in TF-IDF baseline for initial validation, you can feed retrieval results from any external vector database into the evaluation framework by formatting them as the expected JSON query-result pairs. The metrics calculation and failure analysis remain agnostic to the underlying retrieval mechanism.

What metrics indicate I should add a reranker to my pipeline?

According to the analysis logic in retrieval_evaluator.py, consistently low precision@k scores combined with high recall@k indicate that relevant documents are being retrieved but ranked poorly. The evaluator specifically flags these patterns with recommendations to implement cross-encoder reranking, particularly when your requirements JSON specifies accuracy_priority values above 0.8 and latency requirements permit the additional compute overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}