Building RAG Applications with URL Knowledge Bases and LanceDB: A Complete Implementation Guide

You can build a production-ready Retrieval-Augmented Generation (RAG) system that crawls web URLs, stores vector embeddings in LanceDB, and answers questions using GPT-5, with all orchestration handled by the Agno framework and a Streamlit frontend.

This guide examines the agentic_rag implementation in the Arindam200/awesome-ai-apps repository, demonstrating exactly how to construct RAG applications with URL knowledge bases and LanceDB. The working example in rag_apps/agentic_rag/main.py provides a complete blueprint for ingesting web content into a local vector database and querying it through an agentic interface.

Architecture Overview

The system follows a straightforward pipeline: Web URLs → Text Extraction → Vector Embeddings → LanceDB Storage → Semantic Retrieval → LLM Generation.

When a user submits URLs through the Streamlit interface, the UrlKnowledge component fetches the raw HTML, converts it to plain text, and processes it through OpenAIEmbedder using the text-embedding-3-small model. These embeddings are persisted in LanceDB at tmp/lancedb under the table mcp-docs-knowledge-base. Upon querying, the Agno Agent performs vector search against this store, injects the top-k relevant chunks into the prompt, and streams the response from OpenAIChat (configured for GPT-5) back to the user.

Core Components Deep Dive

UrlKnowledge for Web Ingestion

The UrlKnowledge class handles the initial data acquisition. Implemented in the load_knowledge_base() function at lines 41-48 of main.py, this component accepts a list of URLs, downloads the page content, and extracts clean text for the embedding pipeline.

from rag_apps.agentic_rag.main import load_knowledge_base

urls = [
    "https://developer.mozilla.org/en-US/docs/Web/JavaScript",
    "https://fastapi.tiangolo.com/"
]

# Downloads content, generates embeddings, stores in LanceDB

knowledge_base = load_knowledge_base(urls)

LanceDB as Local Vector Storage

LanceDB provides the vector persistence layer using a single-file, high-performance columnar format ideal for local development. According to the source code at lines 43-48, the configuration specifies:

• URI: tmp/lancedb for local disk storage
• Table Name: mcp-docs-knowledge-base
• Search Type: Vector similarity search for fast retrieval

Because LanceDB stores data on disk at tmp/lancedb, the knowledge base persists across application sessions until explicitly cleared via the "🔄 Reset KB" UI button.

OpenAIEmbedder and OpenAIChat

The embedding pipeline uses OpenAIEmbedder to generate dense vectors from the downloaded web content. The LLM interface, configured at lines 60-62, initializes OpenAIChat with the model ID gpt-5-2025-08-07 (or alternatively GPT-4o), handling the final response generation with retrieved context.

Agno Agent Orchestration

The Agno Agent serves as the central orchestrator, initialized with knowledge=knowledge_base and search_knowledge=True. This configuration enables automatic retrieval-augmented generation: the agent queries LanceDB using vector search, prepends the relevant chunks to the system prompt, and manages the streaming response to the Streamlit frontend.

Step-by-Step Implementation

Setting Up the Knowledge Base

The load_knowledge_base() function encapsulates the ingestion logic. It instantiates UrlKnowledge with the provided URLs, configures the OpenAIEmbedder, and initializes the LanceDB vector store:

from agno.knowledge.url import UrlKnowledge
from agno.vectordb.lancedb import LanceDb
from agno.embedder.openai import OpenAIEmbedder

def load_knowledge_base(urls):
    # Lines 41-48 in main.py

    knowledge_base = UrlKnowledge(
        urls=urls,
        vector_db=LanceDb(
            table_name="mcp-docs-knowledge-base",
            uri="tmp/lancedb",
            search_type=SearchType.vector,
            embedder=OpenAIEmbedder(),
        ),
    )
    knowledge_base.load()
    return knowledge_base

Configuring the Vector Store Parameters

The LanceDB configuration at lines 43-48 specifies critical parameters:

• uri="tmp/lancedb": Defines the local storage path for the vector database files
• table_name="mcp-docs-knowledge-base": Creates a named collection for the embeddings
• embedder=OpenAIEmbedder(): Assigns the embedding model (defaults to text-embedding-3-small)

Building the RAG Agent

The agent initialization at lines 60-66 wires together the components:

from agno.agent import Agent
from agno.models.openai import OpenAIChat

agent = Agent(
    model=OpenAIChat(id="gpt-5-2025-08-07"),
    knowledge=knowledge_base,
    search_knowledge=True,
    show_tool_calls=True,
    markdown=True,
)

Setting search_knowledge=True enables automatic vector retrieval from LanceDB before each LLM call.

Creating the Streamlit Interface

The UI implementation spans lines 92-151, providing:

• Sidebar inputs: Text fields for URL entry with dynamic addition/removal
• Load button: Triggers load_knowledge_base() and displays success confirmations
• Chat interface: st.chat_input for queries and streaming response display
• Reset functionality: Clears the tmp/lancedb directory to start fresh

Running the Application

Clone the repository and install dependencies using the provided pyproject.toml (which specifies lancedb>=0.24.2):

git clone https://github.com/Arindam200/awesome-ai-apps.git
cd rag_apps/agentic_rag

# Install dependencies

uv sync

# Or: pip install -r requirements.txt

# Configure environment

cp .env.example .env

# Edit .env to add your OpenAI API key

# Launch the application

uv run streamlit run main.py

Once running, add URLs in the sidebar, click "Load Knowledge Base", then submit queries in the chat interface. The system retrieves relevant passages from the indexed web pages via LanceDB and generates contextual answers using the configured LLM.

Extending the System

You can modify the implementation in rag_apps/agentic_rag/main.py to support alternative configurations:

• Local Embeddings: Replace OpenAIEmbedder with SentenceTransformer or HuggingFace embedders for offline operation
• Different LLMs: Change the OpenAIChat model ID to gpt-4o or configure a self-hosted endpoint via the Agno framework
• Cloud Persistence: Replace the local uri with a cloud storage path (e.g., s3://my-bucket/lancedb) for shared knowledge bases across instances
• Observability: Uncomment the Arize Phoenix integration at lines 20-30 to enable tracing and metrics collection

Summary

• UrlKnowledge fetches and processes web content from supplied URLs automatically
• LanceDB provides a high-performance, file-based vector store at tmp/lancedb that persists across sessions
• OpenAIEmbedder generates text-embedding-3-small vectors for semantic similarity search
• The Agno Agent orchestrates retrieval and generation with search_knowledge=True, streaming responses through a Streamlit interface
• The complete working implementation resides in rag_apps/agentic_rag/main.py with dependencies managed via pyproject.toml

Frequently Asked Questions

How does the URL knowledge base handle dynamic web content?

The UrlKnowledge component fetches content at load time when load_knowledge_base() is called. It does not automatically refresh; to update content from dynamic URLs, you must click the "Load Knowledge Base" button again in the Streamlit UI, which re-downloads the pages and regenerates embeddings stored in LanceDB.

Can I use a different vector database instead of LanceDB?

Yes, though the current implementation in main.py specifically configures LanceDB at lines 43-48. You can swap the LanceDb class for any vector database supported by the Agno framework (such as ChromaDB, Pinecone, or Weaviate) by changing the vector_db parameter in the UrlKnowledge initialization while maintaining the same load_knowledge_base() interface.

What embedding models are supported besides OpenAI?

The repository uses OpenAIEmbedder by default, but Agno supports multiple embedding providers. You can replace OpenAIEmbedder() with HuggingFaceEmbedder or OllamaEmbedder in the LanceDB configuration to use local models like SentenceTransformers or Llama-based embedders, eliminating the dependency on OpenAI API keys.

How do I deploy this RAG application to production?

For production deployment, containerize the application using Docker, mount a persistent volume for tmp/lancedb to retain the vector store across container restarts, and replace the local LanceDB URI with a cloud-backed storage solution (such as S3 or GCS) if running multiple instances. Ensure OPENAI_API_KEY is set via environment variables or a secrets manager rather than .env files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}