# Building RAG Applications with URL Knowledge Bases and LanceDB: A Complete Implementation Guide

> Build RAG applications with URL knowledge bases using LanceDB and GPT-5. This guide details a full implementation with Agno framework and Streamlit for your AI projects.

- Repository: [Arindam Majumder /awesome-ai-apps](https://github.com/Arindam200/awesome-ai-apps)
- Tags: how-to-guide
- Published: 2026-05-06

---

**You can build a production-ready Retrieval-Augmented Generation (RAG) system that crawls web URLs, stores vector embeddings in LanceDB, and answers questions using GPT-5, with all orchestration handled by the Agno framework and a Streamlit frontend.**

This guide examines the `agentic_rag` implementation in the `Arindam200/awesome-ai-apps` repository, demonstrating exactly how to construct RAG applications with URL knowledge bases and LanceDB. The working example in [`rag_apps/agentic_rag/main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/rag_apps/agentic_rag/main.py) provides a complete blueprint for ingesting web content into a local vector database and querying it through an agentic interface.

## Architecture Overview

The system follows a straightforward pipeline: **Web URLs → Text Extraction → Vector Embeddings → LanceDB Storage → Semantic Retrieval → LLM Generation**. 

When a user submits URLs through the Streamlit interface, the `UrlKnowledge` component fetches the raw HTML, converts it to plain text, and processes it through `OpenAIEmbedder` using the `text-embedding-3-small` model. These embeddings are persisted in LanceDB at `tmp/lancedb` under the table `mcp-docs-knowledge-base`. Upon querying, the Agno Agent performs vector search against this store, injects the top-k relevant chunks into the prompt, and streams the response from `OpenAIChat` (configured for GPT-5) back to the user.

## Core Components Deep Dive

### UrlKnowledge for Web Ingestion

The **UrlKnowledge** class handles the initial data acquisition. Implemented in the `load_knowledge_base()` function at lines 41-48 of [`main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/main.py), this component accepts a list of URLs, downloads the page content, and extracts clean text for the embedding pipeline.

```python
from rag_apps.agentic_rag.main import load_knowledge_base

urls = [
    "https://developer.mozilla.org/en-US/docs/Web/JavaScript",
    "https://fastapi.tiangolo.com/"
]

# Downloads content, generates embeddings, stores in LanceDB

knowledge_base = load_knowledge_base(urls)

```

### LanceDB as Local Vector Storage

**LanceDB** provides the vector persistence layer using a single-file, high-performance columnar format ideal for local development. According to the source code at lines 43-48, the configuration specifies:

- **URI**: `tmp/lancedb` for local disk storage
- **Table Name**: `mcp-docs-knowledge-base` 
- **Search Type**: Vector similarity search for fast retrieval

Because LanceDB stores data on disk at `tmp/lancedb`, the knowledge base persists across application sessions until explicitly cleared via the "🔄 Reset KB" UI button.

### OpenAIEmbedder and OpenAIChat

The embedding pipeline uses **OpenAIEmbedder** to generate dense vectors from the downloaded web content. The LLM interface, configured at lines 60-62, initializes **OpenAIChat** with the model ID `gpt-5-2025-08-07` (or alternatively GPT-4o), handling the final response generation with retrieved context.

### Agno Agent Orchestration

The **Agno Agent** serves as the central orchestrator, initialized with `knowledge=knowledge_base` and `search_knowledge=True`. This configuration enables automatic retrieval-augmented generation: the agent queries LanceDB using vector search, prepends the relevant chunks to the system prompt, and manages the streaming response to the Streamlit frontend.

## Step-by-Step Implementation

### Setting Up the Knowledge Base

The `load_knowledge_base()` function encapsulates the ingestion logic. It instantiates `UrlKnowledge` with the provided URLs, configures the `OpenAIEmbedder`, and initializes the LanceDB vector store:

```python
from agno.knowledge.url import UrlKnowledge
from agno.vectordb.lancedb import LanceDb
from agno.embedder.openai import OpenAIEmbedder

def load_knowledge_base(urls):
    # Lines 41-48 in main.py

    knowledge_base = UrlKnowledge(
        urls=urls,
        vector_db=LanceDb(
            table_name="mcp-docs-knowledge-base",
            uri="tmp/lancedb",
            search_type=SearchType.vector,
            embedder=OpenAIEmbedder(),
        ),
    )
    knowledge_base.load()
    return knowledge_base

```

### Configuring the Vector Store Parameters

The LanceDB configuration at lines 43-48 specifies critical parameters:

- **`uri="tmp/lancedb"`**: Defines the local storage path for the vector database files
- **`table_name="mcp-docs-knowledge-base"`**: Creates a named collection for the embeddings
- **`embedder=OpenAIEmbedder()`**: Assigns the embedding model (defaults to `text-embedding-3-small`)

### Building the RAG Agent

The agent initialization at lines 60-66 wires together the components:

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat

agent = Agent(
    model=OpenAIChat(id="gpt-5-2025-08-07"),
    knowledge=knowledge_base,
    search_knowledge=True,
    show_tool_calls=True,
    markdown=True,
)

```

Setting `search_knowledge=True` enables automatic vector retrieval from LanceDB before each LLM call.

### Creating the Streamlit Interface

The UI implementation spans lines 92-151, providing:

- **Sidebar inputs**: Text fields for URL entry with dynamic addition/removal
- **Load button**: Triggers `load_knowledge_base()` and displays success confirmations
- **Chat interface**: `st.chat_input` for queries and streaming response display
- **Reset functionality**: Clears the `tmp/lancedb` directory to start fresh

## Running the Application

Clone the repository and install dependencies using the provided [`pyproject.toml`](https://github.com/Arindam200/awesome-ai-apps/blob/main/pyproject.toml) (which specifies `lancedb>=0.24.2`):

```bash
git clone https://github.com/Arindam200/awesome-ai-apps.git
cd rag_apps/agentic_rag

# Install dependencies

uv sync

# Or: pip install -r requirements.txt

# Configure environment

cp .env.example .env

# Edit .env to add your OpenAI API key

# Launch the application

uv run streamlit run main.py

```

Once running, add URLs in the sidebar, click **"Load Knowledge Base"**, then submit queries in the chat interface. The system retrieves relevant passages from the indexed web pages via LanceDB and generates contextual answers using the configured LLM.

## Extending the System

You can modify the implementation in [`rag_apps/agentic_rag/main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/rag_apps/agentic_rag/main.py) to support alternative configurations:

- **Local Embeddings**: Replace `OpenAIEmbedder` with `SentenceTransformer` or HuggingFace embedders for offline operation
- **Different LLMs**: Change the `OpenAIChat` model ID to `gpt-4o` or configure a self-hosted endpoint via the Agno framework
- **Cloud Persistence**: Replace the local `uri` with a cloud storage path (e.g., `s3://my-bucket/lancedb`) for shared knowledge bases across instances
- **Observability**: Uncomment the Arize Phoenix integration at lines 20-30 to enable tracing and metrics collection

## Summary

- **UrlKnowledge** fetches and processes web content from supplied URLs automatically
- **LanceDB** provides a high-performance, file-based vector store at `tmp/lancedb` that persists across sessions
- **OpenAIEmbedder** generates `text-embedding-3-small` vectors for semantic similarity search
- The **Agno Agent** orchestrates retrieval and generation with `search_knowledge=True`, streaming responses through a **Streamlit** interface
- The complete working implementation resides in [`rag_apps/agentic_rag/main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/rag_apps/agentic_rag/main.py) with dependencies managed via [`pyproject.toml`](https://github.com/Arindam200/awesome-ai-apps/blob/main/pyproject.toml)

## Frequently Asked Questions

### How does the URL knowledge base handle dynamic web content?

The `UrlKnowledge` component fetches content at load time when `load_knowledge_base()` is called. It does not automatically refresh; to update content from dynamic URLs, you must click the **"Load Knowledge Base"** button again in the Streamlit UI, which re-downloads the pages and regenerates embeddings stored in LanceDB.

### Can I use a different vector database instead of LanceDB?

Yes, though the current implementation in [`main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/main.py) specifically configures LanceDB at lines 43-48. You can swap the `LanceDb` class for any vector database supported by the Agno framework (such as ChromaDB, Pinecone, or Weaviate) by changing the `vector_db` parameter in the `UrlKnowledge` initialization while maintaining the same `load_knowledge_base()` interface.

### What embedding models are supported besides OpenAI?

The repository uses `OpenAIEmbedder` by default, but Agno supports multiple embedding providers. You can replace `OpenAIEmbedder()` with `HuggingFaceEmbedder` or `OllamaEmbedder` in the LanceDB configuration to use local models like SentenceTransformers or Llama-based embedders, eliminating the dependency on OpenAI API keys.

### How do I deploy this RAG application to production?

For production deployment, containerize the application using Docker, mount a persistent volume for `tmp/lancedb` to retain the vector store across container restarts, and replace the local LanceDB URI with a cloud-backed storage solution (such as S3 or GCS) if running multiple instances. Ensure `OPENAI_API_KEY` is set via environment variables or a secrets manager rather than `.env` files.