# How DB-GPT's Storage Layer Handles Metadata and Vector Database Integration

> Discover how DB-GPTs dual-layer architecture separates metadata and vector embeddings using SQLAlchemy and pluggable similarity search for seamless vector database integration.

- Repository: [eosphoros/DB-GPT](https://github.com/eosphoros-ai/db-gpt)
- Tags: internals
- Published: 2026-02-23

---

**DB-GPT separates relational metadata from vector embeddings through a dual-layer architecture that uses SQLAlchemy for structured data and a pluggable abstraction for similarity search, enabling seamless integration with multiple vector databases while maintaining referential integrity.**

The eosphoros-ai/DB-GPT repository implements a sophisticated storage subsystem that cleanly partitions knowledge-space definitions and document metadata from high-dimensional embedding vectors. This architecture allows the RAG pipeline to leverage ACID-compliant relational databases for metadata while utilizing specialized vector stores like Chroma, Milvus, or PgVector for similarity search operations.

## Metadata Layer Architecture

The metadata subsystem resides in `dbgpt.storage.metadata` and provides a unified interface for relational data management across the application.

### DatabaseManager and Session Handling

The `DatabaseManager` class in [`packages/dbgpt-core/src/dbgpt/storage/metadata/db_manager.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/storage/metadata/db_manager.py) serves as the central singleton for SQLAlchemy operations. It instantiates and holds the `Engine`, `Session`, and declarative base, exposing a context manager (`db.session()`) that automatically handles transaction commits and rollbacks.

```python
from dbgpt.storage.metadata import db, initialize_db, Model

# Initialize a SQLite DB (any SQLAlchemy URL works)

initialize_db("sqlite:///./my_metadata.db", db_name="dbgpt")

# Define a custom table

class KnowledgeSpace(Model):
    __tablename__ = "knowledge_space"
    id = Column(Integer, primary_key=True)
    name = Column(String(100), unique=True)

# Create tables and insert a record

db.create_all()
with db.session() as sess:
    ks = KnowledgeSpace(name="my_space")
    sess.add(ks)

```

### BaseModel and CRUD Operations

The `BaseModel` and `BaseCRUDMixin` classes provide foundational database functionality. These base classes expose helper methods including `create`, `update`, `delete`, and `list` that operate through the global `db` manager. The `create_model` function generates model classes bound to specific `DatabaseManager` instances, enabling multiple database connections within the same process.

All metadata—including knowledge-space definitions, document-chunk records, and graph schema definitions—resides in this relational store.

## Vector Store Abstraction

The vector storage layer in `dbgpt.storage.vector_store` provides a backend-agnostic interface for embedding operations.

### VectorStoreBase Interface

The `VectorStoreBase` abstract class in [`packages/dbgpt-core/src/dbgpt/storage/vector_store/base.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/storage/vector_store/base.py) defines the contract for all vector database implementations. It handles common logic such as score-threshold filtering, L2-normalisation, and asynchronous document loading. Concrete implementations—including `PgVectorStore`, `ChromaStore`, and `MilvusStore`—reside in the `dbgpt_ext` package and must implement methods like `load_document`, `search`, and `truncate`.

### Configuration and Discovery

The `VectorStoreConfig` class provides a Pydantic-style configuration object that stores connection parameters, pooling settings, and embedding function references. It exposes the `max_chunks_once_load` and `max_threads` limits for controlling ingestion throughput. Concrete vector stores implement the `create_store` factory method and expose a static `__type__` identifier used for runtime discovery:

```python
from dbgpt.storage.vector_store.base import VectorStoreConfig

def get_supported_vector_types() -> List[str]:
    # Walk the subclass tree of VectorStoreConfig

    return [cls.__type__ for cls in VectorStoreConfig.__subclasses__()]

```

The `StorageManager` utilizes this registry to validate user-requested `vector_store_type` values against available backends.

## StorageManager Orchestration

The `StorageManager` class in [`packages/dbgpt-serve/src/dbgpt_serve/rag/storage_manager.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-serve/src/dbgpt_serve/rag/storage_manager.py) functions as the primary integration point between metadata and vector storage layers. It performs four critical functions:

1. **Configuration Parsing**: Reads the global application configuration (`app_config.rag.storage`) to obtain vector store settings
2. **Singleton Caching**: Maintains a `_store_cache` dictionary mapping index names to singleton vector store instances
3. **Embedding Injection**: Retrieves the embedding function from `EmbeddingFactory` to ensure consistent vectorization across all stored data
4. **Backend Abstraction**: Handles knowledge-graph and full-text search backends through similar factory patterns

The vector store instantiation follows this pattern:

```python
def create_vector_store(self, index_name) -> VectorStoreBase:
    storage_config = self.system_app.config.configs["app_config"].rag.storage
    embedding_fn = self.system_app.get_component(
        "embedding_factory", EmbeddingFactory
    ).create()
    vector_store_cfg: VectorStoreConfig = storage_config.vector
    # Concrete store is built via the config's factory method

    new_store = vector_store_cfg.create_store(
        name=index_name,
        embedding_fn=embedding_fn,
        max_chunks_once_load=vector_store_cfg.max_chunks_once_load,
        max_threads=vector_store_cfg.max_threads,
    )
    self._store_cache[index_name] = new_store
    return new_store

```

## Bridging Metadata and Vectors

The two storage layers maintain referential integrity through identifier linking. Each document chunk stored in the relational metadata layer contains a `vector_ids` column that stores comma-separated identifiers referencing vectors in the external database.

### End-to-End Data Flow

The following workflow demonstrates the integration between both layers:

```python
from dbgpt.storage.metadata import initialize_db, Model, db
from dbgpt.storage.vector_store.base import VectorStoreConfig
from dbgpt.core import Chunk, Embeddings

# 1. Initialise metadata DB

initialize_db("sqlite:///./metadata.db", db_name="dbgpt")

class DocumentChunk(Model):
    __tablename__ = "doc_chunk"
    id = Column(Integer, primary_key=True)
    content = Column(Text)
    vector_ids = Column(Text)      # comma-separated ids from vector store

db.create_all()

# 2. Create a vector store implementation

class MyPgVectorConfig(VectorStoreConfig):
    __type__ = "PgVector"
    def create_store(self, **kwargs):
        from dbgpt_ext.storage.vector_store.pgvector import PgVectorStore
        return PgVectorStore(**kwargs)

my_cfg = MyPgVectorConfig(name="my_vectors")
vector_store = my_cfg.create_store(
    name="my_space",
    embedding_fn=Embeddings(),   # embedding model instance

)

# 3. Insert a document chunk

chunk = Chunk(content="DB-GPT is an LLM-powered RAG platform.")
ids = vector_store.load_document([chunk])  # returns list of vector IDs

with db.session() as sess:
    db_chunk = DocumentChunk(
        content=chunk.content,
        vector_ids=",".join(ids)
    )
    sess.add(db_chunk)

```

During retrieval operations, the RAG pipeline executes a three-step process: first retrieving the vector store via `StorageManager.get_storage_connector`, performing similarity search to obtain vector IDs, then joining against the `DocumentChunk` metadata table using the stored `vector_ids` values, optionally applying additional `MetadataFilters` for precision filtering.

## Summary

- **Dual-layer architecture**: DB-GPT strictly separates relational metadata (SQLAlchemy/SQLite/PostgreSQL) from vector embeddings (Chroma/Milvus/PgVector) to optimize for different access patterns.
- **Centralized metadata management**: The `DatabaseManager` singleton in [`packages/dbgpt-core/src/dbgpt/storage/metadata/db_manager.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/storage/metadata/db_manager.py) provides ACID transactions and CRUD abstractions for knowledge-space definitions and document records.
- **Pluggable vector stores**: The `VectorStoreBase` abstraction enables runtime discovery and integration of multiple vector databases through the `VectorStoreConfig` factory pattern.
- **Referential integrity**: The `vector_ids` column in metadata tables maintains the relationship between relational records and external vector embeddings, enabling efficient hybrid queries.
- **Orchestration layer**: `StorageManager` coordinates both layers, handling embedding injection, connection pooling, and singleton caching per index name.

## Frequently Asked Questions

### How does DB-GPT handle metadata storage?

DB-GPT handles metadata storage through the `dbgpt.storage.metadata` module, which utilizes SQLAlchemy for ORM operations. The `DatabaseManager` class manages database connections and sessions, while `BaseModel` provides CRUD operations for tables storing knowledge-space definitions, document chunks, and graph schemas. This layer supports any SQLAlchemy-compatible database including SQLite and PostgreSQL.

### What vector databases does DB-GPT support?

DB-GPT supports multiple vector databases through its pluggable `VectorStoreBase` interface, including Chroma, Milvus, and PgVector. The system discovers available implementations by inspecting subclasses of `VectorStoreConfig` and their `__type__` identifiers. New backends can be added by implementing the abstract methods `load_document`, `search`, and `truncate` in the `dbgpt_ext` package.

### How are metadata and vector stores linked in DB-GPT?

Metadata and vector stores are linked through identifier references stored in the relational database. Each document chunk record in the metadata layer contains a `vector_ids` column storing comma-separated identifiers generated by the vector store during ingestion. During retrieval, the system queries the vector database for similar embeddings, then uses these IDs to fetch the corresponding metadata records, enabling reconstruction of the original document content with associated metadata.

### Where is the storage configuration defined in DB-GPT?

Storage configuration is defined in the global application configuration under `app_config.rag.storage`, specifically within the `vector` section mapped to `VectorStoreConfig`. The `StorageManager` reads these settings to instantiate vector stores with appropriate connection parameters, embedding functions, and performance limits such as `max_chunks_once_load` and `max_threads`. Configuration is typically provided through TOML files and loaded into the `SystemApp` context.