How DB-GPT's Storage Layer Handles Metadata and Vector Database Integration

DB-GPT separates relational metadata from vector embeddings through a dual-layer architecture that uses SQLAlchemy for structured data and a pluggable abstraction for similarity search, enabling seamless integration with multiple vector databases while maintaining referential integrity.

The eosphoros-ai/DB-GPT repository implements a sophisticated storage subsystem that cleanly partitions knowledge-space definitions and document metadata from high-dimensional embedding vectors. This architecture allows the RAG pipeline to leverage ACID-compliant relational databases for metadata while utilizing specialized vector stores like Chroma, Milvus, or PgVector for similarity search operations.

Metadata Layer Architecture

The metadata subsystem resides in dbgpt.storage.metadata and provides a unified interface for relational data management across the application.

DatabaseManager and Session Handling

The DatabaseManager class in packages/dbgpt-core/src/dbgpt/storage/metadata/db_manager.py serves as the central singleton for SQLAlchemy operations. It instantiates and holds the Engine, Session, and declarative base, exposing a context manager (db.session()) that automatically handles transaction commits and rollbacks.

from dbgpt.storage.metadata import db, initialize_db, Model

# Initialize a SQLite DB (any SQLAlchemy URL works)

initialize_db("sqlite:///./my_metadata.db", db_name="dbgpt")

# Define a custom table

class KnowledgeSpace(Model):
    __tablename__ = "knowledge_space"
    id = Column(Integer, primary_key=True)
    name = Column(String(100), unique=True)

# Create tables and insert a record

db.create_all()
with db.session() as sess:
    ks = KnowledgeSpace(name="my_space")
    sess.add(ks)

BaseModel and CRUD Operations

The BaseModel and BaseCRUDMixin classes provide foundational database functionality. These base classes expose helper methods including create, update, delete, and list that operate through the global db manager. The create_model function generates model classes bound to specific DatabaseManager instances, enabling multiple database connections within the same process.

All metadata—including knowledge-space definitions, document-chunk records, and graph schema definitions—resides in this relational store.

Vector Store Abstraction

The vector storage layer in dbgpt.storage.vector_store provides a backend-agnostic interface for embedding operations.

VectorStoreBase Interface

The VectorStoreBase abstract class in packages/dbgpt-core/src/dbgpt/storage/vector_store/base.py defines the contract for all vector database implementations. It handles common logic such as score-threshold filtering, L2-normalisation, and asynchronous document loading. Concrete implementations—including PgVectorStore, ChromaStore, and MilvusStore—reside in the dbgpt_ext package and must implement methods like load_document, search, and truncate.

Configuration and Discovery

The VectorStoreConfig class provides a Pydantic-style configuration object that stores connection parameters, pooling settings, and embedding function references. It exposes the max_chunks_once_load and max_threads limits for controlling ingestion throughput. Concrete vector stores implement the create_store factory method and expose a static __type__ identifier used for runtime discovery:

from dbgpt.storage.vector_store.base import VectorStoreConfig

def get_supported_vector_types() -> List[str]:
    # Walk the subclass tree of VectorStoreConfig

    return [cls.__type__ for cls in VectorStoreConfig.__subclasses__()]

The StorageManager utilizes this registry to validate user-requested vector_store_type values against available backends.

StorageManager Orchestration

The StorageManager class in packages/dbgpt-serve/src/dbgpt_serve/rag/storage_manager.py functions as the primary integration point between metadata and vector storage layers. It performs four critical functions:

1. Configuration Parsing: Reads the global application configuration (app_config.rag.storage) to obtain vector store settings
2. Singleton Caching: Maintains a _store_cache dictionary mapping index names to singleton vector store instances
3. Embedding Injection: Retrieves the embedding function from EmbeddingFactory to ensure consistent vectorization across all stored data
4. Backend Abstraction: Handles knowledge-graph and full-text search backends through similar factory patterns

The vector store instantiation follows this pattern:

def create_vector_store(self, index_name) -> VectorStoreBase:
    storage_config = self.system_app.config.configs["app_config"].rag.storage
    embedding_fn = self.system_app.get_component(
        "embedding_factory", EmbeddingFactory
    ).create()
    vector_store_cfg: VectorStoreConfig = storage_config.vector
    # Concrete store is built via the config's factory method

    new_store = vector_store_cfg.create_store(
        name=index_name,
        embedding_fn=embedding_fn,
        max_chunks_once_load=vector_store_cfg.max_chunks_once_load,
        max_threads=vector_store_cfg.max_threads,
    )
    self._store_cache[index_name] = new_store
    return new_store

Bridging Metadata and Vectors

The two storage layers maintain referential integrity through identifier linking. Each document chunk stored in the relational metadata layer contains a vector_ids column that stores comma-separated identifiers referencing vectors in the external database.

End-to-End Data Flow

The following workflow demonstrates the integration between both layers:

from dbgpt.storage.metadata import initialize_db, Model, db
from dbgpt.storage.vector_store.base import VectorStoreConfig
from dbgpt.core import Chunk, Embeddings

# 1. Initialise metadata DB

initialize_db("sqlite:///./metadata.db", db_name="dbgpt")

class DocumentChunk(Model):
    __tablename__ = "doc_chunk"
    id = Column(Integer, primary_key=True)
    content = Column(Text)
    vector_ids = Column(Text)      # comma-separated ids from vector store

db.create_all()

# 2. Create a vector store implementation

class MyPgVectorConfig(VectorStoreConfig):
    __type__ = "PgVector"
    def create_store(self, **kwargs):
        from dbgpt_ext.storage.vector_store.pgvector import PgVectorStore
        return PgVectorStore(**kwargs)

my_cfg = MyPgVectorConfig(name="my_vectors")
vector_store = my_cfg.create_store(
    name="my_space",
    embedding_fn=Embeddings(),   # embedding model instance

)

# 3. Insert a document chunk

chunk = Chunk(content="DB-GPT is an LLM-powered RAG platform.")
ids = vector_store.load_document([chunk])  # returns list of vector IDs

with db.session() as sess:
    db_chunk = DocumentChunk(
        content=chunk.content,
        vector_ids=",".join(ids)
    )
    sess.add(db_chunk)

During retrieval operations, the RAG pipeline executes a three-step process: first retrieving the vector store via StorageManager.get_storage_connector, performing similarity search to obtain vector IDs, then joining against the DocumentChunk metadata table using the stored vector_ids values, optionally applying additional MetadataFilters for precision filtering.

Summary

• Dual-layer architecture: DB-GPT strictly separates relational metadata (SQLAlchemy/SQLite/PostgreSQL) from vector embeddings (Chroma/Milvus/PgVector) to optimize for different access patterns.
• Centralized metadata management: The DatabaseManager singleton in packages/dbgpt-core/src/dbgpt/storage/metadata/db_manager.py provides ACID transactions and CRUD abstractions for knowledge-space definitions and document records.
• Pluggable vector stores: The VectorStoreBase abstraction enables runtime discovery and integration of multiple vector databases through the VectorStoreConfig factory pattern.
• Referential integrity: The vector_ids column in metadata tables maintains the relationship between relational records and external vector embeddings, enabling efficient hybrid queries.
• Orchestration layer: StorageManager coordinates both layers, handling embedding injection, connection pooling, and singleton caching per index name.

Frequently Asked Questions

How does DB-GPT handle metadata storage?

DB-GPT handles metadata storage through the dbgpt.storage.metadata module, which utilizes SQLAlchemy for ORM operations. The DatabaseManager class manages database connections and sessions, while BaseModel provides CRUD operations for tables storing knowledge-space definitions, document chunks, and graph schemas. This layer supports any SQLAlchemy-compatible database including SQLite and PostgreSQL.

What vector databases does DB-GPT support?

DB-GPT supports multiple vector databases through its pluggable VectorStoreBase interface, including Chroma, Milvus, and PgVector. The system discovers available implementations by inspecting subclasses of VectorStoreConfig and their __type__ identifiers. New backends can be added by implementing the abstract methods load_document, search, and truncate in the dbgpt_ext package.

How are metadata and vector stores linked in DB-GPT?

Metadata and vector stores are linked through identifier references stored in the relational database. Each document chunk record in the metadata layer contains a vector_ids column storing comma-separated identifiers generated by the vector store during ingestion. During retrieval, the system queries the vector database for similar embeddings, then uses these IDs to fetch the corresponding metadata records, enabling reconstruction of the original document content with associated metadata.

Where is the storage configuration defined in DB-GPT?

Storage configuration is defined in the global application configuration under app_config.rag.storage, specifically within the vector section mapped to VectorStoreConfig. The StorageManager reads these settings to instantiate vector stores with appropriate connection parameters, embedding functions, and performance limits such as max_chunks_once_load and max_threads. Configuration is typically provided through TOML files and loaded into the SystemApp context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}