How to Configure and Manage Different Data Sources in OpenDerisk

OpenDerisk configures data sources through polymorphic dataclasses that inherit from DataSourceConfig, parses TOML/JSON files via ConfigurationManager, and registers concrete connectors in the singleton ConnectorManager for retrieval by application services.

OpenDerisk (derisk-ai/openderisk) abstracts every external storage system—whether MySQL, PostgreSQL, ClickHouse, MongoDB, or Elasticsearch—as a datasource that can be configured declaratively and consumed uniformly across the platform. Understanding how to configure and manage different data sources in OpenDerisk is essential for connecting the RAG service, knowledge bases, and agent systems to their underlying persistence layers.

Understanding the DataSource Architecture in OpenDerisk

The architecture follows a layered registration pattern. At the core is the ConfigurationManager in packages/derisk-core/src/derisk/util/configure/manager.py, which handles polymorphic instantiation through the PolymorphicMeta metaclass (lines 44-74). This metaclass automatically registers any subclass of RegisterParameters under a unique type field, enabling the system to resolve concrete implementations from configuration strings.

When the application starts, ConfigurationManager.from_file loads a TOML or JSON configuration, resolves environment-variable placeholders such as ${env:DB_HOST} via _resolve_env_vars (lines 80-84), and converts dictionary sections into typed dataclass instances using _convert_to_dataclass. This method internally calls _get_concrete_class (lines 28-34, 40-48) to look up the appropriate subclass based on the type field.

Once instantiated, datasource configurations are paired with connector implementations (e.g., MySQLConnector in packages/derisk-ext/src/derisk_ext/datasource/rdbms/conn_mysql.py). The singleton ConnectorManager (packages/derisk-serve/src/derisk_serve/datasource/manages/connector_manager.py) stores these connectors in a global registry, allowing services to retrieve active connections by name via ConnectorManager.get_instance(app).get(name) (lines 23-31).

Step 1: Define a DataSource Configuration Schema

To add a new storage backend, create a dataclass inheriting from DataSourceConfig (a subclass of RegisterParameters). The PolymorphicMeta metaclass automatically registers the class under the value assigned to its type field.

# my_datasource.py

from dataclasses import dataclass
from derisk.util.configure.manager import RegisterParameters

@dataclass
class MyCustomDSConfig(RegisterParameters):
    """Configuration schema for a fictional "mycustom" datasource."""
    type: str = "mycustom"                # ← type used in the config file

    host: str
    port: int = 1234
    user: str
    password: str

This registration mechanism is implemented in packages/derisk-core/src/derisk/util/configure/manager.py (lines 44-74).

Step 2: Configure Data Sources in TOML or JSON

Create a configuration file where each datasource entry includes a type field matching the registered class name, along with connection parameters. The ConfigurationManager expands environment variables using the ${env:VAR} syntax through _resolve_env_vars (lines 24-30).

# config.toml

[mydatasource]
type = "mycustom"
host = "${env:MYDS_HOST}"
port = 5678
user = "admin"
password = "${env:MYDS_PWD}"

Step 3: Load and Parse Configuration with ConfigurationManager

Load the configuration file and convert sections into typed dataclass instances. The manager resolves the concrete class from the type field and validates the structure.

from derisk.util.configure.manager import ConfigurationManager

# Load & parse the config

cfg = ConfigurationManager.from_file("config.toml")

# Convert the `mydatasource` section into a typed object

my_ds = cfg._convert_to_dataclass(MyCustomDSConfig, cfg.get("mydatasource"))

The _convert_to_dataclass method detects the type field and instantiates the correct subclass via _get_concrete_class as implemented in manager.py (lines 28-34, 40-48).

Step 4: Register Connectors with ConnectorManager

Each datasource configuration requires a matching connector implementation that handles the actual I/O. Instantiate your connector and register it with the global ConnectorManager singleton.

from derisk_serve.datasource.manages.connector_manager import ConnectorManager
from my_ext.datasource.conn_mycustom import MyCustomConnector

# Register a connector

ConnectorManager.get_instance(app=None).register("mydatasource", MyCustomConnector(my_ds))

The ConnectorManager singleton pattern is defined in packages/derisk-serve/src/derisk_serve/datasource/manages/connector_manager.py (lines 23-31), ensuring connector instances are reused across the application lifecycle.

Step 5: Consume Data Sources in Application Services

Services retrieve registered connectors by name and execute operations through the connector's API. For example, the knowledge service accesses datasources via the manager as shown in packages/derisk-serve/src/derisk_serve/datasource/manages/datasource_service.py (lines 88-91).

from derisk_serve.datasource.manages.connector_manager import ConnectorManager

def list_tables():
    # Retrieve the previously-registered connector

    conn = ConnectorManager.get_instance(app=None).get("mydatasource")
    # Use the connector's API (assume it follows BaseConnector)

    return conn.list_tables()

Built-in DataSource Examples (MySQL, PostgreSQL, ClickHouse)

OpenDerisk includes predefined configuration classes for popular databases. The test suite in packages/derisk-core/src/derisk/util/tests/configure/test_manager.py (lines 675-696) demonstrates the pattern for MySQL:

# Excerpt from test_manager.py

from dataclasses import dataclass
from derisk.util.configure.manager import DataSourceConfig

@dataclass
class MySQLDataSource(DataSourceConfig):
    """MySQL DataSource Configuration"""
    driver: str = "mysql"
    host: str
    port: int = 3306
    user: str
    password: str
    database: str

Similar implementations exist for PostgresDataSource, ClickHouseDataSource, MongoDBDataSource, and ElasticsearchDataSource. Retrieve built-in connectors identically through ConnectorManager:

from derisk_serve.datasource.manages.connector_manager import ConnectorManager

mysql_conn = ConnectorManager.get_instance(app=None).get("my_mysql")
schema = mysql_conn.get_schema()          # API exposed by MySQLConnector in conn_mysql.py

Connector implementations reside in packages/derisk-ext/src/derisk_ext/datasource/rdbms/ (e.g., conn_mysql.py, conn_sqlite.py).

Summary

• Polymorphic Registration: Define datasource schemas by subclassing DataSourceConfig; the PolymorphicMeta metaclass in manager.py handles automatic registration under the type field.
• Configuration Loading: Use ConfigurationManager.from_file to parse TOML/JSON, resolve environment variables via _resolve_env_vars, and instantiate typed configs via _convert_to_dataclass.
• Global Registry: Register concrete connectors with the ConnectorManager singleton (connector_manager.py) for lifecycle management and reuse.
• Service Consumption: Retrieve active connectors by name using ConnectorManager.get_instance(app).get(name) to perform I/O operations in RAG, knowledge, or agent services.
• Extensibility: Implement new storage backends by creating a config dataclass and matching connector without modifying existing service code.

Frequently Asked Questions

How does OpenDerisk resolve environment variables in configuration files?

The ConfigurationManager expands placeholders like ${env:VAR_NAME} through the _resolve_env_vars method (lines 80-84 in packages/derisk-core/src/derisk/util/configure/manager.py). This occurs during the initial parsing of TOML or JSON files, allowing sensitive credentials to be injected at runtime rather than stored in version control.

What is the relationship between DataSourceConfig and RegisterParameters?

DataSourceConfig is a subclass of RegisterParameters, which uses the PolymorphicMeta metaclass (lines 44-74 in manager.py). When you define a dataclass inheriting from DataSourceConfig, the metaclass automatically registers it in a global registry under its type field value, enabling the configuration manager to instantiate the correct class dynamically based on the type string in the config file.

Can I use multiple data sources simultaneously in OpenDerisk?

Yes. The ConnectorManager singleton maintains a mapping from datasource name to connector instance, allowing you to register and use multiple connectors concurrently. Each datasource entry in your TOML file creates a distinct configuration object, and you register each with a unique name via ConnectorManager.get_instance(app).register(name, connector). Services retrieve specific connectors by name using the get(name) method.

Where are the built-in database connectors implemented?

Built-in connectors for MySQL, PostgreSQL, SQLite, and other RDBMS systems are located in packages/derisk-ext/src/derisk_ext/datasource/rdbms/. For example, conn_mysql.py implements MySQLConnector, while conn_sqlite.py provides local testing capabilities. These connectors are instantiated from configuration classes defined in the test suite and core configuration modules.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}