# How to Configure and Manage Different Data Sources in OpenDerisk

> Master OpenDerisk data source configuration and management. Learn how to use polymorphic dataclasses and the ConfigurationManager to easily integrate diverse data sources.

- Repository: [derisk-ai/openderisk](https://github.com/derisk-ai/openderisk)
- Tags: how-to-guide
- Published: 2026-02-28

---

**OpenDerisk configures data sources through polymorphic dataclasses that inherit from `DataSourceConfig`, parses TOML/JSON files via `ConfigurationManager`, and registers concrete connectors in the singleton `ConnectorManager` for retrieval by application services.**

OpenDerisk (derisk-ai/openderisk) abstracts every external storage system—whether MySQL, PostgreSQL, ClickHouse, MongoDB, or Elasticsearch—as a **datasource** that can be configured declaratively and consumed uniformly across the platform. Understanding how to configure and manage different data sources in OpenDerisk is essential for connecting the RAG service, knowledge bases, and agent systems to their underlying persistence layers.

## Understanding the DataSource Architecture in OpenDerisk

The architecture follows a layered registration pattern. At the core is the `ConfigurationManager` in [`packages/derisk-core/src/derisk/util/configure/manager.py`](https://github.com/derisk-ai/openderisk/blob/main/packages/derisk-core/src/derisk/util/configure/manager.py), which handles polymorphic instantiation through the `PolymorphicMeta` metaclass (lines 44-74). This metaclass automatically registers any subclass of `RegisterParameters` under a unique `type` field, enabling the system to resolve concrete implementations from configuration strings.

When the application starts, `ConfigurationManager.from_file` loads a TOML or JSON configuration, resolves environment-variable placeholders such as `${env:DB_HOST}` via `_resolve_env_vars` (lines 80-84), and converts dictionary sections into typed dataclass instances using `_convert_to_dataclass`. This method internally calls `_get_concrete_class` (lines 28-34, 40-48) to look up the appropriate subclass based on the `type` field.

Once instantiated, datasource configurations are paired with connector implementations (e.g., `MySQLConnector` in [`packages/derisk-ext/src/derisk_ext/datasource/rdbms/conn_mysql.py`](https://github.com/derisk-ai/openderisk/blob/main/packages/derisk-ext/src/derisk_ext/datasource/rdbms/conn_mysql.py)). The singleton `ConnectorManager` ([`packages/derisk-serve/src/derisk_serve/datasource/manages/connector_manager.py`](https://github.com/derisk-ai/openderisk/blob/main/packages/derisk-serve/src/derisk_serve/datasource/manages/connector_manager.py)) stores these connectors in a global registry, allowing services to retrieve active connections by name via `ConnectorManager.get_instance(app).get(name)` (lines 23-31).

## Step 1: Define a DataSource Configuration Schema

To add a new storage backend, create a dataclass inheriting from `DataSourceConfig` (a subclass of `RegisterParameters`). The `PolymorphicMeta` metaclass automatically registers the class under the value assigned to its `type` field.

```python

# my_datasource.py

from dataclasses import dataclass
from derisk.util.configure.manager import RegisterParameters

@dataclass
class MyCustomDSConfig(RegisterParameters):
    """Configuration schema for a fictional "mycustom" datasource."""
    type: str = "mycustom"                # ← type used in the config file

    host: str
    port: int = 1234
    user: str
    password: str

```

This registration mechanism is implemented in [`packages/derisk-core/src/derisk/util/configure/manager.py`](https://github.com/derisk-ai/openderisk/blob/main/packages/derisk-core/src/derisk/util/configure/manager.py) (lines 44-74).

## Step 2: Configure Data Sources in TOML or JSON

Create a configuration file where each datasource entry includes a `type` field matching the registered class name, along with connection parameters. The `ConfigurationManager` expands environment variables using the `${env:VAR}` syntax through `_resolve_env_vars` (lines 24-30).

```toml

# config.toml

[mydatasource]
type = "mycustom"
host = "${env:MYDS_HOST}"
port = 5678
user = "admin"
password = "${env:MYDS_PWD}"

```

## Step 3: Load and Parse Configuration with ConfigurationManager

Load the configuration file and convert sections into typed dataclass instances. The manager resolves the concrete class from the `type` field and validates the structure.

```python
from derisk.util.configure.manager import ConfigurationManager

# Load & parse the config

cfg = ConfigurationManager.from_file("config.toml")

# Convert the `mydatasource` section into a typed object

my_ds = cfg._convert_to_dataclass(MyCustomDSConfig, cfg.get("mydatasource"))

```

The `_convert_to_dataclass` method detects the `type` field and instantiates the correct subclass via `_get_concrete_class` as implemented in [`manager.py`](https://github.com/derisk-ai/openderisk/blob/main/manager.py) (lines 28-34, 40-48).

## Step 4: Register Connectors with ConnectorManager

Each datasource configuration requires a matching connector implementation that handles the actual I/O. Instantiate your connector and register it with the global `ConnectorManager` singleton.

```python
from derisk_serve.datasource.manages.connector_manager import ConnectorManager
from my_ext.datasource.conn_mycustom import MyCustomConnector

# Register a connector

ConnectorManager.get_instance(app=None).register("mydatasource", MyCustomConnector(my_ds))

```

The `ConnectorManager` singleton pattern is defined in [`packages/derisk-serve/src/derisk_serve/datasource/manages/connector_manager.py`](https://github.com/derisk-ai/openderisk/blob/main/packages/derisk-serve/src/derisk_serve/datasource/manages/connector_manager.py) (lines 23-31), ensuring connector instances are reused across the application lifecycle.

## Step 5: Consume Data Sources in Application Services

Services retrieve registered connectors by name and execute operations through the connector's API. For example, the knowledge service accesses datasources via the manager as shown in [`packages/derisk-serve/src/derisk_serve/datasource/manages/datasource_service.py`](https://github.com/derisk-ai/openderisk/blob/main/packages/derisk-serve/src/derisk_serve/datasource/manages/datasource_service.py) (lines 88-91).

```python
from derisk_serve.datasource.manages.connector_manager import ConnectorManager

def list_tables():
    # Retrieve the previously-registered connector

    conn = ConnectorManager.get_instance(app=None).get("mydatasource")
    # Use the connector's API (assume it follows BaseConnector)

    return conn.list_tables()

```

## Built-in DataSource Examples (MySQL, PostgreSQL, ClickHouse)

OpenDerisk includes predefined configuration classes for popular databases. The test suite in [`packages/derisk-core/src/derisk/util/tests/configure/test_manager.py`](https://github.com/derisk-ai/openderisk/blob/main/packages/derisk-core/src/derisk/util/tests/configure/test_manager.py) (lines 675-696) demonstrates the pattern for MySQL:

```python

# Excerpt from test_manager.py

from dataclasses import dataclass
from derisk.util.configure.manager import DataSourceConfig

@dataclass
class MySQLDataSource(DataSourceConfig):
    """MySQL DataSource Configuration"""
    driver: str = "mysql"
    host: str
    port: int = 3306
    user: str
    password: str
    database: str

```

Similar implementations exist for `PostgresDataSource`, `ClickHouseDataSource`, `MongoDBDataSource`, and `ElasticsearchDataSource`. Retrieve built-in connectors identically through `ConnectorManager`:

```python
from derisk_serve.datasource.manages.connector_manager import ConnectorManager

mysql_conn = ConnectorManager.get_instance(app=None).get("my_mysql")
schema = mysql_conn.get_schema()          # API exposed by MySQLConnector in conn_mysql.py

```

Connector implementations reside in `packages/derisk-ext/src/derisk_ext/datasource/rdbms/` (e.g., [`conn_mysql.py`](https://github.com/derisk-ai/openderisk/blob/main/conn_mysql.py), [`conn_sqlite.py`](https://github.com/derisk-ai/openderisk/blob/main/conn_sqlite.py)).

## Summary

- **Polymorphic Registration**: Define datasource schemas by subclassing `DataSourceConfig`; the `PolymorphicMeta` metaclass in [`manager.py`](https://github.com/derisk-ai/openderisk/blob/main/manager.py) handles automatic registration under the `type` field.
- **Configuration Loading**: Use `ConfigurationManager.from_file` to parse TOML/JSON, resolve environment variables via `_resolve_env_vars`, and instantiate typed configs via `_convert_to_dataclass`.
- **Global Registry**: Register concrete connectors with the `ConnectorManager` singleton ([`connector_manager.py`](https://github.com/derisk-ai/openderisk/blob/main/connector_manager.py)) for lifecycle management and reuse.
- **Service Consumption**: Retrieve active connectors by name using `ConnectorManager.get_instance(app).get(name)` to perform I/O operations in RAG, knowledge, or agent services.
- **Extensibility**: Implement new storage backends by creating a config dataclass and matching connector without modifying existing service code.

## Frequently Asked Questions

### How does OpenDerisk resolve environment variables in configuration files?

The `ConfigurationManager` expands placeholders like `${env:VAR_NAME}` through the `_resolve_env_vars` method (lines 80-84 in [`packages/derisk-core/src/derisk/util/configure/manager.py`](https://github.com/derisk-ai/openderisk/blob/main/packages/derisk-core/src/derisk/util/configure/manager.py)). This occurs during the initial parsing of TOML or JSON files, allowing sensitive credentials to be injected at runtime rather than stored in version control.

### What is the relationship between DataSourceConfig and RegisterParameters?

`DataSourceConfig` is a subclass of `RegisterParameters`, which uses the `PolymorphicMeta` metaclass (lines 44-74 in [`manager.py`](https://github.com/derisk-ai/openderisk/blob/main/manager.py)). When you define a dataclass inheriting from `DataSourceConfig`, the metaclass automatically registers it in a global registry under its `type` field value, enabling the configuration manager to instantiate the correct class dynamically based on the `type` string in the config file.

### Can I use multiple data sources simultaneously in OpenDerisk?

Yes. The `ConnectorManager` singleton maintains a mapping from datasource name to connector instance, allowing you to register and use multiple connectors concurrently. Each datasource entry in your TOML file creates a distinct configuration object, and you register each with a unique name via `ConnectorManager.get_instance(app).register(name, connector)`. Services retrieve specific connectors by name using the `get(name)` method.

### Where are the built-in database connectors implemented?

Built-in connectors for MySQL, PostgreSQL, SQLite, and other RDBMS systems are located in `packages/derisk-ext/src/derisk_ext/datasource/rdbms/`. For example, [`conn_mysql.py`](https://github.com/derisk-ai/openderisk/blob/main/conn_mysql.py) implements `MySQLConnector`, while [`conn_sqlite.py`](https://github.com/derisk-ai/openderisk/blob/main/conn_sqlite.py) provides local testing capabilities. These connectors are instantiated from configuration classes defined in the test suite and core configuration modules.