# How the DynamoDB Configuration System Merges Default Configurations with Custom Settings in AWS IDP

> Discover how the DynamoDB configuration system merges default and custom settings in AWS IDP. Learn about the ConfigurationManager and deep-update utility for seamless integration.

- Repository: [aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws)
- Tags: internals
- Published: 2026-02-25

---

**The DynamoDB configuration system in the Accelerated Intelligent Document Processing (IDP) solution uses a `ConfigurationManager` class to recursively merge default configurations with custom settings using a deep-update utility, automatically detecting and migrating legacy sparse formats to full-configuration records.**

The `aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws` repository implements a sophisticated runtime configuration system that stores all processing parameters in DynamoDB. Understanding how this **DynamoDB configuration system** combines baseline defaults with environment-specific overrides is essential for customizing the IDP pipeline without breaking core functionality.

## Understanding the ConfigurationManager Architecture

At the heart of the system lies the **`ConfigurationManager`** class defined in [`lib/idp_common_pkg/idp_common/config/configuration_manager.py`](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/blob/main/lib/idp_common_pkg/idp_common/config/configuration_manager.py). This class acts as the central authority for reading, merging, and persisting configuration states. Rather than requiring operators to define complete configuration documents for every deployment, the system maintains **default configurations** as the baseline and layers **custom settings** on top through a safe, recursive merge process.

The architecture supports two distinct merge workflows: one for the full IDP processing configuration (OCR, classification, extraction) and another specifically for pricing parameters.

## How Default and Custom Configurations Are Merged

The system implements separate merge strategies depending on the configuration type, both utilizing the `deep_update` utility to ensure nested dictionaries are merged rather than replaced.

### Merging Full IDP Configurations

The primary merge logic resides in `ConfigurationManager.get_merged_configuration` (lines 200-229). This method handles the **DynamoDB configuration system**'s core responsibility: resolving a requested version into a complete, valid `IDPConfig` object.

The process follows four distinct steps:

1. **Full-format detection**: The method first checks if the requested version exists as a **full-format** configuration using `_is_full_config` (lines 51-72). This checks for the explicit marker `_config_format: "full"` or validates that enough top-level sections exist to consider the record complete.

2. **Legacy delta handling**: If the item is a legacy **sparse delta** (containing only changed fields), the system loads the **default version** (`DEFAULT_VERSION`) and retrieves the raw delta via `get_raw_configuration`.

3. **Deep merge execution**: The default model is converted to a plain dictionary using `model_dump`, and the delta is **deep-merged** into it using `deep_update` (defined in [`lib/idp_common_pkg/idp_common/config/merge_utils.py`](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/blob/main/lib/idp_common_pkg/idp_common/config/merge_utils.py), lines 55-71). This utility recursively overwrites nested keys while preserving untouched sections.

4. **Validation and auto-migration**: The merged dictionary is validated back into an `IDPConfig` object. The system then **auto-migrates** the result to full format by calling `save_configuration` (lines 81-85), writing the merged result back to DynamoDB with the full-config marker so subsequent reads skip the merge step.

### Merging Pricing Configurations

For pricing-specific overrides, the system uses `ConfigurationManager.get_merged_pricing` (lines 81-114). This method follows a similar pattern but targets the `PricingConfig` model:

1. Loads the **default pricing** configuration (`CONFIG_TYPE_DEFAULT_PRICING`).
2. Loads optional **custom pricing** (`CONFIG_TYPE_CUSTOM_PRICING`) if it exists.
3. If custom pricing is present, applies `deep_update` to merge the custom dictionary over the default dictionary.
4. Returns a new `PricingConfig` instance reflecting the effective pricing with all overrides applied.

## The Deep Merge Utility Behind the Scenes

The recursive merging capability relies on `deep_update` in [`lib/idp_common_pkg/idp_common/config/merge_utils.py`](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/blob/main/lib/idp_common_pkg/idp_common/config/merge_utils.py) (lines 55-71). This utility function enables the **DynamoDB configuration system** to apply fine-grained overrides without requiring operators to duplicate entire configuration sections.

The function recursively traverses nested dictionaries, overwriting existing keys with new values while preserving sibling keys that aren't mentioned in the override. This behavior is critical for maintaining backward compatibility: when new features are added to the default configuration, existing custom deltas automatically inherit the new defaults unless explicitly overridden.

## Full-Format Detection and Auto-Migration

To optimize read performance, the system distinguishes between **full configurations** and **sparse deltas** using `_is_full_config` (lines 51-72). This method checks for the `_config_format: "full"` marker or validates structural completeness by examining top-level section presence.

When a legacy sparse configuration is detected, the system performs **auto-migration**: after merging the delta with defaults, it writes the complete result back to DynamoDB with the full-format marker. This ensures that subsequent requests for the same version bypass the merge logic entirely, reducing latency and computational overhead for frequently accessed configurations.

## Practical Implementation Examples

To retrieve a merged configuration using the **DynamoDB configuration system**, instantiate the `ConfigurationManager` and request a specific version:

```python
from idp_common.config.configuration_manager import ConfigurationManager
from idp_common.config.constants import CONFIG_TYPE_CONFIG, DEFAULT_VERSION

# Initialize the configuration manager

mgr = ConfigurationManager()

# Load a specific version, automatically merging with defaults if needed

config = mgr.get_merged_configuration(version="v2024-09")

# `config` is a fully-resolved IDPConfig ready for runtime use

```

For pricing-specific overrides, use the dedicated pricing merge method:

```python

# Retrieve the effective pricing configuration (default + custom overrides)

pricing = mgr.get_merged_pricing()

# `pricing` now contains the base pricing with any custom changes applied

```

These examples demonstrate how the system abstracts the complexity of merging **default configurations** with **custom settings**, providing clean, typed objects ready for production use.

## Summary

The **DynamoDB configuration system** in the AWS Accelerated IDP solution implements a robust merging strategy through the `ConfigurationManager` class:

- **Two merge paths** handle full IDP configurations and pricing configurations separately via `get_merged_configuration` and `get_merged_pricing`.
- **Deep recursive merging** uses `deep_update` in [`merge_utils.py`](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/blob/main/merge_utils.py) to apply sparse deltas over complete defaults without destroying nested structures.
- **Full-format detection** via `_is_full_config` identifies legacy sparse records and triggers auto-migration to optimize future read performance.
- **Validation safety** ensures all merged results are validated against Pydantic models (`IDPConfig`, `PricingConfig`) before runtime use.

## Frequently Asked Questions

### How does the system handle legacy configuration formats?

The system detects legacy sparse configurations using the `_is_full_config` method in `ConfigurationManager`, which checks for the `_config_format: "full"` marker or validates structural completeness. When a legacy format is detected, the system loads the default configuration, merges the sparse delta using `deep_update`, validates the result, and writes it back to DynamoDB with the full-format marker to optimize future reads.

### What is the difference between `get_merged_configuration` and `get_merged_pricing`?

`get_merged_configuration` handles the full IDP processing pipeline configuration including OCR, classification, and extraction parameters, supporting both full-format configs and legacy sparse deltas with auto-migration. `get_merged_pricing` specifically manages pricing overrides, loading default pricing configurations and applying custom pricing deltas without the complex format detection logic required for the main IDP configuration.

### How does `deep_update` prevent configuration corruption during merging?

The `deep_update` utility in [`merge_utils.py`](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/blob/main/merge_utils.py) recursively traverses nested dictionaries, overwriting only the specific keys present in the override while preserving all sibling keys and nested structures from the default configuration. This ensures that custom settings can target specific nested parameters without requiring operators to duplicate entire configuration sections or risk losing newly added default features.

### Can custom configurations override any default parameter?

Yes, the deep-merge architecture allows custom configurations to override any parameter present in the default configuration, including deeply nested values in the OCR, classification, or extraction settings. However, the system validates the final merged result against Pydantic models (`IDPConfig` or `PricingConfig`), so overrides must conform to the expected schema types and structure to pass validation and be usable at runtime.