How the DynamoDB Configuration System Merges Default Configurations with Custom Settings in AWS IDP

The DynamoDB configuration system in the Accelerated Intelligent Document Processing (IDP) solution uses a ConfigurationManager class to recursively merge default configurations with custom settings using a deep-update utility, automatically detecting and migrating legacy sparse formats to full-configuration records.

The aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws repository implements a sophisticated runtime configuration system that stores all processing parameters in DynamoDB. Understanding how this DynamoDB configuration system combines baseline defaults with environment-specific overrides is essential for customizing the IDP pipeline without breaking core functionality.

Understanding the ConfigurationManager Architecture

At the heart of the system lies the ConfigurationManager class defined in lib/idp_common_pkg/idp_common/config/configuration_manager.py. This class acts as the central authority for reading, merging, and persisting configuration states. Rather than requiring operators to define complete configuration documents for every deployment, the system maintains default configurations as the baseline and layers custom settings on top through a safe, recursive merge process.

The architecture supports two distinct merge workflows: one for the full IDP processing configuration (OCR, classification, extraction) and another specifically for pricing parameters.

How Default and Custom Configurations Are Merged

The system implements separate merge strategies depending on the configuration type, both utilizing the deep_update utility to ensure nested dictionaries are merged rather than replaced.

Merging Full IDP Configurations

The primary merge logic resides in ConfigurationManager.get_merged_configuration (lines 200-229). This method handles the DynamoDB configuration system's core responsibility: resolving a requested version into a complete, valid IDPConfig object.

The process follows four distinct steps:

1. Full-format detection: The method first checks if the requested version exists as a full-format configuration using _is_full_config (lines 51-72). This checks for the explicit marker _config_format: "full" or validates that enough top-level sections exist to consider the record complete.

2. Legacy delta handling: If the item is a legacy sparse delta (containing only changed fields), the system loads the default version (DEFAULT_VERSION) and retrieves the raw delta via get_raw_configuration.

3. Deep merge execution: The default model is converted to a plain dictionary using model_dump, and the delta is deep-merged into it using deep_update (defined in lib/idp_common_pkg/idp_common/config/merge_utils.py, lines 55-71). This utility recursively overwrites nested keys while preserving untouched sections.

4. Validation and auto-migration: The merged dictionary is validated back into an IDPConfig object. The system then auto-migrates the result to full format by calling save_configuration (lines 81-85), writing the merged result back to DynamoDB with the full-config marker so subsequent reads skip the merge step.

Merging Pricing Configurations

For pricing-specific overrides, the system uses ConfigurationManager.get_merged_pricing (lines 81-114). This method follows a similar pattern but targets the PricingConfig model:

1. Loads the default pricing configuration (CONFIG_TYPE_DEFAULT_PRICING).
2. Loads optional custom pricing (CONFIG_TYPE_CUSTOM_PRICING) if it exists.
3. If custom pricing is present, applies deep_update to merge the custom dictionary over the default dictionary.
4. Returns a new PricingConfig instance reflecting the effective pricing with all overrides applied.

The Deep Merge Utility Behind the Scenes

The recursive merging capability relies on deep_update in lib/idp_common_pkg/idp_common/config/merge_utils.py (lines 55-71). This utility function enables the DynamoDB configuration system to apply fine-grained overrides without requiring operators to duplicate entire configuration sections.

The function recursively traverses nested dictionaries, overwriting existing keys with new values while preserving sibling keys that aren't mentioned in the override. This behavior is critical for maintaining backward compatibility: when new features are added to the default configuration, existing custom deltas automatically inherit the new defaults unless explicitly overridden.

Full-Format Detection and Auto-Migration

To optimize read performance, the system distinguishes between full configurations and sparse deltas using _is_full_config (lines 51-72). This method checks for the _config_format: "full" marker or validates structural completeness by examining top-level section presence.

When a legacy sparse configuration is detected, the system performs auto-migration: after merging the delta with defaults, it writes the complete result back to DynamoDB with the full-format marker. This ensures that subsequent requests for the same version bypass the merge logic entirely, reducing latency and computational overhead for frequently accessed configurations.

Practical Implementation Examples

To retrieve a merged configuration using the DynamoDB configuration system, instantiate the ConfigurationManager and request a specific version:

from idp_common.config.configuration_manager import ConfigurationManager
from idp_common.config.constants import CONFIG_TYPE_CONFIG, DEFAULT_VERSION

# Initialize the configuration manager

mgr = ConfigurationManager()

# Load a specific version, automatically merging with defaults if needed

config = mgr.get_merged_configuration(version="v2024-09")

# `config` is a fully-resolved IDPConfig ready for runtime use

For pricing-specific overrides, use the dedicated pricing merge method:

# Retrieve the effective pricing configuration (default + custom overrides)

pricing = mgr.get_merged_pricing()

# `pricing` now contains the base pricing with any custom changes applied

These examples demonstrate how the system abstracts the complexity of merging default configurations with custom settings, providing clean, typed objects ready for production use.

Summary

The DynamoDB configuration system in the AWS Accelerated IDP solution implements a robust merging strategy through the ConfigurationManager class:

• Two merge paths handle full IDP configurations and pricing configurations separately via get_merged_configuration and get_merged_pricing.
• Deep recursive merging uses deep_update in merge_utils.py to apply sparse deltas over complete defaults without destroying nested structures.
• Full-format detection via _is_full_config identifies legacy sparse records and triggers auto-migration to optimize future read performance.
• Validation safety ensures all merged results are validated against Pydantic models (IDPConfig, PricingConfig) before runtime use.

Frequently Asked Questions

How does the system handle legacy configuration formats?

The system detects legacy sparse configurations using the _is_full_config method in ConfigurationManager, which checks for the _config_format: "full" marker or validates structural completeness. When a legacy format is detected, the system loads the default configuration, merges the sparse delta using deep_update, validates the result, and writes it back to DynamoDB with the full-format marker to optimize future reads.

What is the difference between get_merged_configuration and get_merged_pricing?

get_merged_configuration handles the full IDP processing pipeline configuration including OCR, classification, and extraction parameters, supporting both full-format configs and legacy sparse deltas with auto-migration. get_merged_pricing specifically manages pricing overrides, loading default pricing configurations and applying custom pricing deltas without the complex format detection logic required for the main IDP configuration.

How does deep_update prevent configuration corruption during merging?

The deep_update utility in merge_utils.py recursively traverses nested dictionaries, overwriting only the specific keys present in the override while preserving all sibling keys and nested structures from the default configuration. This ensures that custom settings can target specific nested parameters without requiring operators to duplicate entire configuration sections or risk losing newly added default features.

Can custom configurations override any default parameter?

Yes, the deep-merge architecture allows custom configurations to override any parameter present in the default configuration, including deeply nested values in the OCR, classification, or extraction settings. However, the system validates the final merged result against Pydantic models (IDPConfig or PricingConfig), so overrides must conform to the expected schema types and structure to pass validation and be usable at runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}