How the Configuration Versioning System Enables Version-Specific Document Processing in AWS IDP

The AWS IDP configuration versioning system stores complete, compressed configuration snapshots in DynamoDB and attaches version metadata to S3 objects, allowing each document to be processed with its specific configuration version throughout the pipeline without stack redeployment.

The accelerated intelligent document processing (IDP) solution on AWS uses a sophisticated configuration versioning system to manage OCR, classification, and extraction settings. This system enables teams to maintain multiple configuration snapshots—such as default, Production, or Experiment-A—and process documents against specific versions without redeploying infrastructure. Each version represents a complete, immutable snapshot stored in Amazon DynamoDB, ensuring reproducible processing across the entire document pipeline.

How Configuration Snapshots Are Stored in DynamoDB

The system persists versioned configurations in a DynamoDB table named ConfigurationTable. Unlike incremental updates, each entry constitutes a full, self-contained snapshot containing every setting required for document processing.

Full-Config Markers and Compression

To ensure completeness, the system marks snapshots with _config_format: "full" in lib/idp_common_pkg/idp_common/config/configuration_manager.py (lines 31-34). This marker guarantees that the version contains complete OCR, classification, extraction, class, assessment, and summarization settings.

To overcome DynamoDB's 400 KB item size limit, configurations are gzip-compressed into the _compressed_config attribute (lines 35-39 in the same file). This compression enables storage of complex, multi-page extraction templates without truncation.

Runtime Version Selection via S3 Metadata

Documents specify their target configuration version through S3 object metadata. When a file enters the system, the config-version metadata key determines which snapshot the pipeline loads.

CLI and Batch Processing Injection

The BatchProcessor class in lib/idp_sdk/idp_sdk/core/batch_processor.py handles metadata attachment during uploads. When processing local files, it injects the version via put_object with the Metadata parameter:

if config_version:
    logger.info(f"Adding config-version metadata: {config_version} to {s3_key}")
    self.s3.put_object(
        Bucket=input_bucket,
        Key=s3_key,
        Body=file_data,
        Metadata={"config-version": config_version},
    )

For existing S3 objects, the processor uses copy_object with MetadataDirective="REPLACE" to inject or update the version metadata (lines 62-68).

Lambda Propagation

The queue_sender Lambda function in src/lambda/queue_sender/index.py extracts the metadata and populates the document.config_version field (lines 49-71). This ensures the version identifier flows through the entire Step Functions workflow, making it available to every downstream processing stage.

Version-Aware Processing in Pipeline Functions

Each processing stage—OCR, classification, extraction—retrieves its settings by calling get_config() with the specific version parameter.

For example, the OCR function in patterns/pattern-2/src/ocr_function/index.py loads version-specific settings:

from idp_common.config import ConfigurationReader

# version comes from the event payload (originally S3 metadata)

config_version = event.get("configVersion")
reader = ConfigurationReader(table_name="ConfigurationTable")
config = reader.get_merged_configuration(version=config_version, as_model=True)

This pattern repeats across classification, extraction, and assessment functions, ensuring each document processes against its designated configuration snapshot regardless of when it entered the system.

Managing Versions Through the CLI and UI

The solution exposes version management through both command-line tools and the web interface.

CLI Commands

The idp-cli tool provides direct version management as documented in docs/configuration-versions.md:

Download a specific version:

idp-cli config-download \
  --stack-name my-stack \
  --config-version Production

Upload a new version:

idp-cli config-upload \
  --stack-name my-stack \
  --config-file ./config.yaml \
  --config-version Experiment-A \
  --version-description "Testing nova-2-lite extraction prompts"

Run inference with a chosen version:

idp-cli run-inference \
  --stack-name my-stack \
  --dir ./documents/ \
  --config-version Production \
  --monitor

Web Interface

The web UI provides a Configuration Version dropdown in the document upload, re-process, and discovery panels. This dropdown lists all versions stored in DynamoDB. When users select a version and upload a file, the UI attaches the config-version metadata to the S3 object, triggering the version-aware processing pipeline described above.

Summary

The AWS IDP configuration versioning system enables precise, reproducible document processing through these key mechanisms:

• Immutable snapshots: Each version stores a complete, gzip-compressed configuration in DynamoDB with the _config_format: "full" marker, ensuring all OCR, classification, and extraction settings are self-contained.
• Metadata-driven selection: The config-version S3 object metadata determines which configuration snapshot processes each document, injected via the BatchProcessor in lib/idp_sdk/idp_sdk/core/batch_processor.py.
• Pipeline propagation: The queue_sender Lambda extracts metadata into document.config_version, and each processing function calls get_config(version=config_version) to load settings, as seen in patterns/pattern-2/src/ocr_function/index.py.
• Operational flexibility: CLI commands and UI dropdowns enable version management without infrastructure redeployment, supporting A/B testing and staged rollouts.

Frequently Asked Questions

How does the system ensure a configuration version contains complete settings?

The system marks every stored version with _config_format: "full" in lib/idp_common_pkg/idp_common/config/configuration_manager.py. This marker guarantees that the snapshot includes all required sections—OCR engines, classification rules, extraction prompts, assessment criteria, and summarization parameters—preventing partial configuration errors during processing.

What happens if a document is uploaded without specifying a configuration version?

If no config-version metadata is attached to the S3 object, the pipeline typically defaults to a version named default or falls back to the baseline configuration. The get_config() method in the configuration manager handles version resolution, ensuring documents always process against a valid configuration snapshot even when explicit versioning is omitted.

How does the configuration versioning system handle large configuration files?

To overcome DynamoDB's 400 KB item size limit, the system compresses configuration JSON using gzip before storage. The _compressed_config attribute in the DynamoDB item stores this payload, while the _config_format field maintains the uncompressed marker. This compression enables storage of complex, multi-page extraction templates and extensive prompt libraries without truncation.

Can I process the same document with different configuration versions for A/B testing?

Yes. The system supports A/B testing by allowing you to upload the same document multiple times with different config-version metadata values, or use the CLI to re-process existing documents with alternate versions. Each processing run retrieves its designated configuration snapshot from DynamoDB, enabling direct comparison of OCR accuracy, extraction quality, or classification performance across different prompt engineering experiments or model configurations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}