Nested CloudFormation Stack Architecture in AWS IDP: How the Patterns Interconnect

The Accelerated Intelligent Document Processing on AWS solution uses a single top-level SAM template that orchestrates multiple nested CloudFormation stacks, where each processing pattern operates as an isolated nested stack while sharing core infrastructure like S3 buckets, DynamoDB tables, and IAM roles through cross-stack parameter passing.

The aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws repository implements this nested CloudFormation stack architecture to enable modular deployment of intelligent document processing capabilities. This design allows organizations to select specific AI/ML patterns—such as Bedrock Data Automation or Textract with SageMaker—without deploying unnecessary resources, while maintaining a consistent foundation of shared services.

Core Architecture Overview

Top-Level Orchestration Template

The architecture centers on a single root template located at template.yaml in the repository root. This top-level stack acts as the orchestration layer, defining the foundational infrastructure and conditionally instantiating nested stacks based on deployment parameters.

The root template declares three primary pattern conditions—IsPattern1, IsPattern2, and IsPattern3—which evaluate the IDPPattern parameter to determine which nested stacks to provision. This conditional logic ensures that only the selected pattern's resources are deployed, optimizing cost and complexity.

Shared Infrastructure Components

The top-level template provisions core resources that all patterns consume:

• S3 Buckets: InputBucket for document ingestion, OutputBucket for processed results, and ConfigurationBucket for workflow configurations
• DynamoDB Tables: TrackingTable for workflow state management across all patterns
• IAM Roles: Cross-service execution roles with least-privilege permissions
• KMS Keys: CustomerManagedEncryptionKey for encryption at rest

These resources are created once and referenced by all nested stacks through parameter passing, ensuring consistent security posture and data lineage across different processing patterns.

Pattern-Specific Nested Stacks

Each processing pattern resides in its own nested stack under the patterns/ directory, isolating pattern-specific Lambda functions, Step Functions state machines, and IAM policies.

Pattern 1: Bedrock Data Automation

Located at patterns/pattern-1/template.yaml, this nested stack implements document processing using Amazon Bedrock Data Automation (BDA). The stack defines:

• An Amazon ECR repository for custom container images
• Docker-based Lambda functions for BDA invocation
• Step Functions workflows that orchestrate BDA project execution
• Pattern-specific IAM roles with permissions to invoke Bedrock models

The stack receives the Pattern1BDAProjectArn parameter from the top level, or optionally integrates with the BDASampleProject nested stack when users do not provide an existing BDA project ARN.

Pattern 2: Textract and Bedrock

The patterns/pattern-2/template.yaml nested stack combines Amazon Textract for OCR with Amazon Bedrock for intelligent classification and extraction. Key components include:

• Container image build resources for custom processing logic
• Lambda functions for Textract result processing and Bedrock prompt engineering
• Step Functions state machines that coordinate Textract async jobs with Bedrock inference
• Dedicated CloudWatch dashboards for pattern-specific metrics

This pattern consumes the shared InputBucket and OutputBucket references to read source documents and write structured extraction results.

Pattern 3: Textract, SageMaker, and Bedrock

Found at patterns/pattern-3/template.yaml, this nested stack extends Pattern 2 by adding Amazon SageMaker endpoints for custom document understanding models (UDOP). The stack provisions:

• SageMaker model artifact handling and endpoint configuration
• Lambda functions for SageMaker inference integration
• Enhanced Step Functions workflows that branch between Textract, SageMaker, and Bedrock based on document characteristics
• Pattern-specific VPC configurations for SageMaker networking

All three pattern stacks export their StateMachineArn and CloudWatchDashboardName outputs, enabling the top-level stack to aggregate monitoring resources and UI integrations.

Optional Supporting Stacks

The architecture includes several optional nested stacks under the nested/ directory that enhance functionality without requiring pattern modifications.

AppSync GraphQL API

The nested/appsync/template.yaml stack provisions a GraphQL API that powers the solution's web interface and enables external client integrations. The stack creates:

• AWS AppSync API and schema definitions
• DynamoDB data sources linked to the shared TrackingTable
• Lambda resolvers that invoke pattern Step Functions using the exported StateMachineArn references
• Cognito user pool integration for authentication

This stack is conditionally created when the CreateAgentCoreLambda parameter is set to true, and it receives the pattern state machine ARNs through the Pattern1StateMachineArn, Pattern2StateMachineArn, and Pattern3StateMachineArn parameters.

Bedrock Knowledge Base

Located at nested/bedrockkb/template.yaml, this stack implements a Retrieval-Augmented Generation (RAG) pipeline for document search use cases. Components include:

• Amazon Bedrock Knowledge Base configuration
• OpenSearch Serverless vector store or Aurora PostgreSQL vector storage (configurable via VectorStoreType)
• S3 data source pointing to the shared OutputBucket
• IAM roles for Bedrock model invocation and vector store access

The stack is instantiated when ShouldCreateDocumentKnowledgeBase evaluates to true, and it integrates with pattern outputs to ensure processed documents are automatically indexed for semantic search.

Sample BDA Project

The nested/bda-lending-project/template.yaml stack provides a ready-to-use Bedrock Data Automation project for users who do not have an existing BDA configuration. This stack:

• Creates a sample BDA project with pre-configured extraction schemas
• Outputs the BDAProjectArn for consumption by Pattern 1
• Includes sample blueprints for lending document processing

This optional stack bridges the gap for quick starts, allowing Pattern 1 to function immediately without manual BDA project creation.

Cross-Stack Communication and Resource Sharing

Parameter Passing Mechanism

The nested CloudFormation stack architecture relies on explicit parameter passing to share resources between the top-level stack and nested stacks. The top-level template declares each nested stack with a Properties.Parameters block that maps shared resource references:

PATTERN2STACK:
  Type: AWS::CloudFormation::Stack
  Condition: IsPattern2
  Properties:
    TemplateURL: ./patterns/pattern-2/.aws-sam/packaged.yaml
    Parameters:
      InputBucket: !Ref InputBucket
      ConfigurationBucket: !Ref ConfigurationBucket
      OutputBucket: !Ref OutputBucket
      TrackingTable: !Ref TrackingTable
      CustomerManagedEncryptionKeyArn: !GetAtt CustomerManagedEncryptionKey.Arn

This mechanism ensures that nested stacks receive the actual resource ARNs and names at deployment time, enabling them to configure IAM policies and environment variables correctly without hardcoding values.

Inter-Pattern Connectivity

While each pattern stack operates independently, they interconnect through the shared core infrastructure:

Document Ingestion Flow The core InputBucket triggers a Queue Sender Lambda (defined in the top-level stack) that writes messages to an SQS queue. The Queue Processor Lambda then initiates the appropriate pattern's Step Functions execution using the state machine ARN exported by that pattern's nested stack.

Workflow Tracking All pattern Lambdas read from and write to the shared TrackingTable DynamoDB table, maintaining a unified view of document processing status across different AI/ML approaches. The table schema is defined once in the top-level stack and consumed by all patterns.

Result Storage Pattern-specific Lambdas write processed JSON outputs and PDFs to the shared OutputBucket. This centralization allows the optional Bedrock Knowledge Base stack to index results regardless of which pattern processed the document, and enables unified reporting through the core infrastructure.

UI Integration The optional AppSync nested stack creates GraphQL resolvers that reference the pattern state machine ARNs (passed as parameters from the top-level stack). This allows a single web interface to trigger executions across Pattern 1, Pattern 2, or Pattern 3 without knowing the internal implementation details of each.

Deployment Examples

Deploying Pattern 2 via AWS CLI

To deploy the nested CloudFormation stack architecture with Pattern 2 (Textract + Bedrock) enabled, use the following command structure:

aws cloudformation deploy \
    --template-file template.yaml \
    --stack-name idp-accelerator \
    --parameter-overrides \
        IDPPattern=pattern-2 \
        AdminEmail=admin@example.com \
        MaxConcurrentExecutions=200 \
        ShouldCreateDocumentKnowledgeBase=true \
    --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND

The IDPPattern=pattern-2 parameter triggers the IsPattern2 condition, causing CloudFormation to instantiate the PATTERN2STACK nested stack from patterns/pattern-2/.aws-sam/packaged.yaml.

Referencing Pattern Outputs in Custom Resources

When extending the solution with custom monitoring Lambdas, reference the pattern state machine ARN using the nested stack output:

Resources:
  CustomMonitoringFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/monitoring/
      Handler: index.handler
      Runtime: python3.12
      Environment:
        Variables:
          PATTERN_STATE_MACHINE_ARN: !GetAtt PATTERN2STACK.Outputs.StateMachineArn
          TRACKING_TABLE_NAME: !Ref TrackingTable

This configuration allows the monitoring function to poll Step Functions execution history using the exact ARN exported by the Pattern 2 nested stack, while reading status updates from the shared TrackingTable defined in the top-level stack.

Summary

• The Accelerated Intelligent Document Processing on AWS solution uses a single top-level SAM template (template.yaml) to orchestrate multiple nested CloudFormation stacks, isolating pattern-specific resources while sharing core infrastructure.
• Three primary patterns (Bedrock Data Automation, Textract+Bedrock, Textract+SageMaker+Bedrock) each reside in their own nested stack under patterns/pattern-{1,2,3}/template.yaml, receiving shared resource references via parameters.
• Optional nested stacks for AppSync, Bedrock Knowledge Base, and sample BDA projects extend functionality without modifying core or pattern templates, connecting through the same parameter-passing mechanism.
• Cross-stack communication occurs through shared S3 buckets, DynamoDB tables, and explicit CloudFormation outputs, enabling unified workflow tracking and UI integration across disparate AI/ML processing paths.

Frequently Asked Questions

How does the top-level template decide which pattern stack to deploy?

The top-level template.yaml defines Boolean conditions such as IsPattern1, IsPattern2, and IsPattern3 that evaluate the IDPPattern parameter value. When you specify --parameter-overrides IDPPattern=pattern-2, the IsPattern2 condition becomes true, causing CloudFormation to create the PATTERN2STACK resource of type AWS::CloudFormation::Stack while skipping the others. This conditional logic ensures you deploy only the AI/ML pipeline required for your use case.

What shared resources do all pattern stacks receive from the top-level template?

Every pattern nested stack receives critical infrastructure references through the Parameters property of the AWS::CloudFormation::Stack resource. These include the InputBucket (for document ingestion), OutputBucket (for results), ConfigurationBucket (for workflow settings), TrackingTable (DynamoDB for status tracking), and CustomerManagedEncryptionKeyArn (KMS for encryption). By passing these ARNs and names as parameters, the nested stacks can configure IAM policies and environment variables without hardcoding resource identifiers.

How do the optional nested stacks integrate with the processing patterns?

Optional stacks such as nested/appsync/template.yaml and nested/bedrockkb/template.yaml integrate through the same parameter-passing mechanism used by required patterns. The AppSync stack receives pattern state machine ARNs (e.g., Pattern1StateMachineArn) as parameters, allowing GraphQL resolvers to trigger specific workflows. The Bedrock Knowledge Base stack receives the OutputBucket parameter, enabling automatic indexing of documents processed by any active pattern. These stacks are conditionally created based on parameters like ShouldCreateDocumentKnowledgeBase and CreateAgentCoreLambda.

Can I deploy multiple patterns simultaneously in the same account?

The current architecture supports deploying only one pattern per stack instance due to the IDPPattern parameter and mutually exclusive conditions (IsPattern1, IsPattern2, IsPattern3). However, you can deploy multiple instances of the entire solution by using different stack names (e.g., idp-pattern1-prod and idp-pattern2-prod) in the same AWS account and region. Each instance maintains its own isolated set of nested stacks while sharing no state, though you should verify service quotas for resources like Step Functions and Lambda concurrent executions when running multiple patterns concurrently.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}