How the Human-in-the-Loop Review Workflow Integrates with Amazon Augmented AI (A2I)

The Accelerated Intelligent Document Processing (IDP) solution automatically routes low-confidence extractions to Amazon SageMaker Augmented AI (A2I) for human review, using a Step Functions state machine that pauses for reviewer input and resumes based on the callback response.

The human-in-the-loop review workflow in the AWS Accelerated IDP solution leverages Amazon Augmented AI (A2I) to ensure data accuracy when automated extraction confidence scores fall below defined thresholds. This integration allows document processing pipelines to automatically escalate questionable fields to human reviewers through a managed SageMaker A2I portal, then seamlessly continue processing once review is complete. The implementation spans multiple Lambda functions, CloudFormation resources, and Step Functions orchestration logic within the aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws repository.

Confidence Evaluation and A2I Task Creation

After the extraction Lambda function processes a document, it evaluates confidence scores for each extracted field. In patterns/pattern-2/src/extraction_function/index.py, the extraction Lambda invokes document.validate_confidence() from the shared idp_common_pkg library to compare each field's confidence score against the stack parameter A2IConfidenceThreshold.

If any field falls below the threshold, the pipeline automatically creates a SageMaker A2I Human Loop. This threshold-based routing ensures that only questionable extractions reach human reviewers, minimizing unnecessary manual intervention while maintaining data quality. The architectural flow is visualized in images/hitl_a2i_workflow.drawio, which shows the complete data path from extraction through confidence checking to A2I task creation.

CloudFormation Infrastructure for A2I Integration

The underlying A2I infrastructure is provisioned through CloudFormation resources defined in scripts/generate_govcloud_template.py. This script generates the FlowDefinition, which references a HumanTaskUI resource containing the HTML form that reviewers interact with.

Key generated resources include:

• A2IFlowDefinitionRole: IAM role granting SageMaker permissions to invoke the human loop and access S3
• A2IHumanTaskUILambdaRole: Execution role for the callback Lambda that processes reviewer submissions
• CreateA2IResourcesLambda: Custom resource Lambda that provisions A2I components during stack deployment
• SageMakerA2IReviewPortalURL: CloudFormation output providing direct access to the review interface

Note that in AWS GovCloud regions, the SageMakerA2IReviewPortalURL output returns an empty string due to service availability constraints, as handled in the template generation logic.

Reviewer Experience in the A2I Portal

Human reviewers interact with documents through the SageMaker A2I Review Portal, accessible via the CloudFormation output URL. The portal renders the HumanTaskUI defined in the CloudFormation template, presenting a simple HTML form that displays only the extracted fields that fell below the confidence threshold.

Reviewers can:

• Approve the extraction as-is if the automated extraction is correct despite low confidence
• Edit specific field values to correct extraction errors
• Reject the document entirely if the extraction is unusable or the document is invalid

This interface requires no custom application development, as SageMaker A2I provides the managed hosting, authentication, and rendering for the review workflow.

Callback Processing and State Machine Resumption

When a reviewer submits their assessment, SageMaker A2I invokes the Human Task UI Lambda (configured with A2IHumanTaskUILambdaRole). This callback function writes the corrected data into the DynamoDB Document Tracking table and updates the Workflow Status to HITL_COMPLETE.

The Step Functions state machine uses an orchestration pattern implemented in patterns/pattern-2/src/rule-validation-orchestration-function/index.py to poll for this status change. Once the state machine detects HITL_COMPLETE, it resumes execution and transfers control to patterns/pattern-2/src/processresults_function/index.py.

The process results function evaluates the hitl_status field to determine the final disposition:

• If approved, the pipeline continues to store validated results in the Output S3 bucket
• If rejected, the workflow halts and marks the document as failed, preventing downstream processing of invalid data

This asynchronous polling mechanism ensures the state machine waits efficiently for human review without consuming execution resources continuously during the review period.

Configuration Parameters

The HITL-A2I integration exposes three key CloudFormation parameters that control behavior without code changes:

Parameter	Description	Default
A2IConfidenceThreshold	Minimum confidence score (0-1) required to bypass human review. Fields scoring below this value trigger the A2I workflow.	0.8
EnableHumanInTheLoop	Boolean toggle to enable or disable the entire A2I integration. When false, the pipeline processes all documents regardless of confidence scores.	true
SageMakerA2IReviewPortalURL	Exported URL of the A2I review portal. Populated automatically in commercial regions; empty in GovCloud deployments.	empty string

Adjusting A2IConfidenceThreshold allows fine-tuning of the precision-recall trade-off: lower values reduce reviewer workload but may allow more errors through, while higher values increase accuracy at the cost of additional manual reviews.

Summary

• The human-in-the-loop review workflow automatically triggers when extraction confidence falls below the A2IConfidenceThreshold parameter, routing documents to Amazon SageMaker A2I for manual verification.
• CloudFormation resources in scripts/generate_govcloud_template.py provision the FlowDefinition, HumanTaskUI, and IAM roles required for secure A2I integration.
• Reviewers interact with extracted fields through the SageMaker A2I Review Portal, approving, editing, or rejecting values via a managed HTML interface generated by the HumanTaskUI resource.
• Callback processing via the Human Task UI Lambda updates the DynamoDB Document Tracking table with reviewer decisions, enabling the Step Functions state machine in patterns/pattern-2/src/rule-validation-orchestration-function/index.py to resume execution.
• The workflow supports configurable thresholds and can be disabled entirely via the EnableHumanInTheLoop parameter, making it adaptable to varying accuracy and compliance requirements across different document types.

Frequently Asked Questions

What triggers the human-in-the-loop review in the IDP pipeline?

The review triggers when any extracted field's confidence score falls below the A2IConfidenceThreshold parameter (default 0.8). In patterns/pattern-2/src/extraction_function/index.py, the code calls document.validate_confidence() to evaluate scores against this threshold, automatically creating an A2I human loop when values are insufficient.

How does the Step Functions state machine know when human review is complete?

The state machine uses an orchestration pattern implemented in patterns/pattern-2/src/rule-validation-orchestration-function/index.py to poll the DynamoDB Document Tracking table for the HITL_COMPLETE status. When the A2I callback Lambda updates this status after reviewer submission, the state machine detects the change and resumes execution within seconds.

Can I disable the A2I integration if I don't need human review?

Yes. Set the EnableHumanInTheLoop CloudFormation parameter to false during stack deployment. This disables the A2I FlowDefinition creation and bypasses the confidence-checking logic in the extraction Lambda, allowing documents to proceed through the pipeline without human intervention regardless of confidence scores.

What happens if a reviewer rejects a document in the A2I portal?

When a reviewer rejects a document, the callback Lambda stores this decision in the hitl_status field of the DynamoDB record. The patterns/pattern-2/src/processresults_function/index.py function evaluates this status and halts the workflow rather than proceeding to store results in the Output bucket, effectively marking the document as failed processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}