How to Configure Confidence Thresholds for Entity Filtering in OpenMed

OpenMed filters entity predictions by comparing each detection's confidence score against a configurable threshold, validating the input through validate_confidence_threshold in openmed/utils/validation.py (lines 108-126) and applying the filter during output formatting in openmed/processing/outputs.py (line 139).

OpenMed is an open-source medical text processing library that uses confidence thresholds to control the precision of entity predictions. When you configure confidence thresholds for entity filtering, you determine which detected entities—such as Protected Health Information (PHI) or clinical concepts—are included in the final output based on their model confidence scores.

How Confidence Thresholds Work in OpenMed

OpenMed implements confidence thresholds across four architectural layers to ensure consistent validation and filtering throughout the pipeline.

Input Validation in validation.py

The validate_confidence_threshold function in openmed/utils/validation.py (lines 108-126) serves as the single source of truth for threshold validation. It ensures any user-supplied value is a float between 0.0 and 1.0, raising a ValueError for out-of-range inputs.

Schema Enforcement in schemas.py

Pydantic models in openmed/service/schemas.py expose the confidence_threshold field with built-in constraints (ge=0.0, le=1.0). These schemas automatically invoke the validation utility, ensuring API payloads and configuration objects contain valid values before reaching the processing pipeline.

Default Thresholds in batch.py

The BatchProcessor class in openmed/processing/batch.py defines operation-specific defaults through the _default_confidence_threshold method (line 224). For example, deidentify operations default to 0.7, while extract_pii uses 0.5, and some operations use 0.0.

Filtering Logic in outputs.py

The actual filtering occurs in openmed/processing/outputs.py (line 139) within the OutputFormatter.format method. Here, the system compares each EntityPrediction.confidence against the configured threshold, omitting any predictions that fall below the cutoff.

Configuring Thresholds Via the Python API

To set a confidence threshold programmatically, validate the value and pass it to the OpenMed class constructor or individual method calls:

from openmed.core import OpenMed
from openmed.utils.validation import validate_confidence_threshold

# Validate the threshold (0.0 to 1.0)

threshold = validate_confidence_threshold(0.85)

# Apply to all operations via the core class

om = OpenMed(confidence_threshold=threshold)

# Extract entities with confidence >= 0.85

result = om.extract_pii("Patient John Doe, MRN 123456", confidence_threshold=threshold)

When you instantiate OpenMed in openmed/core/__init__.py, the threshold propagates to the underlying BatchProcessor.

Setting Thresholds in the FastAPI Service

For HTTP API deployments, send the confidence_threshold parameter in the JSON payload to the /analyze endpoint:

curl -X POST http://localhost:8000/analyze \
     -H "Content-Type: application/json" \
     -d '{
           "text": "Contact Dr. Smith at 555-0199",
           "confidence_threshold": 0.9
         }'

In openmed/service/app.py (line 280), the endpoint parses the request through OpenMedRequest schema validation and forwards payload.confidence_threshold to the processing pipeline.

Using Confidence Thresholds in CLI Scripts

Command-line interfaces built with OpenMed accept the --confidence-threshold flag:

python -m openmed.examples.privacy_filter_unified \
    --text "Alice lives in Paris." \
    --confidence-threshold 0.6

The CLI wrapper in examples/privacy_filter_unified.py parses arguments, calls validate_confidence_threshold, and initializes the OpenMed instance with the validated value.

Overriding Defaults Per Operation

Each operation type maintains distinct defaults in BatchProcessor._default_confidence_threshold. Override these by explicitly passing the threshold when creating a BatchProcessor instance:

from openmed.processing.batch import BatchProcessor

# Use stricter threshold for deidentification (default is 0.7)

processor = BatchProcessor(
    operation="deidentify", 
    confidence_threshold=0.95
)
output = processor.process("SSN: 123-45-6789")

The constructor stores the value as self.confidence_threshold, ensuring the OutputFormatter applies your custom filter during result generation.

Summary

• Validation: Use validate_confidence_threshold in openmed/utils/validation.py to ensure values fall between 0.0 and 1.0.
• Propagation: Thresholds pass through Pydantic schemas in openmed/service/schemas.py and into BatchProcessor in openmed/processing/batch.py.
• Filtering: The OutputFormatter in openmed/processing/outputs.py compares entity confidences against your threshold and excludes low-confidence predictions.
• Entry Points: Configure thresholds via the Python API (OpenMed class), FastAPI endpoints (/analyze), or CLI arguments (--confidence-threshold).
• Defaults: Operations like deidentify (0.7), extract_pii (0.5), and others have sensible defaults that you can override per instance.

Frequently Asked Questions

What is the valid range for confidence thresholds in OpenMed?

According to openmed/utils/validation.py (lines 108-126), confidence thresholds must be floating-point numbers between 0.0 and 1.0 inclusive. The validate_confidence_threshold function raises a ValueError for any value outside this range, and Pydantic schemas in schemas.py enforce these constraints at the API boundary.

Where does the actual filtering of low-confidence entities occur?

The filtering logic resides in openmed/processing/outputs.py within the OutputFormatter.format method (line 139). During output generation, each EntityPrediction object's confidence attribute is compared against the configured threshold. Predictions scoring below the threshold are omitted from the returned results while those meeting or exceeding the threshold are included.

Can I use different confidence thresholds for different operations?

Yes. While BatchProcessor._default_confidence_threshold in openmed/processing/batch.py (line 224) provides operation-specific defaults—such as 0.7 for deidentify and 0.5 for extract_pii—you can override these per instance. Pass a specific confidence_threshold value when constructing BatchProcessor or calling methods on the OpenMed class to apply custom thresholds to individual operations.

What happens if I do not specify a confidence threshold?

If no threshold is provided, OpenMed uses the default value defined in BatchProcessor._default_confidence_threshold for the specific operation being performed. For example, deidentification tasks default to 0.7, meaning only entities with confidence scores of 0.7 or higher will be returned. The validation layer ensures these defaults are always valid floats within the acceptable range.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}