How to Run OpenMed Tests: A Complete Guide to Unit, Integration, and Evaluation Testing

Run OpenMed tests using pytest after installing development dependencies with pip install -e ".[dev]", then execute pytest -m unit for unit tests, pytest -m integration for integration tests, or pytest -m eval for evaluation tests from the repository root.

OpenMed is a Python package for medical text processing that includes a comprehensive test suite covering unit, integration, and evaluation checks. To ensure code quality and reliability, the project uses pytest as its testing framework with tests organized under the tests/ directory. This guide explains how to set up your environment and execute the OpenMed test suite efficiently according to the maziyarpanahi/openmed source code.

Prerequisites and Installation

Before running tests, install the package in editable mode with development dependencies. The pyproject.toml file defines these under [tool.poetry.dev-dependencies], including pytest, pytest-cov, torch, and datasets.

Clone the repository and run the installation command:

git clone https://github.com/maziyarpanahi/openmed.git
cd openmed
pip install -e ".[dev]"

The .[dev] extra ensures all testing utilities and compatible versions of PyTorch are available.

Test Suite Architecture

The OpenMed test suite follows a structured directory pattern under tests/ with separate categories for different testing levels.

Directory Structure

The repository organizes tests into three distinct categories:

• tests/unit/ – Contains unit tests for individual components like the core processing pipeline in tests/unit/test_core.py. These tests validate isolated functionality without external dependencies.
• tests/integration/ – Houses integration tests such as tests/integration/test_end_to_end.py that verify the full service API and component interactions.
• tests/eval/ – Includes evaluation tests like tests/eval/test_metrics.py that compare model outputs against golden fixtures to ensure accuracy.

Fixtures and Configuration

The tests/conftest.py file defines global pytest fixtures including temporary directories, deterministic random seeds, and reusable objects like Processor and ModelRegistry. Category-specific fixtures in tests/unit/conftest.py provide mock services and sample medical texts for their respective test suites.

Running the OpenMed Test Suite

Execute tests from the repository root using pytest commands. The test discovery pattern follows test_*.py conventions throughout the codebase.

Run All Tests

To execute the complete suite including unit, integration, and evaluation tests:

pytest

Run Specific Test Categories

Use pytest markers to filter by test type:

# Run only unit tests

pytest -m unit

# Run only integration tests  

pytest -m integration

# Run only evaluation tests

pytest -m eval

These markers correspond to the directory structure and are configured in the project configuration files.

Run with Coverage

Generate coverage reports to identify untested code paths:

pytest --cov=openmed --cov-report=html
open htmlcov/index.html

The --cov=openmed flag tracks coverage specifically for the main package source code under the openmed/ directory.

CI/CD Pipeline Configuration

The repository uses GitHub Actions to automate testing on every push. The workflow file .github/workflows/ci.yml implements the following steps:

1. Environment Setup – Installs the package with pip install -e ".[dev]"
2. Caching – Preserves ~/.cache directories for torch and datasets to accelerate subsequent runs
3. Test Execution – Runs pytest --cov=openmed to collect coverage metrics

This configuration ensures that tests pass consistently across different environments and that coverage thresholds are maintained.

Troubleshooting Common Test Failures

When running OpenMed tests, you may encounter specific errors related to dependencies or environment configuration.

ImportError: cannot import name 'torch' – Indicates PyTorch is missing or incorrectly versioned. Resolve by reinstalling with pip install -e ".[dev]" to pull compatible torch builds.

Tests failing on data download – Network restrictions or missing cached datasets prevent HuggingFace datasets from loading. Ensure internet access or pre-populate ~/.cache/huggingface/datasets before running tests.

Slow test execution – Large model checkpoints downloading repeatedly causes delays. The CI caches ~/.cache locally; ensure you reuse this path or set the appropriate cache environment variables.

AssertionError in privacy-filter tests – These tests use placeholder values and do not require the OPENMED_PRIVACY_KEY environment variable. No real secrets are needed for the test suite to pass.

Summary

• Install OpenMed with development dependencies using pip install -e ".[dev]" before testing
• Structure includes three categories: tests/unit/, tests/integration/, and tests/eval/
• Execute specific test types with markers: pytest -m unit, pytest -m integration, or pytest -m eval
• Configure coverage reporting with pytest --cov=openmed to track code quality
• Reference .github/workflows/ci.yml for the official continuous integration setup

Frequently Asked Questions

Do I need a GPU to run OpenMed tests?

No, the OpenMed test suite runs on CPU. While the package supports GPU acceleration for inference, the pytest fixtures in tests/conftest.py configure tests to use CPU-only mode by default, ensuring compatibility across all development environments.

How do I run a single test file instead of the entire suite?

Use the file path as an argument to pytest. For example, to run only the core unit tests: pytest tests/unit/test_core.py. You can also run specific test functions using the :: syntax: pytest tests/unit/test_core.py::test_specific_function.

What should I do if integration tests fail due to missing external services?

The integration tests in tests/integration/test_end_to_end.py use mock services configured in tests/integration/conftest.py. If failures occur, verify that fixtures are properly loading by checking that you installed all dev dependencies. No external service endpoints are required for the test suite to execute.

Where are the test fixtures and sample data defined?

Global fixtures reside in tests/conftest.py, while category-specific fixtures appear in subdirectories like tests/unit/conftest.py. These files define reusable objects including sample processors, model registries, and medical text samples that inject deterministic test data into the suite.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}