What Tests Can Open SWE Generate? A Complete Guide to the pytest Suite

Open SWE generates pytest-based unit tests that validate deterministic thread-ID generation, webhook handling, sandbox path resolution, and safety checks across Slack, GitHub, and Linear integrations.

The langchain-ai/open-swe repository ships with a comprehensive test suite designed to verify the reliability of its internal coding agent. These tests focus on unit-level correctness for critical behaviors such as message formatting, authentication validation, and multimodal content extraction, ensuring the framework operates reliably within sandboxed environments.

pytest-Based Unit Testing Architecture

Open SWE leverages the standard pytest framework for all test validation. Unlike full-stack integration tests, the suite emphasizes self-contained unit tests that isolate specific utility functions and integration points. This approach allows the CI pipeline to handle broader integration testing while the test suite verifies deterministic behaviors in modules like agent/utils/slack.py and agent/webapp.py.

The tests execute within the same sandboxed environment that Open SWE provisions for agent tasks, ensuring consistent behavior across development and production contexts.

Core Integration Test Categories

The test suite organizes validation by feature area, covering the most critical behaviors an internal coding agent must handle when processing external events and managing sandboxed execution.

Slack Integration Tests (tests/test_slack_context.py)

The Slack integration tests in tests/test_slack_context.py validate deterministic thread-ID generation and proper message context handling. These tests verify correct selection of context messages based on mentions, proper stripping and replacing of bot mentions, and the formatting of Slack messages for LLM prompts. The underlying logic resides in agent/utils/slack.py.

GitHub Webhook Handling (tests/test_github_issue_webhook.py)

Located in tests/test_github_issue_webhook.py, these tests ensure reliable processing of GitHub events through agent/webapp.py. Validation includes deterministic thread-ID generation for issues, prompt construction that includes issue details, and correct handling of issue-comment events and reaction tokens.

Sandbox Path Resolution (tests/test_sandbox_paths.py)

The tests/test_sandbox_paths.py file validates the sandbox path helper in agent/utils/sandbox_paths.py. Tests confirm the use of the provider's working directory, fallback to the user's home directory when the sandbox path is not writable, and proper caching of resolved paths to prevent redundant filesystem operations.

Comment Processing and Extraction

Open SWE handles comment streams from multiple sources, requiring robust extraction and sanitization logic.

Recent Comment Extraction (tests/test_recent_comments.py): Validates proper handling of empty comment streams, ignoring bot-generated messages, and collection of comments since the last bot message.

GitHub Comment Prompt Generation (tests/test_github_comment_prompts.py): Tests the wrapping of external comments with trust sections and sanitization of reserved tags in comment bodies to prevent prompt injection.

Multimodal Content Extraction (tests/test_multimodal.py)

The tests/test_multimodal.py suite verifies image URL extraction capabilities in agent/utils/multimodal.py. Tests cover extraction from both markdown and direct links, deduplication and case-insensitive handling, and proper filtering of non-image URLs.

Safety and Authentication Checks

Tool-Message Safety (tests/test_ensure_no_empty_msg.py): Validates detection of commit_and_open_pr calls, identification of completion-confirmation messages, and proper filtering of non-tool messages to prevent empty agent responses.

Authentication Sources (tests/test_auth_sources.py): Verifies validation of required environment variables for service authentication, ensuring the agent fails gracefully when credentials are missing.

Creating and Running Custom Tests

Developers can extend the test suite by adding new pytest files to the tests/ directory. Each test should import the relevant utilities from the agent/ package and validate specific function outputs.

Here is an example of adding a unit test for a utility function:

# Example: Adding a new unit test for a utility function

# Save as tests/test_my_utils.py

import pytest
from agent.utils.my_utils import compute_similarity

def test_compute_similarity_returns_expected_value() -> None:
    a = "hello world"
    b = "hello there"
    # Expect a similarity score around 0.8 (implementation-specific)

    assert 0.7 < compute_similarity(a, b) < 0.9

Run the suite locally using standard pytest commands:

pytest -q tests/test_my_utils.py

The repository's CI configuration automatically discovers and executes tests in the tests/ directory alongside existing validations.

Summary

• Open SWE generates pytest-based unit tests that validate core agent behaviors without requiring full-stack integration setups.
• The test suite covers Slack and GitHub integrations, sandbox path resolution, multimodal content extraction, and safety checks across eight specialized test files.
• All tests are self-contained and execute within the sandboxed environment defined in agent/utils/sandbox_paths.py.
• Developers can extend coverage by adding files to the tests/ directory using standard pytest patterns.

Frequently Asked Questions

What testing framework does Open SWE use?

Open SWE uses pytest as its primary testing framework. All tests are written as standard pytest functions and stored in the tests/ directory at the repository root.

Are Open SWE tests integration tests or unit tests?

The tests are primarily unit tests that validate specific functions and deterministic behaviors. While they verify integration points like Slack and GitHub webhooks, they isolate these components rather than testing full-stack workflows. The CI pipeline handles broader integration testing.

How do I run the Open SWE test suite locally?

Execute tests using pytest -q tests/ from the repository root. For individual test files, specify the path directly, such as pytest -q tests/test_slack_context.py. The framework automatically picks up any Python files matching the test_*.py pattern in the tests/ directory.

Where are the test files located in the repository?

All test files reside in the tests/ directory. Key files include tests/test_slack_context.py for Slack logic, tests/test_github_issue_webhook.py for GitHub processing, and tests/test_sandbox_paths.py for filesystem validation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}