# What Tests Can Open SWE Generate? A Complete Guide to the pytest Suite

> Discover the expansive pytest suite Open SWE generates for comprehensive testing of Slack, GitHub, and Linear integrations including thread-ID generation, webhook handling, and safety checks.

- Repository: [LangChain/open-swe](https://github.com/langchain-ai/open-swe)
- Tags: how-to-guide
- Published: 2026-03-19

---

**Open SWE generates pytest-based unit tests that validate deterministic thread-ID generation, webhook handling, sandbox path resolution, and safety checks across Slack, GitHub, and Linear integrations.**

The langchain-ai/open-swe repository ships with a comprehensive test suite designed to verify the reliability of its internal coding agent. These tests focus on unit-level correctness for critical behaviors such as message formatting, authentication validation, and multimodal content extraction, ensuring the framework operates reliably within sandboxed environments.

## pytest-Based Unit Testing Architecture

Open SWE leverages the standard **pytest** framework for all test validation. Unlike full-stack integration tests, the suite emphasizes **self-contained unit tests** that isolate specific utility functions and integration points. This approach allows the CI pipeline to handle broader integration testing while the test suite verifies deterministic behaviors in modules like [`agent/utils/slack.py`](https://github.com/langchain-ai/open-swe/blob/main/agent/utils/slack.py) and [`agent/webapp.py`](https://github.com/langchain-ai/open-swe/blob/main/agent/webapp.py).

The tests execute within the same sandboxed environment that Open SWE provisions for agent tasks, ensuring consistent behavior across development and production contexts.

## Core Integration Test Categories

The test suite organizes validation by feature area, covering the most critical behaviors an internal coding agent must handle when processing external events and managing sandboxed execution.

### Slack Integration Tests ([`tests/test_slack_context.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_slack_context.py))

The Slack integration tests in [`tests/test_slack_context.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_slack_context.py) validate deterministic thread-ID generation and proper message context handling. These tests verify correct selection of context messages based on mentions, proper stripping and replacing of bot mentions, and the formatting of Slack messages for LLM prompts. The underlying logic resides in [`agent/utils/slack.py`](https://github.com/langchain-ai/open-swe/blob/main/agent/utils/slack.py).

### GitHub Webhook Handling ([`tests/test_github_issue_webhook.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_github_issue_webhook.py))

Located in [`tests/test_github_issue_webhook.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_github_issue_webhook.py), these tests ensure reliable processing of GitHub events through [`agent/webapp.py`](https://github.com/langchain-ai/open-swe/blob/main/agent/webapp.py). Validation includes deterministic thread-ID generation for issues, prompt construction that includes issue details, and correct handling of issue-comment events and reaction tokens.

### Sandbox Path Resolution ([`tests/test_sandbox_paths.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_sandbox_paths.py))

The [`tests/test_sandbox_paths.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_sandbox_paths.py) file validates the sandbox path helper in [`agent/utils/sandbox_paths.py`](https://github.com/langchain-ai/open-swe/blob/main/agent/utils/sandbox_paths.py). Tests confirm the use of the provider's working directory, fallback to the user's home directory when the sandbox path is not writable, and proper caching of resolved paths to prevent redundant filesystem operations.

### Comment Processing and Extraction

Open SWE handles comment streams from multiple sources, requiring robust extraction and sanitization logic.

**Recent Comment Extraction** ([`tests/test_recent_comments.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_recent_comments.py)): Validates proper handling of empty comment streams, ignoring bot-generated messages, and collection of comments since the last bot message.

**GitHub Comment Prompt Generation** ([`tests/test_github_comment_prompts.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_github_comment_prompts.py)): Tests the wrapping of external comments with trust sections and sanitization of reserved tags in comment bodies to prevent prompt injection.

### Multimodal Content Extraction ([`tests/test_multimodal.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_multimodal.py))

The [`tests/test_multimodal.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_multimodal.py) suite verifies image URL extraction capabilities in [`agent/utils/multimodal.py`](https://github.com/langchain-ai/open-swe/blob/main/agent/utils/multimodal.py). Tests cover extraction from both markdown and direct links, deduplication and case-insensitive handling, and proper filtering of non-image URLs.

### Safety and Authentication Checks

**Tool-Message Safety** ([`tests/test_ensure_no_empty_msg.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_ensure_no_empty_msg.py)): Validates detection of `commit_and_open_pr` calls, identification of completion-confirmation messages, and proper filtering of non-tool messages to prevent empty agent responses.

**Authentication Sources** ([`tests/test_auth_sources.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_auth_sources.py)): Verifies validation of required environment variables for service authentication, ensuring the agent fails gracefully when credentials are missing.

## Creating and Running Custom Tests

Developers can extend the test suite by adding new pytest files to the `tests/` directory. Each test should import the relevant utilities from the `agent/` package and validate specific function outputs.

Here is an example of adding a unit test for a utility function:

```python

# Example: Adding a new unit test for a utility function

# Save as tests/test_my_utils.py

import pytest
from agent.utils.my_utils import compute_similarity

def test_compute_similarity_returns_expected_value() -> None:
    a = "hello world"
    b = "hello there"
    # Expect a similarity score around 0.8 (implementation-specific)

    assert 0.7 < compute_similarity(a, b) < 0.9

```

Run the suite locally using standard pytest commands:

```bash
pytest -q tests/test_my_utils.py

```

The repository's CI configuration automatically discovers and executes tests in the `tests/` directory alongside existing validations.

## Summary

- Open SWE generates **pytest-based unit tests** that validate core agent behaviors without requiring full-stack integration setups.
- The test suite covers **Slack and GitHub integrations**, **sandbox path resolution**, **multimodal content extraction**, and **safety checks** across eight specialized test files.
- All tests are **self-contained** and execute within the sandboxed environment defined in [`agent/utils/sandbox_paths.py`](https://github.com/langchain-ai/open-swe/blob/main/agent/utils/sandbox_paths.py).
- Developers can extend coverage by adding files to the `tests/` directory using standard pytest patterns.

## Frequently Asked Questions

### What testing framework does Open SWE use?

Open SWE uses **pytest** as its primary testing framework. All tests are written as standard pytest functions and stored in the `tests/` directory at the repository root.

### Are Open SWE tests integration tests or unit tests?

The tests are primarily **unit tests** that validate specific functions and deterministic behaviors. While they verify integration points like Slack and GitHub webhooks, they isolate these components rather than testing full-stack workflows. The CI pipeline handles broader integration testing.

### How do I run the Open SWE test suite locally?

Execute tests using `pytest -q tests/` from the repository root. For individual test files, specify the path directly, such as `pytest -q tests/test_slack_context.py`. The framework automatically picks up any Python files matching the `test_*.py` pattern in the `tests/` directory.

### Where are the test files located in the repository?

All test files reside in the `tests/` directory. Key files include [`tests/test_slack_context.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_slack_context.py) for Slack logic, [`tests/test_github_issue_webhook.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_github_issue_webhook.py) for GitHub processing, and [`tests/test_sandbox_paths.py`](https://github.com/langchain-ai/open-swe/blob/main/tests/test_sandbox_paths.py) for filesystem validation.