What Tests Does test_skill_artifact_bundles.py Run Against Skill Artifact Bundles?

The test_skill_artifact_bundles.py file in the rohitg00/ai-engineering-from-scratch repository contains 19 comprehensive unit tests that verify secure discovery, validation, installation, catalog generation, and manifest creation for skill-artifact bundles while rigorously defending against symlink attacks and path traversal vulnerabilities.

The rohitg00/ai-engineering-from-scratch repository provides infrastructure for managing AI engineering curriculum artifacts. Within this codebase, scripts/test_skill_artifact_bundles.py implements a hardened test suite that exercises every stage of the bundle lifecycle—from initial filesystem discovery through final manifest generation—ensuring that the install_skills.py module safely handles directory-based skill artifacts without exposing the system to directory traversal or symlink-based attacks.

Discovery Tests

The discovery phase ensures that bundles are correctly identified and that their metadata is accurately populated while rejecting unsafe filesystem configurations.

test_installer_discovers_one_bundle_from_its_skill_entrypoint

This test verifies that the installer correctly discovers a bundle and populates its metadata—including type, name, version, tags, phase/lesson identifiers, source path, and file list—when scanning from the skill entrypoint. Located at lines 56-89, it establishes the baseline for successful bundle identification.

test_bundle_file_lists_share_sorted_posix_order

Deterministic output is critical for reproducible builds. This test (lines 94-118) confirms that both the catalog utility and the installer return file lists in a consistent, POSIX-sorted order, preventing non-deterministic behavior across different filesystems or platform-specific ordering.

test_discovery_rejects_symlinked_bundle_before_reading_skill_file

Security validation begins early. At lines 433-455, this test ensures that a bundle reachable only through a symlinked directory is rejected as unsafe before the system attempts to read any skill configuration files, preventing symlink-based directory traversal attacks during the discovery phase.

test_discovery_ignores_unrecognized_symlinked_file_entries

This test demonstrates robustness against stray symlinks. Located at lines 58-78, it verifies that unrecognized symlinks pointing outside the repository are ignored during discovery, leaving valid bundles untouched and preventing information leakage from external filesystem paths.

Validation and Security Tests

The validation suite focuses on preventing symlink attacks and ensuring atomic safety guarantees during bundle processing.

test_installer_rejects_bundle_symlinks_before_writing_any_artifact

At lines 173-242, this test validates that the installer inspects bundle contents for symlinks before writing any artifacts to disk. If any symlink is detected, the system raises an UnsafeBundleError and aborts before creating files, ensuring no malicious links are established in the target directory.

test_installer_rejects_bundle_file_symlink_swap_at_open_boundary

This test simulates a Time-of-Check to Time-of-Use (TOCTOU) attack where a malicious actor swaps a regular file for a symlink during the open phase. At lines 194-288, it confirms that attempting to open a symlinked file at the boundary raises an immediate UnsafeBundleError, closing the race condition window.

test_installer_rejects_symlinked_layout_parent

Bundle safety extends to parent directory validation. Located at lines 226-294, this test guarantees that a bundle whose immediate parent directory is a symlink is treated as unsafe and rejected, preventing attackers from redirecting bundle contents through parent directory manipulation.

test_installer_rejects_a_bundle_reached_through_an_escaping_parent_symlink

Path escaping represents a critical security vulnerability. At lines 298-322, this test detects bundles that attempt to escape the repository root via parent-directory symlinks (e.g., ../../etc/passwd patterns) and raises UnsafeBundleError before processing continues.

Installation Tests

Installation tests verify that valid bundles are copied correctly while handling edge cases like atomic swaps and naming conflicts.

test_installer_copies_the_complete_bundle_to_one_skill_directory

This functional test (lines 119-158) validates that the installer performs verbatim copies of complete bundle structures, including nested subdirectories, executable scripts, and binary assets, preserving the exact layout from the source to the target skill directory.

test_failed_forced_bundle_swap_restores_the_previous_directory

Atomicity is crucial for reliable updates. At lines 160-199, this test forces a failure during a staged bundle swap operation and verifies that the installer correctly restores the previous directory state, preventing partial or corrupted installations.

test_duplicate_flat_and_bundle_names_choose_the_flat_artifact_once

When naming conflicts occur between flat files and bundles, the system must behave predictably. This test (lines 174-216) confirms that the installer prefers flat artifacts over bundles when names collide, issuing an appropriate warning while ensuring deterministic selection behavior.

Catalog Generation Tests

The catalog system generates curriculum metadata and must correctly represent bundle characteristics while enforcing security boundaries.

test_catalog_surfaces_bundle_metadata_files_and_skill_entrypoint_once

Located at lines 306-395, this test ensures that the catalog correctly records bundle-specific metadata, including the entrypoint path, bundle flag status, complete file list, and resolved filesystem path, surfacing this information exactly once to prevent duplicate entries.

test_catalog_rejects_a_bundle_that_resolves_outside_the_repository

Catalog building must respect repository boundaries. At lines 797-817, this test verifies that attempting to catalog a bundle whose resolved root path escapes the repository root raises an immediate exception, preventing external filesystem enumeration through catalog generation.

Manifest Creation Tests

Manifests provide installation records and must handle filesystem edge cases safely.

test_manifest_includes_bundle_entrypoint_root_and_files

This test (lines 327-366) validates that generated manifests accurately capture the bundle's root directory, entrypoint location, and complete file list after installation, creating a reliable record for verification and uninstallation procedures.

test_manifest_uses_cached_bundle_files_after_install

Performance and consistency require caching. At lines 367-399, this test confirms that manifests use pre-computed file lists cached during installation rather than re-scanning the source bundle, ensuring consistency even if the source bundle changes after installation.

test_manifest_atomically_replaces_symlink_without_touching_target

Atomic replacement prevents race conditions. Located at lines 406-421, this test verifies that when writing a manifest over an existing symlink, the operation replaces the symlink atomically without dereferencing or modifying the symlink's target, preventing accidental corruption of unrelated files.

CLI Safety Tests

The command-line interface provides user-facing operations with comprehensive safety checks.

test_dry_run_rejects_unsafe_bundle_before_previewing_it

Dry-run mode should catch errors early. At lines 405-430, this test confirms that the --dry-run flag aborts immediately upon detecting an unsafe bundle (containing symlinks) before displaying any installation preview, ensuring users receive immediate security feedback.

test_cli_reports_an_unsafe_bundle_without_a_traceback

User experience matters for security errors. This test (lines 560-566) verifies that the CLI prints a concise, user-friendly error message for unsafe bundles without emitting a Python traceback, presenting professional error handling that doesn't expose internal implementation details.

test_cli_rejects_an_artifact_name_that_escapes_the_target

Path traversal through malicious names must be blocked. At lines 709-728, this test checks that artifact names containing path-traversal sequences (such as ../../../etc/critical-file) are rejected with an appropriate error message before any filesystem operations occur.

Practical Code Examples

Discovering Bundles in a Temporary Workspace

When writing custom tooling against the install_skills module, you can discover bundles programmatically using a temporary workspace:

import tempfile
from pathlib import Path
from install_skills import discover_artifacts, ROOT, PHASES_DIR
from unittest.mock import patch

with tempfile.TemporaryDirectory() as tmp:
    root = Path(tmp)
    # Create a bundle under <root>/phases/14-agent-engineering/22-skill-runtime/outputs

    # Patch globals so the installer sees the temporary layout

    with patch.object(ROOT, root), patch.object(PHASES_DIR, root / "phases"):
        bundles = list(discover_artifacts())
        print([b.name for b in bundles])   # → ['my-bundle']

Generating a Manifest After Installation

To generate installation manifests programmatically after processing bundles:

from install_skills import write_manifest, build_plan, apply_plan

target = Path("/opt/skills")
plan = build_plan(bundles, target, layout="skills", force=False)
apply_plan(plan)
manifest_path = write_manifest(target, bundles, layout="skills")
print(manifest_path.read_text())

Summary

• Comprehensive Lifecycle Coverage: test_skill_artifact_bundles.py validates 19 distinct behaviors across discovery, validation, installation, catalog generation, and manifest creation.
• Security-First Design: The suite prioritizes defense-in-depth against symlink attacks, TOCTOU race conditions, and path traversal attempts at every lifecycle stage.
• Atomic Operations: Tests verify that failed installations roll back cleanly and that manifest writes occur atomically without affecting symlink targets.
• Deterministic Behavior: File lists maintain POSIX-sorted order, and duplicate name resolution follows predictable flat-file precedence rules.

Frequently Asked Questions

What is a skill-artifact bundle in the ai-engineering-from-scratch repository?

A skill-artifact bundle is a directory-based collection of files—including scripts, binaries, and metadata—that represents a complete lesson or phase artifact in the AI engineering curriculum. Unlike flat files, bundles maintain internal directory structures and are processed as atomic units by the install_skills.py module, with special handling for entrypoint detection and metadata extraction.

Why does test_skill_artifact_bundles.py focus heavily on symlink security?

According to the source code analysis, the test suite emphasizes symlink validation because bundles are extracted to filesystem locations where malicious symlinks could redirect writes to sensitive system files (such as /etc/passwd or SSH keys). The tests implement defense-in-depth by checking for symlinks in bundle contents, parent directories, and entry paths before any write operations occur, preventing both direct attacks and TOCTOU race conditions during installation.

How does the installer handle naming conflicts between bundles and flat files?

When a bundle and a flat file share the same name in the source directory, the installer prefers the flat artifact and emits a warning, as verified by test_duplicate_flat_and_bundle_names_choose_the_flat_artifact_once. This deterministic behavior prevents ambiguity while allowing curriculum authors to override bundle content with single-file hotfixes when necessary.

What happens when the dry-run mode encounters an unsafe bundle?

The CLI's dry-run mode performs full validation before displaying any preview output. As tested in test_dry_run_rejects_unsafe_bundle_before_previewing_it, if a bundle contains symlinks or escapes the repository root, the command aborts immediately with a concise error message without showing the installation plan, ensuring users receive immediate security feedback without exposing target path information for unsafe artifacts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}