How Anomaly Detection Algorithms in ai-engineering-from-scratch Handle High-Dimensional Data Without Labels

The repository implements three unsupervised detectors—Z-Score, IQR, and a from-scratch Isolation Forest—that use per-feature statistics or random subspace projections to identify outliers in high-dimensional spaces without requiring labeled anomaly examples.

The rohitg00/ai-engineering-from-scratch repository provides educational implementations of core machine learning algorithms built entirely from NumPy. When working with high-dimensional datasets where anomalies are rare and unlabeled, traditional supervised approaches fail. The codebase solves this through fully unsupervised methods that treat each feature independently or use random feature selection to isolate outliers efficiently.

Unsupervised Anomaly Detection Algorithms for High-Dimensional Data

The repository contains three distinct algorithms in phases/02-ml-fundamentals/16-anomaly-detection/code/anomaly_detection.py. Each method operates without labeled training data and scales to high-dimensional inputs through vectorized operations or randomized subspace sampling.

Z-Score Detection Using Per-Feature Statistics

The Z-Score detector computes univariate statistics independently for each dimension. For an input matrix X with shape (n_samples, n_features), the implementation calculates:

mean = X.mean(axis=0)
std = X.std(axis=0)
z_scores = np.abs((X - mean) / std)
scores = z_scores.max(axis=1)

By taking the maximum absolute Z-Score across all dimensions (z.max(axis=1)), the algorithm aggregates outlier information without requiring labels. The axis=0 reduction ensures the computation scales linearly with the number of features, making it suitable for datasets with thousands of dimensions. The zscore_detect function in anomaly_detection.py (lines 4-10) implements this logic using pure NumPy operations.

IQR Detection with Vectorized Quartile Analysis

The IQR (Inter-Quartile Range) method uses robust statistics to handle skewed high-dimensional distributions. The implementation evaluates the 25th and 75th percentiles per feature:

q1 = np.percentile(X, 25, axis=0)
q3 = np.percentile(X, 75, axis=0)
iqr = q3 - q1
lower = q1 - factor * iqr
upper = q3 + factor * iqr

The final anomaly score is the largest deviation from any fence across all dimensions (np.maximum(...).max(axis=1)). Because the computation is fully vectorized over the feature axis using axis=0 reductions, the iqr_detect function handles high-dimensional data efficiently without labels. This implementation appears in lines 14-25 of the source file.

Isolation Forest from Scratch with Random Feature Selection

The Isolation Forest implementation avoids the curse of dimensionality through stochastic subspace sampling. Rather than examining all dimensions simultaneously, each tree randomly selects a single feature per split:

self.feature = self.rng.randint(p)  # p = number of features

self.threshold = self.rng.uniform(
    X[self.feature].min(), 
    X[self.feature].max()
)

This random feature selection strategy means the algorithm naturally scales to high-dimensional data because each split only evaluates one dimension. The anomaly score derives from the average path length across all trees; shorter paths indicate easier isolation and higher anomaly probability. The IsolationForest class (lines 90-133) and the underlying IsolationTree class (lines 37-88) provide the complete implementation without any dependency on labeled data.

Why These Methods Scale to High Dimensions

Three architectural decisions enable these algorithms to handle high-dimensional data without labeled anomalies:

• Vectorized NumPy operations – Both Z-Score and IQR use axis=0 reductions that run in O(N·D) time and remain memory-efficient because they process features in parallel.
• Random subspace projection – The Isolation Forest never examines the full feature space at once. By randomly selecting one feature per split, tree depth remains logarithmic relative to sample size rather than the number of dimensions.
• Distribution-based boundaries – All three methods derive decision boundaries from the data's intrinsic distribution (statistics or random partitions), eliminating the need for labeled anomaly examples during training.

Practical Implementation Example

The following example demonstrates how to apply these detectors to a 50-dimensional synthetic dataset containing 20 unlabeled anomalies:

import numpy as np
from phases.02_ml_fundamentals.16_anomaly_detection.code.anomaly_detection import (
    zscore_detect, iqr_detect, IsolationForest,
)

# Generate high-dimensional data

rng = np.random.RandomState(0)
X_normal = rng.normal(loc=0.0, scale=1.0, size=(500, 50))
X_anomaly = rng.uniform(low=6, high=10, size=(20, 50))
X = np.vstack([X_normal, X_anomaly])

# Z-Score detection (unsupervised)

z_labels, z_scores = zscore_detect(X, threshold=3.0)
print("Z-Score → anomalies:", z_labels.sum())

# IQR detection (unsupervised)

iqr_labels, iqr_scores = iqr_detect(X, factor=1.5)
print("IQR → anomalies:", iqr_labels.sum())

# Isolation Forest from scratch

iso = IsolationForest(n_estimators=200, max_samples=256, seed=42)
iso.fit(X)
iso_scores = iso.anomaly_score(X)
iso_labels = iso_scores > 0.6
print("Isolation Forest → anomalies:", iso_labels.sum())

Notice that the code operates completely without labels—the y_true array is omitted entirely from the detection calls. The Isolation Forest's max_depth automatically sets to ⌈log₂(sample_size)⌉, remaining modest even when p=50 dimensions.

Summary

• Z-Score and IQR detectors use axis=0 vectorized statistics to compute per-feature outliers and aggregate via maximum deviation, requiring no labels and scaling to thousands of dimensions.
• Isolation Forest implements random feature selection per tree split, avoiding the curse of dimensionality by isolating points in single-dimensional subspaces.
• All implementations reside in phases/02-ml-fundamentals/16-anomaly-detection/code/anomaly_detection.py and use only NumPy, making them dependency-free educational tools for unsupervised high-dimensional anomaly detection.

Frequently Asked Questions

How does the Isolation Forest handle the curse of dimensionality?

The Isolation Forest avoids distance-based metrics that degrade in high-dimensional spaces by randomly selecting single features for each split. Because each tree only examines one dimension at a time, the algorithm does not suffer from the "concentration of measure" problem that affects Euclidean distance in high dimensions. The path length depends only on how easily a point can be isolated in random subspaces, not on the total feature count.

Can I use these detectors on sparse high-dimensional datasets?

Yes, though with considerations. The Z-Score and IQR methods work best with dense arrays because they compute means and percentiles across all features. For sparse data, the Isolation Forest is most appropriate because it only touches specific features during splits, naturally respecting sparsity patterns. You may need to modify the random threshold calculation in anomaly_detection.py to handle sparse matrix inputs if using scipy.sparse.

What threshold should I use for the Z-Score and IQR detectors?

For Z-Score, a threshold of 3.0 captures approximately 99.7% of normally distributed data points, marking anything beyond as anomalous. For IQR, the standard factor=1.5 defines mild outliers, while factor=3.0 identifies extreme outliers. These thresholds are statistical conventions, but you should adjust them based on your specific dataset's contamination ratio and business requirements for false positive rates.

Are these implementations suitable for production environments?

These implementations are designed for educational purposes to demonstrate the underlying mechanics of anomaly detection algorithms. While they handle high-dimensional data correctly, production systems should use optimized libraries like scikit-learn or PyOD for performance, input validation, and robust handling of edge cases. The repository code serves as a reference for understanding how these algorithms work under the hood before deploying optimized versions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}