# What Model Architectures Does Heretic Support? Dense LLMs and Four MoE Families Explained

> Explore Heretic's model architecture support, including dense transformers and four MoE families like Qwen and Phi-3.5 MoE. Learn what Heretic currently excludes.

- Repository: [Philipp Emanuel Weidmann/heretic](https://github.com/p-e-w/heretic)
- Tags: deep-dive
- Published: 2026-02-19

---

**Heretic supports any dense transformer architecture out-of-the-box and specifically recognizes four distinct Mixture-of-Experts (MoE) families—Qwen 3, Phi-3.5-MoE, and two Granite MoE Hybrid variants—while currently excluding state space models (SSMs), hybrid layers, and novel attention mechanisms.**

Heretic is a Python library designed for model abliteration and introspection of large language models. Understanding what model architectures Heretic supports is essential for researchers working with modern transformer variants beyond standard dense implementations. While the library universally handles dense transformers, its MoE compatibility targets specific implementations found in recent open-weight models.

## Dense Transformer Support

Heretic works automatically with **any dense transformer**, including most multimodal variants that follow standard transformer layer conventions. According to the repository README at lines 66-68, the library is architected to handle dense models without requiring manual configuration or architecture-specific code paths. This universal compatibility applies to popular implementations like Llama, Mistral, and Qwen dense variants.

The detection logic resides in [`src/heretic/model.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/model.py), where the `get_layer_modules` method introspects model layers to identify standard MLP structures versus expert-routed alternatives.

## MoE Architectures Supported by Heretic

Beyond dense transformers, Heretic explicitly implements detection and handling for four specific MoE families. The `get_layer_modules` method in [`src/heretic/model.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/model.py) (lines 351-368) contains hardcoded detection patterns for each variant.

### Qwen 3 MoE Detection

Heretic identifies **Qwen 3** models (e.g., `Qwen3-4B-Instruct-2507`) by searching for an `mlp.experts` iterable inside each transformer layer. When this pattern is detected at lines 351-355, the library extracts individual expert parameters from the iterable container for abliteration analysis.

### Phi-3.5-MoE Detection

For **Phi-3.5-MoE** and architecturally similar models, Heretic checks for a `block_sparse_moe.experts` container at lines 357-360. The detection logic specifically extracts each expert’s `w2` weight matrix to enable per-expert introspection during the abliteration process.

### Granite MoE Hybrid: Shared MLP Attention

Heretic handles the first variant of **Granite MoE Hybrid** architecture by detecting a `shared_mlp.output_linear` module within the attention branch (lines 361-364). This pattern identifies models that employ shared MLP layers alongside expert-routed components.

### Granite MoE Hybrid: Expert MoE Layers

The second Granite variant is detected by finding a `moe.experts` list at lines 365-368. Heretic pulls the `output_linear` parameter from each expert in this list, enabling fine-grained analysis of the expert-specific components distinct from the shared attention mechanisms.

## How Heretic Detects Architecture Types

The automatic architecture detection occurs during model initialization through the `get_layer_modules` method in [`src/heretic/model.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/model.py). This method sequentially checks for the attribute patterns listed above, returning the appropriate module collection for either dense MLPs or specific MoE expert groupings.

No manual configuration is required. When you load a model via `Model(settings)`, Heretic introspects the PyTorch module hierarchy and automatically routes to the correct handler based on the presence of `mlp.experts`, `block_sparse_moe.experts`, or Granite-specific `shared_mlp` and `moe` attributes.

## Usage Examples

### Running Dense Models (Llama 3.1)

The following example demonstrates Heretic running on a standard dense transformer without requiring architecture-specific flags:

```python
from heretic import Model, Settings, Prompt

settings = Settings(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    max_response_length=128,
    device_map="auto",
)
heretic = Model(settings)

# Simple test generation

response = heretic.get_responses([
    Prompt(system="You are a helpful assistant.", user="What is the capital of France?")
])
print(response[0])

```

### Running MoE Models (Qwen 3)

For MoE models, Heretic automatically applies the appropriate detection logic. This example includes optional 4-bit quantization to accommodate larger models on limited VRAM:

```python
from heretic import Model, Settings, Prompt

settings = Settings(
    model="Qwen/Qwen3-4B-Instruct-2507",
    max_response_length=128,
    device_map="auto",
    # optional: enable 4-bit quantization to fit on modest VRAM

    quantization="bnb_4bit",
)
heretic = Model(settings)

# Perform a quick generation to trigger MoE-specific handling

response = heretic.get_responses([
    Prompt(system="You are a helpful assistant.", user="Explain quantum tunnelling in simple terms.")
])
print(response[0])

```

Both snippets trigger the automatic module discovery in `get_layer_modules`, ensuring the correct abliteration logic is applied regardless of whether the underlying architecture is dense or MoE.

## Unsupported Architectures

As documented in the README at lines 66-68, Heretic explicitly does **not** support state space models (SSMs), hybrid layers with non-standard topologies, inhomogeneous transformer layers, or architectures implementing novel attention mechanisms distinct from standard multi-head attention. Attempting to use Heretic with such models (e.g., Mamba, RetNet, or custom experimental hybrids) will likely result in incorrect layer identification or failed abliteration.

## Summary

- Heretic universally supports **dense transformers** including multimodal variants without manual configuration.
- Four specific **MoE families** are explicitly handled: **Qwen 3**, **Phi-3.5-MoE**, and two **Granite MoE Hybrid** configurations (shared MLP attention and expert MoE layers).
- Detection occurs automatically in [`src/heretic/model.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/model.py) via `get_layer_modules` by inspecting layer attributes like `mlp.experts` and `block_sparse_moe.experts` at initialization.
- **State space models (SSMs)** and other exotic architectures are currently unsupported according to README lines 66-68.
- The library supports **quantized MoE inference** through the `Settings` configuration to reduce VRAM requirements.

## Frequently Asked Questions

### Does Heretic support state space models like Mamba or RetNet?

No. According to the README at lines 66-68, Heretic does not yet handle state space models (SSMs), hybrid layers, inhomogeneous layers, or novel attention mechanisms. The library focuses exclusively on dense transformers and specific MoE implementations with standard attention mechanisms.

### How does Heretic automatically detect which MoE architecture a model uses?

Heretic inspects the module structure within each transformer layer through the `get_layer_modules` method in [`src/heretic/model.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/model.py). It checks for specific attribute patterns: `mlp.experts` indicates Qwen 3, `block_sparse_moe.experts` indicates Phi-3.5, and `shared_mlp.output_linear` or `moe.experts` indicate Granite variants. This detection requires no user configuration.

### Can I use Heretic with quantized MoE models?

Yes. Heretic supports quantization options such as `bnb_4bit` through its `Settings` configuration, allowing MoE models like Qwen 3 to run on modest hardware. The quantization is applied during model loading in [`src/heretic/model.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/model.py) before MoE-specific module discovery occurs.

### Will Heretic work with custom or experimental MoE implementations?

Unlikely. Heretic only recognizes the four specific MoE families hardcoded in [`src/heretic/model.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/model.py) lines 351-368. Custom MoE architectures with different module naming conventions, alternative expert routing mechanisms, or non-standard layer structures will not trigger the specialized MoE handlers and may be treated incorrectly as dense models.