# How to Use fit_from_preprocessed for Custom TabPFN Training Loops

> Learn how to use fit_from_preprocessed for custom TabPFN training. Bypass preprocessing, feed PyTorch tensors, and enable gradient accumulation, mixed precision, and custom loss functions.

- Repository: [Prior Labs/TabPFN](https://github.com/PriorLabs/TabPFN)
- Tags: how-to-guide
- Published: 2026-05-06

---

**The `fit_from_preprocessed` method in the PriorLabs/TabPFN repository lets you bypass automatic data preprocessing and feed pre-normalized PyTorch tensors directly into the model, enabling fully custom training loops with gradient accumulation, mixed precision, and bespoke loss functions.**

The `fit_from_preprocessed` method exposes the underlying PyTorch machinery of `TabPFNClassifier` and `TabPFNRegressor`, allowing advanced users to implement custom TabPFN training loops that go beyond the standard sklearn-compatible API. By supplying already-preprocessed tensors, you retain complete control over optimization steps, learning rate schedules, and hardware acceleration while leveraging TabPFN's pretrained transformer backbone for tabular data.

## What fit_from_preprocessed Does

`fit_from_preprocessed` is a specialized fine-tuning entry point exposed on both `TabPFNClassifier` and `TabPFNRegressor` objects. Unlike the standard `.fit()` method— which automatically handles categorical encoding, normalization, and missing value imputation—this method expects **already-preprocessed tensors** as input.

Under the hood, as implemented in [`src/tabpfn/finetuning/finetuned_base.py`](https://github.com/PriorLabs/TabPFN/blob/main/src/tabpfn/finetuning/finetuned_base.py), the method performs three core operations:

1. **Creates a `TorchPreprocessingPipeline`** that holds the encoder steps required for the model's target processing, keeping these frozen while allowing the transformer backbone to update.
2. **Wraps input tensors into a `torch.utils.data.TensorDataset`** and optionally constructs a `DataLoader` for mini-batch iteration.
3. **Executes a standard PyTorch training step** on the pretrained TabPFN backbone, giving you full access to gradients, optimizers, and device placement.

This architecture lives in the `finetuning` sub-package, deliberately separated from the inference-only code path to provide maximum flexibility for research and production fine-tuning.

## When to Implement a Custom Training Loop

Use `fit_from_preprocessed` when you require granular control over the optimization process that the high-level API cannot provide:

- **Mixed-precision training** – Implement `torch.autocast` or manual AMP control to reduce memory footprint and accelerate training on modern GPUs.
- **Gradient accumulation** – Process effective batch sizes larger than GPU memory allows by splitting preprocessed batches across multiple forward passes.
- **Custom loss functions** – Apply quantile loss, pinball loss, or add regularization terms directly to the encoder weights during backpropagation.
- **Early stopping and checkpointing** – Insert validation loops after every epoch or implement sophisticated convergence criteria that the default API does not expose.
- **Non-standard data pipelines** – Bypass built-in preprocessing when you have already performed domain-specific feature engineering, dimensionality reduction, or custom missing-value handling that must be preserved.

## Step-by-Step Implementation Guide

The following pattern demonstrates a complete custom training loop using mixed-precision training on a `TabPFNRegressor`:

```python
import torch
from tabpfn import TabPFNRegressor, TabPFNClassifier
from tabpfn.constants import ModelVersion

# 1. Load a default model (or instantiate a custom architecture)

reg = TabPFNRegressor.create_default_for_version(ModelVersion.V2_5)

# 2. Prepare already-preprocessed tensors

#    (e.g., after your own scaling, categorical encoding, etc.)

X_tensor = torch.randn(500, 30, device="cuda")    # shape: (n_samples, n_features)

y_tensor = torch.randn(500, 1, device="cuda")    # shape: (n_samples, 1)

# 3. Optional: Create a DataLoader for mini-batching

batch_size = 64
train_loader = torch.utils.data.DataLoader(
    torch.utils.data.TensorDataset(X_tensor, y_tensor),
    batch_size=batch_size,
    shuffle=True,
)

# 4. Configure optimizer and loss function

optimizer = torch.optim.Adam(reg.parameters(), lr=1e-4)
criterion = torch.nn.MSELoss()

# 5. Custom training loop with automatic mixed precision

scaler = torch.cuda.amp.GradScaler()
n_epochs = 5

for epoch in range(n_epochs):
    reg.train()
    for xb, yb in train_loader:
        optimizer.zero_grad()
        
        # Mixed-precision context

        with torch.autocast(device_type=xb.device.type, enabled=True):
            preds = reg.forward_from_preprocessed(xb)   # Low-level forward pass

            loss = criterion(preds, yb)
        
        # Scaled backpropagation

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
    
    print(f"Epoch {epoch+1}/{n_epochs} – Loss: {loss.item():.4f}")

# 6. Standard inference after custom training

reg.eval()
with torch.no_grad():
    y_pred = reg.predict(X_tensor.cpu().numpy())

```

**Key implementation details:**
- **`forward_from_preprocessed`** is the low-level entry point that accepts preprocessed feature tensors and returns raw model outputs.
- The **preprocessing pipeline remains frozen** during this loop, which is why input tensors must exactly match the expected format (scaled numeric features, encoded categoricals).
- Because `TabPFNRegressor` inherits from `torch.nn.Module`, standard PyTorch optimizers, schedulers, and AMP utilities work without modification.

## Source Code Reference

The fine-tuning infrastructure is organized across these specific files in the PriorLabs/TabPFN repository:

- **[`src/tabpfn/finetuning/finetuned_base.py`](https://github.com/PriorLabs/TabPFN/blob/main/src/tabpfn/finetuning/finetuned_base.py)** – Implements `fit_from_preprocessed`, constructs `TensorDataset` objects, and manages the optimizer step execution.
- **[`src/tabpfn/base.py`](https://github.com/PriorLabs/TabPFN/blob/main/src/tabpfn/base.py)** – Defines the base class for classifiers and regressors, providing the common forward path used by fine-tuning routines.
- **[`src/tabpfn/finetuning/data_util.py`](https://github.com/PriorLabs/TabPFN/blob/main/src/tabpfn/finetuning/data_util.py)** – Contains helper utilities for converting NumPy or pandas data into the torch tensor format expected by `fit_from_preprocessed`.
- **[`src/tabpfn/finetuning/_torch_compat.py`](https://github.com/PriorLabs/TabPFN/blob/main/src/tabpfn/finetuning/_torch_compat.py)** – Compatibility shims for older PyTorch versions handling changes in `torch.nn.functional`.
- **[`examples/finetune_regressor.py`](https://github.com/PriorLabs/TabPFN/blob/main/examples/finetune_regressor.py)** – Complete working example demonstrating data preparation through custom training loop execution.
- **[`examples/finetune_classifier.py`](https://github.com/PriorLabs/TabPFN/blob/main/examples/finetune_classifier.py)** – Equivalent implementation for classification tasks.

## Critical Considerations

When implementing custom loops with `fit_from_preprocessed`, observe these constraints to avoid runtime errors:

- **Tensor shapes matter.** Input tensors must conform exactly to the model's expected dimensions: `(batch_size, n_features)` for `X` and `(batch_size, 1)` for regression targets. Classification targets require specific encoding handled by the frozen `MulticlassClassificationTargetEncoderStep`.
- **Device consistency.** Ensure both the model and input tensors reside on the same device. Call `reg.to(device)` before training to move the pretrained backbone to CUDA, MPS, or CPU.
- **Learning rate scheduling.** While the example uses Adam, you can attach any `torch.optim.lr_scheduler` after optimizer initialization to adjust learning rates dynamically.
- **Model persistence.** After fine-tuning, save the fitted state using `save_fitted_tabpfn_model(reg, "path/to/model.tabpfn_fit")` and reload later with `load_fitted_tabpfn_model` to preserve your custom training results.

## Summary

- **`fit_from_preprocessed`** bypasses automatic preprocessing in TabPFN, accepting raw PyTorch tensors for maximum training flexibility.
- The method enables **custom training loops** with gradient accumulation, mixed precision, and bespoke loss functions while keeping the preprocessing pipeline frozen.
- Implementation requires careful attention to **tensor shapes** and **device placement**, with core logic residing in [`src/tabpfn/finetuning/finetuned_base.py`](https://github.com/PriorLabs/TabPFN/blob/main/src/tabpfn/finetuning/finetuned_base.py).
- Standard PyTorch workflows apply, allowing integration of any optimizer, scheduler, or training technique compatible with `torch.nn.Module`.

## Frequently Asked Questions

### What is the difference between fit() and fit_from_preprocessed?

The standard `.fit()` method handles end-to-end training including categorical encoding, normalization, and missing value imputation automatically. In contrast, `fit_from_preprocessed` expects you to supply already-encoded tensors and provides no automatic preprocessing, instead exposing the raw PyTorch training interface for custom optimization loops.

### Do I need to disable preprocessing when using fit_from_preprocessed?

Yes, you must supply data that has already been preprocessed to match TabPFN's expected format. The method keeps the internal `TorchPreprocessingPipeline` frozen, so your tensors must reflect the encoding, scaling, and missing-value handling that the model expects, or you must fine-tune the pipeline separately before calling this method.

### Can I use fit_from_preprocessed for both classification and regression?

Yes, both `TabPFNClassifier` and `TabPFNRegressor` expose `fit_from_preprocessed`. The interface remains consistent across task types, though classification requires targets encoded according to the model's `MulticlassClassificationTargetEncoderStep`, while regression expects continuous values of shape `(n_samples, 1)`.

### How do I save a model trained with fit_from_preprocessed?

After completing your custom training loop, persist the fine-tuned model using the utility function `save_fitted_tabpfn_model(model, "filename.tabpfn_fit")`. To restore the model later, use `load_fitted_tabpfn_model("filename.tabpfn_fit")`, which restores both the trained weights and the preprocessing state.