How to Use fit_from_preprocessed for Custom TabPFN Training Loops

The fit_from_preprocessed method in the PriorLabs/TabPFN repository lets you bypass automatic data preprocessing and feed pre-normalized PyTorch tensors directly into the model, enabling fully custom training loops with gradient accumulation, mixed precision, and bespoke loss functions.

The fit_from_preprocessed method exposes the underlying PyTorch machinery of TabPFNClassifier and TabPFNRegressor, allowing advanced users to implement custom TabPFN training loops that go beyond the standard sklearn-compatible API. By supplying already-preprocessed tensors, you retain complete control over optimization steps, learning rate schedules, and hardware acceleration while leveraging TabPFN's pretrained transformer backbone for tabular data.

What fit_from_preprocessed Does

fit_from_preprocessed is a specialized fine-tuning entry point exposed on both TabPFNClassifier and TabPFNRegressor objects. Unlike the standard .fit() method— which automatically handles categorical encoding, normalization, and missing value imputation—this method expects already-preprocessed tensors as input.

Under the hood, as implemented in src/tabpfn/finetuning/finetuned_base.py, the method performs three core operations:

1. Creates a TorchPreprocessingPipeline that holds the encoder steps required for the model's target processing, keeping these frozen while allowing the transformer backbone to update.
2. Wraps input tensors into a torch.utils.data.TensorDataset and optionally constructs a DataLoader for mini-batch iteration.
3. Executes a standard PyTorch training step on the pretrained TabPFN backbone, giving you full access to gradients, optimizers, and device placement.

This architecture lives in the finetuning sub-package, deliberately separated from the inference-only code path to provide maximum flexibility for research and production fine-tuning.

When to Implement a Custom Training Loop

Use fit_from_preprocessed when you require granular control over the optimization process that the high-level API cannot provide:

• Mixed-precision training – Implement torch.autocast or manual AMP control to reduce memory footprint and accelerate training on modern GPUs.
• Gradient accumulation – Process effective batch sizes larger than GPU memory allows by splitting preprocessed batches across multiple forward passes.
• Custom loss functions – Apply quantile loss, pinball loss, or add regularization terms directly to the encoder weights during backpropagation.
• Early stopping and checkpointing – Insert validation loops after every epoch or implement sophisticated convergence criteria that the default API does not expose.
• Non-standard data pipelines – Bypass built-in preprocessing when you have already performed domain-specific feature engineering, dimensionality reduction, or custom missing-value handling that must be preserved.

Step-by-Step Implementation Guide

The following pattern demonstrates a complete custom training loop using mixed-precision training on a TabPFNRegressor:

import torch
from tabpfn import TabPFNRegressor, TabPFNClassifier
from tabpfn.constants import ModelVersion

# 1. Load a default model (or instantiate a custom architecture)

reg = TabPFNRegressor.create_default_for_version(ModelVersion.V2_5)

# 2. Prepare already-preprocessed tensors

#    (e.g., after your own scaling, categorical encoding, etc.)

X_tensor = torch.randn(500, 30, device="cuda")    # shape: (n_samples, n_features)

y_tensor = torch.randn(500, 1, device="cuda")    # shape: (n_samples, 1)

# 3. Optional: Create a DataLoader for mini-batching

batch_size = 64
train_loader = torch.utils.data.DataLoader(
    torch.utils.data.TensorDataset(X_tensor, y_tensor),
    batch_size=batch_size,
    shuffle=True,
)

# 4. Configure optimizer and loss function

optimizer = torch.optim.Adam(reg.parameters(), lr=1e-4)
criterion = torch.nn.MSELoss()

# 5. Custom training loop with automatic mixed precision

scaler = torch.cuda.amp.GradScaler()
n_epochs = 5

for epoch in range(n_epochs):
    reg.train()
    for xb, yb in train_loader:
        optimizer.zero_grad()
        
        # Mixed-precision context

        with torch.autocast(device_type=xb.device.type, enabled=True):
            preds = reg.forward_from_preprocessed(xb)   # Low-level forward pass

            loss = criterion(preds, yb)
        
        # Scaled backpropagation

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
    
    print(f"Epoch {epoch+1}/{n_epochs} – Loss: {loss.item():.4f}")

# 6. Standard inference after custom training

reg.eval()
with torch.no_grad():
    y_pred = reg.predict(X_tensor.cpu().numpy())

Key implementation details:

• forward_from_preprocessed is the low-level entry point that accepts preprocessed feature tensors and returns raw model outputs.
• The preprocessing pipeline remains frozen during this loop, which is why input tensors must exactly match the expected format (scaled numeric features, encoded categoricals).
• Because TabPFNRegressor inherits from torch.nn.Module, standard PyTorch optimizers, schedulers, and AMP utilities work without modification.

Source Code Reference

The fine-tuning infrastructure is organized across these specific files in the PriorLabs/TabPFN repository:

• src/tabpfn/finetuning/finetuned_base.py – Implements fit_from_preprocessed, constructs TensorDataset objects, and manages the optimizer step execution.
• src/tabpfn/base.py – Defines the base class for classifiers and regressors, providing the common forward path used by fine-tuning routines.
• src/tabpfn/finetuning/data_util.py – Contains helper utilities for converting NumPy or pandas data into the torch tensor format expected by fit_from_preprocessed.
• src/tabpfn/finetuning/_torch_compat.py – Compatibility shims for older PyTorch versions handling changes in torch.nn.functional.
• examples/finetune_regressor.py – Complete working example demonstrating data preparation through custom training loop execution.
• examples/finetune_classifier.py – Equivalent implementation for classification tasks.

Critical Considerations

When implementing custom loops with fit_from_preprocessed, observe these constraints to avoid runtime errors:

• Tensor shapes matter. Input tensors must conform exactly to the model's expected dimensions: (batch_size, n_features) for X and (batch_size, 1) for regression targets. Classification targets require specific encoding handled by the frozen MulticlassClassificationTargetEncoderStep.
• Device consistency. Ensure both the model and input tensors reside on the same device. Call reg.to(device) before training to move the pretrained backbone to CUDA, MPS, or CPU.
• Learning rate scheduling. While the example uses Adam, you can attach any torch.optim.lr_scheduler after optimizer initialization to adjust learning rates dynamically.
• Model persistence. After fine-tuning, save the fitted state using save_fitted_tabpfn_model(reg, "path/to/model.tabpfn_fit") and reload later with load_fitted_tabpfn_model to preserve your custom training results.

Summary

• fit_from_preprocessed bypasses automatic preprocessing in TabPFN, accepting raw PyTorch tensors for maximum training flexibility.
• The method enables custom training loops with gradient accumulation, mixed precision, and bespoke loss functions while keeping the preprocessing pipeline frozen.
• Implementation requires careful attention to tensor shapes and device placement, with core logic residing in src/tabpfn/finetuning/finetuned_base.py.
• Standard PyTorch workflows apply, allowing integration of any optimizer, scheduler, or training technique compatible with torch.nn.Module.

Frequently Asked Questions

What is the difference between fit() and fit_from_preprocessed?

The standard .fit() method handles end-to-end training including categorical encoding, normalization, and missing value imputation automatically. In contrast, fit_from_preprocessed expects you to supply already-encoded tensors and provides no automatic preprocessing, instead exposing the raw PyTorch training interface for custom optimization loops.

Do I need to disable preprocessing when using fit_from_preprocessed?

Yes, you must supply data that has already been preprocessed to match TabPFN's expected format. The method keeps the internal TorchPreprocessingPipeline frozen, so your tensors must reflect the encoding, scaling, and missing-value handling that the model expects, or you must fine-tune the pipeline separately before calling this method.

Can I use fit_from_preprocessed for both classification and regression?

Yes, both TabPFNClassifier and TabPFNRegressor expose fit_from_preprocessed. The interface remains consistent across task types, though classification requires targets encoded according to the model's MulticlassClassificationTargetEncoderStep, while regression expects continuous values of shape (n_samples, 1).

How do I save a model trained with fit_from_preprocessed?

After completing your custom training loop, persist the fine-tuned model using the utility function save_fitted_tabpfn_model(model, "filename.tabpfn_fit"). To restore the model later, use load_fitted_tabpfn_model("filename.tabpfn_fit"), which restores both the trained weights and the preprocessing state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}