n_estimators in TabPFN: Controlling Ensemble Aggregation Through Prompt Tuning

The n_estimators parameter in TabPFN specifies exactly how many independent forward passes (prompts) are executed during inference, with each pass receiving a uniquely preprocessed view of the data and their predictions being aggregated via averaging to reduce variance and improve calibration.

TabPFN is a large pretrained transformer for tabular data maintained in the PriorLabs/TabPFN repository. Unlike traditional tree-based ensembles, this model treats inference as an aggregation of multiple prompt-tuned configurations. The n_estimators argument controls the size of this ensemble, directly impacting prediction stability and computational cost.

How n_estimators Drives the Prompt Ensemble

TabPFN leverages a prompt-ensemble strategy rather than training multiple models. During inference, each of the n_estimators members receives a slightly different view of the input data—such as random feature subsampling, varied class index ordering, or different target transformations. By averaging predictions across these diverse prompts, the model reduces stochastic variance and enhances calibration without requiring multiple trained models.

Ensemble Configuration Generation

The ensemble members are created by generator functions in src/tabpfn/preprocessing/ensemble.py. The functions generate_regression_ensemble_configs (lines 898-904) and generate_classification_ensemble_configs (lines 882-889) produce exactly n_estimators distinct configuration objects.

Each configuration defines a unique preprocessing pipeline handled by the TabPFNEnsemblePreprocessor class (lines 41-75). This includes feature subsampling, fingerprinting, and target transformation specific to that ensemble member.

Prediction Aggregation Logic

During prediction, TabPFN executes n_estimators forward passes and aggregates the results. In src/tabpfn/regressor.py, the predict method (lines 982-1002) initializes accumulated_logits and loops over the ensemble, summing outputs from each forward pass.

The averaging behavior depends on the average_before_softmax parameter:

• When True, logits are averaged before applying softmax (around line 994), which improves calibration for multi-class problems
• When False, probabilities are averaged after the softmax transformation

The final result divides the accumulated logits by n_estimators to produce the mean prediction.

Fit-Time Preprocessing Integration

The same n_estimators value determines preprocessing during training. The fit_from_preprocessed method asserts that len(ensemble_configs) == self.n_estimators (lines 708-710), ensuring consistency between training and inference. Even when handling constant targets in _handle_constant_target, the code asserts n_estimators > 0 (line 1004) to maintain shape requirements.

Practical Code Examples

Configuring Ensemble Size for Regression

Increase n_estimators to reduce prediction variance on noisy regression tasks:

from tabpfn import TabPFNRegressor
import sklearn.datasets

X, y = sklearn.datasets.make_regression(n_samples=200, n_features=20)

# Initialize with 16 ensemble members instead of the default 8

model = TabPFNRegressor(n_estimators=16, device="cpu")
model.fit(X, y)

# Returns mean of ensemble predictions (averaged logits)

pred = model.predict(X)

Key implementation details: The constructor stores self.n_estimators = n_estimators (line 444 in src/tabpfn/regressor.py). During fit, generate_regression_ensemble_configs creates exactly n_estimators configs (line 692).

Classification with Logit Averaging

For classification, control whether averaging happens before or after the softmax:

from tabpfn import TabPFNClassifier
import sklearn.datasets

X, y = sklearn.datasets.make_classification(
    n_samples=150, n_features=10, n_classes=3, n_informative=5
)

# Average logits before softmax for better calibration

clf = TabPFNClassifier(
    n_estimators=12,
    average_before_softmax=True,
    device="cuda"
)
clf.fit(X, y)
proba = clf.predict_proba(X)

This triggers the code path at lines 994-1014 in src/tabpfn/regressor.py, where the ensemble logits are averaged before probability conversion.

Inspecting Generated Configurations

Examine exactly what n_estimators creates under the hood:

from tabpfn.preprocessing import generate_classification_ensemble_configs

configs = generate_classification_ensemble_configs(
    num_estimators=5,
    add_fingerprint_feature=False,
    feature_shift_decoder="none",
    polynomial_features=0,
    preprocessor_configs=[],
    target_transforms=[],
    random_state=42,
    num_models=1,
    outlier_removal_std=None,
)

print(f"Generated {len(configs)} ensemble configurations")  # Output: 5

This function in src/tabpfn/preprocessing/ensemble.py (lines 882-910) returns a list of EnsembleConfig objects defined in src/tabpfn/preprocessing/configs.py.

Performance Characteristics and Defaults

The default n_estimators value is defined in src/tabpfn/constants.py and typically set to 8, offering a balance between variance reduction and computational efficiency.

• Higher values (16-32): Reduce variance and improve calibration, particularly beneficial for small datasets or high-dimensional classification, but increase inference time linearly
• Lower values (1-4): Faster inference but higher prediction variance and potential calibration issues

Each increment of n_estimators executes one additional transformer forward pass, making the parameter the primary control for the speed/accuracy trade-off in TabPFN.

Summary

• n_estimators defines the number of prompt-based ensemble members in TabPFN's aggregation scheme
• Configurations are generated by generate_*_ensemble_configs in src/tabpfn/preprocessing/ensemble.py
• Each member receives unique preprocessing via TabPFNEnsemblePreprocessor (feature subsampling, transformations)
• Predictions aggregate via averaging in src/tabpfn/regressor.py (lines 982-1002), either pre-softmax or post-softmax based on average_before_softmax
• The default value of 8 provides a practical balance, while values above 16 suit high-variance datasets at the cost of linearly increased compute time

Frequently Asked Questions

What does n_estimators control in TabPFN?

The n_estimators parameter controls the size of the prompt ensemble—the number of distinct preprocessing configurations and independent forward passes executed during inference. According to the PriorLabs/TabPFN source code, this determines how many "views" of the data are aggregated via averaging to produce the final prediction, directly reducing model variance.

How does increasing n_estimators affect inference speed?

Increasing n_estimators linearly increases compute time because TabPFN executes one complete transformer forward pass per ensemble member. While the default of 8 offers efficient inference, setting n_estimators=32 requires four times the computation. The parameter represents the primary scalability bottleneck in the predict loop of src/tabpfn/regressor.py.

What is the difference between average_before_softmax=True and False?

When average_before_softmax=True (lines 994-1014 in src/tabpfn/regressor.py), the model averages raw logits across all n_estimators members before applying the softmax function, which typically improves calibration for multi-class problems. When False, each member produces probabilities that are then averaged, which can sharpen predictions but may reduce calibration on uncertain samples.

Where are the ensemble configurations generated?

The ensemble configurations are generated in src/tabpfn/preprocessing/ensemble.py by generate_classification_ensemble_configs (lines 882-889) or generate_regression_ensemble_configs (lines 898-904). These functions produce exactly n_estimators configuration objects that define the per-member preprocessing pipelines, including feature subsampling and target transformations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}