# How Outfancy Efficiently Handles Large Datasets with Thousands of Rows

> Discover how Outfancy efficiently handles large datasets by sampling rows and using memory-efficient streaming for fast, resource-light rendering. Optimize your data processing.

- Repository: [Carlos A. Planchón/outfancy](https://github.com/carlosplanchon/outfancy)
- Tags: performance
- Published: 2026-02-26

---

**Outfancy processes massive tables by sampling only a subset of rows to infer layout decisions, then streams each row independently through memory-efficient rendering loops that avoid building gigantic intermediate strings.**

Outfancy is a Python library designed for rendering formatted terminal tables. When handling large datasets with many rows, the library leverages specialized sampling algorithms and streaming architectures to ensure that both CPU usage and memory consumption remain constant regardless of dataset size.

## Dual Engine Architecture for Scalable Rendering

Outfancy implements two distinct rendering engines to handle different data scales. The system automatically optimizes for large datasets through intelligent truncation and sampling mechanisms.

### Table Engine with Maximum Row Limits

The general-purpose **`Table`** class in [`outfancy/table.py`](https://github.com/carlosplanchon/outfancy/blob/main/outfancy/table.py) controls memory usage through the `self.maximum_number_of_rows` parameter. This setting allows the library to truncate output when the terminal cannot display the full dataset. All expensive calculations—including column width determination, data type detection, and priority ordering—are performed **once** before the rendering loop begins. The engine then processes lists of rows that have already been stripped to the required width, never operating on the entire dataset simultaneously.

### LargeTable Engine with Intelligent Sampling

For datasets containing thousands of rows, **`LargeTable`** provides optimized performance through statistical sampling. The engine analyzes only the first `self.rows_to_analyze` rows (defaulting to **100**) to infer column widths and priorities, avoiding an expensive O(N) scan of the entire dataset. After this initial analysis, the measurements are **re-used** for every subsequent row, ensuring the analysis cost does not scale with the number of rows.

## Sampling-Based Column Width Calculation

The efficiency of `LargeTable` begins with its selective data analysis. In the `render` method at lines 55-60 of [`outfancy/table.py`](https://github.com/carlosplanchon/outfancy/blob/main/outfancy/table.py), the engine slices the input data to create a representative sample:

```python
if len(data) > self.rows_to_analyze:
    data_to_analyze = data[:self.rows_to_analyze]
else:
    data_to_analyze = data

```

This sample powers the **`assign_column_width`** method (lines 91-126), which computes maximum string lengths and assigns widths respecting the terminal's `screen_x` dimension. The calculation incorporates a **`self.corrector`** value (default `-2`) to adjust for off-by-one errors that could cause unnecessary line wrapping in constrained terminals.

## Memory-Efficient Per-Row Streaming

After determining column layouts, `LargeTable.render` processes data through a streaming loop that keeps memory usage proportional to a single row. At lines 84-124 in [`outfancy/table.py`](https://github.com/carlosplanchon/outfancy/blob/main/outfancy/table.py), each row is rendered independently through three distinct phases:

```python
for x_row in range(len(data)):
    rearranged_row = self.rearrange_row(row=data[x_row], order=order)
    frame_lines = self.table.generate_table_frames(
        rearranged_data=[rearranged_row],
        width=width,
        maximum=maximum,
        screen_y=screen_y,
    )
    pre_table = self.table.generate_pre_table(
        frame_lines=frame_lines,
        separator=separator,
        row_separator=row_separator,
    )
    output_lines.append(pre_table)

```

This architecture avoids constructing a massive intermediate string representation. Instead, individual row strings are appended to `output_lines`, and only at line 224 does the method perform a single join operation: `return '\n'.join(output_lines)`.

## Terminal Optimization Utilities

Outfancy maintains alignment accuracy through specialized helper functions in [`outfancy/widgets.py`](https://github.com/carlosplanchon/outfancy/blob/main/outfancy/widgets.py). The **`printed_length`** function (lines 99-103) strips ANSI color codes before measuring string length, preventing width miscalculations in colored output. Additionally, the **`compress_list`** utility (lines 20-33) keeps priority lists compact when columns are dropped, saving both CPU cycles and memory during width redistribution calculations.

## Practical Examples for Large Dataset Rendering

### Rendering 10,000 Rows with LargeTable

```python
from outfancy.table import LargeTable
from outfancy.example_dataset import dataset

# Create a massive dataset (approximately 10,000 rows)

big_data = dataset * 770

lt = LargeTable()
lt.set_rows_to_analyze(200)    # Increase sample size for better accuracy

lt.set_corrector(-1)            # Adjust for specific terminal characteristics

# Render efficiently without memory exhaustion

output = lt.render(big_data)
print(output)

```

This call samples only the first 200 rows to compute layouts, then streams all 10,000 rows through the per-row rendering pipeline.

### Limiting Output with Table Pagination

```python
from outfancy.table import Table
from outfancy.example_dataset import dataset

tbl = Table()
tbl.set_maximum_number_of_rows(30)   # Hard limit on output rows

tbl.set_check_data(True)              # Enable integrity validation

# Process huge input but display only 30 rows

print(tbl.render(dataset * 100))

```

### Streaming Single Rows with Oneline

```python
from outfancy.table import Oneline
from outfancy.example_dataset import dataset

liner = Oneline()
liner.set_maximum_number_of_rows(1)   # One row per render call

for row in dataset:
    print(liner.render(row))          # Instantaneous per-row output

```

The `Oneline` class internally creates a `Table` instance with a row limit of 1, providing a lightweight interface for streaming applications.

## Summary

- **Sampling reduces complexity**: `LargeTable` analyzes only 100 rows by default to determine column widths, avoiding O(N) scans of large datasets.
- **Streaming conserves memory**: The rendering loop processes one row at a time through `rearrange_row`, `generate_table_frames`, and `generate_pre_table`, keeping memory usage proportional to single-row width rather than total dataset size.
- **Width calculations are reused**: Column measurements from the sample are applied to all subsequent rows without recalculation.
- **Output is built incrementally**: The library appends individual row strings to a list and performs a single final join, avoiding massive intermediate string concatenation.
- **Terminal-aware adjustments**: The `corrector` parameter and `printed_length` utility ensure accurate width calculations across different terminal environments and color configurations.

## Frequently Asked Questions

### How does Outfancy prevent memory exhaustion with datasets containing 10,000+ rows?

Outfancy prevents memory exhaustion through the `LargeTable` engine's sampling and streaming architecture. By analyzing only the first `rows_to_analyze` rows (default 100) to determine column widths, the library avoids loading metadata for the entire dataset into memory. The rendering loop then processes each row independently, storing only the final string representation of individual rows before joining them at the end.

### What is the primary difference between Table and LargeTable when handling large datasets?

The **`Table`** class truncates output using `maximum_number_of_rows` and performs full-data analysis, making it suitable for smaller datasets where complete analysis is feasible. The **`LargeTable`** class specifically optimizes for thousands of rows by sampling a subset for analysis, then re-using those measurements across all rows. This allows `LargeTable` to maintain consistent memory usage regardless of whether the dataset contains 1,000 or 100,000 rows.

### Does increasing the rows_to_analyze parameter improve rendering accuracy?

Increasing `rows_to_analyze` can improve column width accuracy for datasets with highly variable row lengths, as the sample better represents the full data distribution. However, the default value of 100 provides sufficient accuracy for most structured data while maintaining the performance benefits of sampling. You can adjust this value using `set_rows_to_analyze()` when working with irregular data patterns.

### How does Outfancy handle ANSI color codes without breaking column alignment?

According to [`outfancy/widgets.py`](https://github.com/carlosplanchon/outfancy/blob/main/outfancy/widgets.py), the **`printed_length`** function (lines 99-103) strips ANSI escape sequences before calculating string lengths. This ensures that color codes do not contribute to width calculations, preventing misalignment issues that typically occur when terminal formatting codes are counted as visible characters.