How Outfancy Auto-Detects Column Data Types: id, name, date, time, value, and desc

Outfancy automatically detects column data types by sampling up to 10 rows from your dataset and applying heuristics inside Table.check_data_type_list_integrity to classify fields as identifiers, timestamps, or text descriptions based on content patterns.

Outfancy is a Python library for rendering tabular data in terminal environments. When you render a dataset without explicitly providing a data_type_list, the library analyzes your data to determine whether each column represents an id, name, date, time, value, or desc (description). This automatic detection happens in outfancy/table.py and relies on configurable thresholds and pattern matching heuristics.

The Detection Pipeline

The core logic resides in Table.check_data_type_list_integrity (lines 699‑754 of outfancy/table.py). This method examines a representative sample of rows when data_type_list is missing or incomplete.

Sampling Strategy

To maintain performance on large datasets, Outfancy limits analysis to a configurable analysis threshold defined by self.analyze_threshold (default: 10 rows). The method iterates over this subset and converts each field to a string for inspection. If you supply a partial data_type_list, only columns marked with None undergo automatic detection, while specified types remain untouched.

Heuristic Classification

For each sampled field, Outfancy applies the following checks in order:

• Hour detection: Uses widgets.is_complete_hour(field) to validate against %H:%M:%S or %H:%M formats via time.strptime.
• Date detection: Uses widgets.is_date(field) to normalize text and test multiple date formats.
• Numeric detection: Checks field.isdigit() to identify potential id or value columns.
• Long text detection: If the field exceeds self.chk_dlti_num_letters_in_field (default 15 characters) and contains more than self.chk_dlti_pecentage_letters_in_field (default 90%) alphabetic characters, it is classified as name.
• Fallback: Fields that are short or contain mixed content default to desc.

Distinguishing ID from Value

When a field contains only digits, Outfancy performs a sequence analysis to differentiate sequential identifiers from arbitrary numeric values. The algorithm examines neighboring rows in the sample:

• If the next row exists and int(next_field) - int(field) == 1, the column is marked as id.
• If the previous row exists and int(field) - int(prev_field) == 1, this reinforces the id classification.
• Non-consecutive numeric fields or single-row datasets default to value.

Threshold-Based Final Selection

After processing the sample, Outfancy counts occurrences of each temporary type and converts them to percentages. The final column type is the first category to meet its configured threshold (checked in priority order):

Type	Threshold Attribute	Default
date	self.chk_dtli_date	70%
time	self.chk_dtli_time	70%
id	self.chk_dtli_id	50%
value	self.chk_dlti_value	100%
name	self.chk_dlti_name	60%
desc	self.chk_dlti_desc	60%

If no threshold is satisfied, the column falls back to desc. Once complete, the method logs column types auto-detected successfully and returns the finalized list.

Configuration and Threshold Defaults

The detection behavior is governed by attributes initialized in Table.__init__ (lines 129‑146 of outfancy/table.py). You can adjust these thresholds after instantiation to tune sensitivity:

• ID sensitivity: Lower chk_dtli_id (default 50) to catch sparser sequences, or raise it to require nearly perfect sequentiality.
• Name length: Modify chk_dlti_num_letters_in_field (default 15) to change the minimum character count for name classification.
• Sample size: Increase analyze_threshold (default 10) to scan more rows for detection, improving accuracy on heterogeneous columns.

Practical Code Examples

Automatic Detection

Render a dataset without specifying types to trigger auto-detection:

from outfancy import Table
from outfancy.example_dataset import dataset

tbl = Table()
print(tbl.render(dataset))

Overriding Specific Columns

Pass a data_type_list with None for auto-detected columns and explicit types for others:

tbl = Table()

# Force column 3 to desc, keep auto-detection for the rest

custom_types = [None, None, 'desc', None, None, None, 'value', None]
print(tbl.render(dataset, data_type_list=custom_types))

Tuning Detection Thresholds

Make ID detection stricter by requiring 80% consecutive sequences:

tbl = Table()
tbl.chk_dtli_id = 80  # Default is 50

tbl.analyze_threshold = 20  # Scan more rows for better accuracy

print(tbl.render(dataset))

Summary

• Outfancy detects column types automatically when data_type_list is omitted, using Table.check_data_type_list_integrity in outfancy/table.py.
• The algorithm samples 10 rows by default and applies heuristics to identify date, time, id, value, name, and desc fields.
• ID columns are distinguished from value columns by checking for consecutive integer sequences across neighboring rows.
• Final classification uses percentage thresholds (e.g., 70% for dates, 100% for values) with a fallback to desc.
• All thresholds and sample sizes are configurable via attributes on the Table instance.

Frequently Asked Questions

How does Outfancy distinguish between id and value columns?

Outfancy checks if numeric fields form a consecutive sequence. If int(next_field) - int(current_field) == 1 (or the reverse for previous rows), the column is classified as id; otherwise, it becomes value. Single-row datasets default to value since no sequence can be established.

Can I disable automatic type detection in Outfancy?

Yes. Provide a complete data_type_list parameter to Table.render() with no None values. When every column has an explicit type assigned, check_data_type_list_integrity skips analysis entirely and uses your specifications.

What happens if no data type thresholds are met?

If none of the percentage thresholds (70% for date/time, 50% for id, 100% for value, 60% for name/desc) are satisfied during sampling, the column automatically falls back to desc (description), ensuring the renderer always has a valid type assignment.

Which source files handle the detection logic?

The primary logic lives in outfancy/table.py (lines 699‑754) within the check_data_type_list_integrity method. Helper functions for time and date validation reside in outfancy/widgets.py (is_complete_hour and is_date), while default threshold values are defined in the Table class constructor (lines 129‑146).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}