How CLI-Anything Estimates Text Size for ASCII and CJK Characters

CLI-Anything estimates text size by counting UTF-8 bytes: 1 byte per ASCII character and 3 bytes per CJK character, with terminal display widths calculated separately using the wcwidth library.

The HKUDS/CLI-Anything repository handles multilingual text in terminal environments by distinguishing between storage size (bytes) and display width (columns). This dual approach ensures accurate file size reporting and proper text alignment when working with mixed ASCII and CJK content.

UTF-8 Byte-Based Size Estimation

CLI-Anything treats text size as the number of UTF-8 bytes required to store a string. The calculation is straightforward: encode the string and measure the resulting bytes.

In unrealinsights/utils/output.py, the core logic uses Python's built-in encode() method:

def estimate_text_size(text: str) -> int:
    """Return the size of *text* in bytes (UTF-8 encoding)."""
    return len(text.encode("utf-8"))

ASCII characters (U+0000–U+007F) occupy a single byte each. CJK characters (U+0800–U+FFFF and other full-width glyphs) require three bytes in UTF-8 encoding.

ASCII Example

>>> from cli_anything.unrealinsights.utils.output import format_size
>>> text = "Hello world!"
>>> size_bytes = len(text.encode('utf-8'))   # 13 bytes → 1 byte per ASCII char

>>> format_size(size_bytes)
'13 B'

CJK Example

>>> cjk = "你好，世界！"                     # 5 CJK characters + punctuation

>>> size_bytes = len(cjk.encode('utf-8'))   # 5 × 3 + 2 = 17 bytes

>>> format_size(size_bytes)
'17 B'

Terminal Display Width with wcwidth

For proper column alignment in terminal output, CLI-Anything uses the wcwidth library to compute display columns separately from byte size. This distinction is critical because CJK characters typically occupy two terminal columns despite requiring three storage bytes.

The implementation in unrealinsights/utils/output.py calls wcwidth.wcwidth() to determine how many columns each character consumes:

>>> from wcwidth import wcwidth
>>> sum(wcwidth(ch) for ch in "Hello")   # ASCII → 5 columns

5
>>> sum(wcwidth(ch) for ch in "你好世界")   # CJK → 2 × 4 = 8 columns

8

Key differences between the two measurements:

• Byte size: Determines storage requirements and file transfer metrics
• Display width: Ensures table columns and progress bars align correctly in the terminal

Integrated Size and Width Reporting

The CLI combines both measurements when presenting text information to users. This pattern appears throughout the codebase, particularly in formatting utilities:

def show_text_info(txt: str):
    byte_len = len(txt.encode('utf-8'))          # byte-size estimation

    col_len = sum(wcwidth(c) for c in txt)       # column-width for terminal

    click.echo(f"Bytes: {byte_len}, Columns: {col_len}")

Windows UTF-8 Handling

For environments where the console code page might corrupt CJK data, the SiYuan CLI component forces UTF-8 output. In siyuan/siyuan_cli.py, the code reconfigures stdout and stderr to preserve multi-byte character integrity:

# UTF-8 enforcement prevents CJK corruption on Windows terminals

This ensures that size calculations and display widths remain consistent across platforms.

Source File Locations

File	Purpose
unrealinsights/utils/output.py	Core format_size() function and wcwidth integration for display width
siyuan/siyuan_cli.py	UTF-8 output enforcement for Windows compatibility
unrealinsights/unrealinsights_cli.py	User-facing size formatting presentation

Summary

• Byte estimation: CLI-Anything uses len(text.encode('utf-8')) to count UTF-8 bytes—1 for ASCII, 3 for CJK
• Display alignment: The wcwidth library provides column counts—1 for ASCII, 2 for CJK
• Implementation location: Primary logic resides in unrealinsights/utils/output.py
• Cross-platform safety: siyuan/siyuan_cli.py forces UTF-8 mode on Windows to prevent character corruption

Frequently Asked Questions

How does CLI-Anything handle mixed ASCII and CJK strings?

CLI-Anything processes each character individually. The encode('utf-8') method automatically applies the correct byte length per character—1 byte for ASCII, 2 bytes for extended Latin, 3 bytes for CJK—producing an accurate total without explicit branching logic.

Why use wcwidth instead of Unicode category detection?

Unicode categories alone cannot determine terminal display width. Characters like combining accents (U+0300) have zero width despite being valid Unicode. The wcwidth library implements the Unicode East Asian Width specification and accounts for terminal-specific rendering rules.

Does the byte size match file system reported sizes?

Yes. Since CLI-Anything uses UTF-8 encoding—the default for most modern systems and explicitly enforced in the codebase—the byte count matches what ls, dir, or file managers display for text content.

Where can I modify the size formatting behavior?

The format_size() function in unrealinsights/utils/output.py controls how byte counts are rendered for user display. Modify this function to change unit thresholds (B, KB, MB) or decimal precision.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}