how-to-guide

How the html_unescape Parameter Handles HTML Entities in BetterHTMLChunking

February 26, 2026 carlosplanchon/betterhtmlchunking ↗

The html_unescape parameter is a boolean flag that controls whether HTML entities like < and & are decoded into literal characters (<, &) before the DOM parsing and chunking pipeline processes the input content.

The html_unescape parameter is a critical configuration option in the BetterHTMLChunking library (carlosplanchon/betterhtmlchunking) that determines how encoded characters are processed during initial content ingestion. When enabled, this parameter ensures that HTML entities are converted to their literal equivalents, allowing downstream chunking algorithms to work with clean markup rather than encoded strings.

What is the html_unescape Parameter?

The html_unescape parameter is a boolean attribute of the DomRepresentation class that defaults to True. It governs whether the input HTML string undergoes entity decoding via Python's standard library html.unescape() function before being stored and processed. This setting affects how special characters appear in the final chunked output and influences length calculations throughout the pipeline.

Source Code Location and Default Configuration

In the BetterHTMLChunking source code, the html_unescape attribute is defined in betterhtmlchunking/main.py at lines 54-57 using the attrs library:

html_unescape: bool = attrs.field(
    validator=type_validator(),
    default=True
)

This declaration establishes True as the default behavior, meaning HTML entities are automatically decoded unless explicitly disabled by the user.

How the Unescaping Mechanism Works

The actual entity decoding occurs during the __attrs_post_init__ method of the DomRepresentation class (lines 71-75 in betterhtmlchunking/main.py). When html_unescape is True, the raw HTML content passes through html.unescape():

if self.html_unescape:
    self.website_code = html.unescape(self.website_code)

This standard library function converts common HTML entities such as < to <, > to >, & to &, and " to ". The transformation happens in-place, replacing the encoded string stored in self.website_code with its decoded equivalent before any DOM parsing begins.

Impact on the Chunking Pipeline

The state of the html_unescape parameter directly influences downstream processing in several critical ways:

Node Representation Accuracy: When unescaping is enabled, length calculations performed by ReprLengthComparisionBy.HTML_LENGTH count the actual characters rather than the longer entity strings. This produces more accurate size estimates for chunking boundaries.
Final Output Quality: Chunks rendered through the RenderSystem contain literal characters rather than encoded entities, resulting in cleaner HTML output that is easier to read and process by subsequent systems.
Preservation of Original Encoding: When set to False, the pipeline operates on the raw entity-encoded strings, preserving the exact input format throughout the chunking process. This is useful when maintaining entity encoding is required for downstream XML processors or specific security policies.

Practical Configuration Examples

Enabling Unescaping (Default Behavior)

By default, html_unescape is True, so entities are automatically decoded:

from betterhtmlchunking.main import DomRepresentation
from betterhtmlchunking.tree_regions_system import ReprLengthComparisionBy

raw_html = "<html><body><p>Price: &lt;$100&gt; &amp; tax</p></body></html>"

dom = DomRepresentation(
    MAX_NODE_REPR_LENGTH=200,
    website_code=raw_html,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH
)

# Entities are decoded: "<$100> & tax"

assert "<$100>" in dom.website_code
assert "&lt;$100&gt;" not in dom.website_code

Disabling Unescaping

To preserve HTML entities in their encoded form, explicitly set the parameter to False:

dom = DomRepresentation(
    MAX_NODE_REPR_LENGTH=200,
    website_code=raw_html,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
    html_unescape=False      # turn it off

)

# Original encoding is preserved

assert "&lt;$100&gt;" in dom.website_code
assert "<$100>" not in dom.website_code

Command-Line Interface Usage

The betterhtmlchunking CLI exposes this parameter via the --html-unescape flag:


# Disable unescaping via CLI

betterhtmlchunking --html-unescape false input.html

# Enable unescaping (default)

betterhtmlchunking input.html

Internally, the CLI passes this boolean value directly to the DomRepresentation constructor in betterhtmlchunking/cli.py.

Summary

The html_unescape parameter is a boolean attribute of the DomRepresentation class defined in betterhtmlchunking/main.py that defaults to True.
When enabled, it invokes Python's html.unescape() during __attrs_post_init__ to convert HTML entities like <, >, and & into their literal characters.
This affects downstream chunking accuracy by ensuring length calculations and rendered output use actual characters rather than encoded entity strings.
Users can disable the parameter via the constructor or CLI (--html-unescape false) to preserve original entity encoding throughout the pipeline.

Frequently Asked Questions

What happens if html_unescape is set to False?

When html_unescape is set to False, the input HTML string remains in its original entity-encoded form throughout the entire chunking pipeline. The DomRepresentation class skips the html.unescape() call in __attrs_post_init__, meaning entities like < and & persist into the DOM parsing and rendering stages. This preserves the exact input format but may cause length calculations to count entity strings rather than the characters they represent.

Does html_unescape affect the final output of chunked HTML?

Yes, the html_unescape parameter directly influences the content of rendered HTML chunks. When enabled (the default), entities are decoded before processing, so the RenderSystem outputs chunks containing literal characters like < and &. When disabled, the encoded entities remain in the final output, producing chunks that retain the original < and & strings. This distinction is critical when downstream systems expect either raw HTML or entity-encoded XML.

What HTML entities does the html_unescape parameter decode?

The html_unescape parameter leverages Python's standard library html.unescape() function, which decodes all standard HTML5 entities including named entities (e.g., <, >, &, ", '), decimal numeric entities (e.g., <), and hexadecimal numeric entities (e.g., <). This comprehensive decoding ensures that any valid HTML entity in the input is converted to its corresponding Unicode character before the DOM representation is constructed.

Can I change the html_unescape setting when using the command-line interface?

Yes, the BetterHTMLChunking CLI exposes the html_unescape parameter through the --html-unescape flag. By default, the CLI enables unescaping, but you can disable it by passing --html-unescape false followed by your input file path. This flag is forwarded directly to the DomRepresentation constructor in betterhtmlchunking/cli.py, ensuring consistent behavior between programmatic and command-line usage.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:

curl -s "https://instagit.com/install.md"

Add to your MCP client configuration:

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}

Ask your agent:

"Use Instagit MCP to understand how carlosplanchon/betterhtmlchunking works."

Works with

Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →