How the RenderSystem Generates Both HTML and Plain Text Outputs in Parallel

The RenderSystem class in betterhtmlchunking processes each region of interest (ROI) by simultaneously applying BeautifulSoup's prettify() for HTML and parsel_text.get_bs4_soup_text() for plain text, storing results in parallel dictionaries before aggregating them into final per-ROI chunks.

The RenderSystem is a core component of the carlosplanchon/betterhtmlchunking repository that transforms identified tree regions into dual-format outputs. When you need to generate both structured HTML and readable plain text from the same document regions, this system handles the parallel rendering process efficiently within a single traversal of the regions of interest.

Core Architecture of the RenderSystem

Initialization and Dependencies

The RenderSystem is initialized with two critical dependencies: a TreeRegionsSystem (which contains sorted_roi_by_pos_xpath) and a DOMTreeRepresentation. According to the source code in betterhtmlchunking/render_system.py, the constructor triggers the entire rendering pipeline immediately through __attrs_post_init__, which calls render() at lines 30-32.

Data Containers for Dual Output

To support parallel generation, the system maintains four distinct dictionaries that are initialized at the start of the render() method (lines 68-74):

self.html_render_with_pos_xpath: dict[int, RegionOfInterestRenderT] = {}
self.text_render_with_pos_xpath: dict[int, RegionOfInterestRenderT] = {}
self.html_render_roi: dict[int, str] = {}
self.text_render_roi: dict[int, str] = {}

The *_with_pos_xpath dictionaries store individual XPath renderings, while the *_roi dictionaries hold the final aggregated chunks for each region of interest.

Step-by-Step Parallel Rendering Process

Step 1: Clearing and Initializing Containers

Inside render(), the system first clears any existing data and initializes fresh containers for both HTML and plain-text outputs. This ensures that multiple calls to render() produce consistent results without state pollution.

Step 2: Iterating Over Regions of Interest

The system walks through each ROI using the order supplied by TreeRegionsSystem.sorted_roi_by_pos_xpath (lines 81-84):

for roi_idx, roi in self.tree_regions_system.sorted_roi_by_pos_xpath.items():
    ...

This iteration ensures that both HTML and text outputs maintain the same regional structure and ordering.

Step 3: Generating HTML with BeautifulSoup Prettify

For every pos_xpath within an ROI, the corresponding bs4_elem (BeautifulSoup node) is converted to a compact HTML string using the minimal formatter (lines 91-96):

prettified_pos_xpath_html: str = \
    self.tree_regions_system.tree_representation.xpaths_metadata[
        pos_xpath].bs4_elem.prettify(formatter="minimal")

The formatter="minimal" option ensures the output is clean without aggressive entity escaping, preserving the original HTML structure while adding consistent indentation.

Step 4: Extracting Plain Text with Parsel

Simultaneously, the same BeautifulSoup element is processed by parsel_text.get_bs4_soup_text() to extract readable text content (lines 99-104):

pos_xpath_text: str = parsel_text.get_bs4_soup_text(
    bs4_soup=self.tree_regions_system.tree_representation.xpaths_metadata[
        pos_xpath].bs4_elem
)

This function walks the element tree, concatenates visible text nodes, and normalizes whitespace, producing a clean plain-text representation that corresponds exactly to the HTML content just generated.

Step 5: Aggregating Per-ROI Outputs

After processing all XPaths within an ROI, the system stores the individual results (lines 106-109):

self.html_render_with_pos_xpath[roi_idx][pos_xpath] = prettified_pos_xpath_html
self.text_render_with_pos_xpath[roi_idx][pos_xpath] = pos_xpath_text

Finally, the system aggregates these per-XPath dictionaries into single strings for each ROI using get_roi_html_render_with_pos_xpath() and get_roi_text_render_with_pos_xpath() (lines 113-120). These helper methods simply join the dictionary values with newlines, preserving the original XPath order to maintain document structure.

Accessing the Final Outputs

After render() completes, the RenderSystem instance exposes two ready-to-use dictionaries:

• render_system.html_render_roi – Maps roi_index to a complete HTML chunk containing all XPaths in that region
• render_system.text_render_roi – Maps roi_index to the corresponding plain-text chunk

These dictionaries are consumed by the top-level DomRepresentation class through compute_render_system() in main.py, providing the final dual-format output to users.

Complete Usage Example

The following example demonstrates how to process an HTML document and retrieve both representations using the high-level API:

from betterhtmlchunking.main import DomRepresentation
from betterhtmlchunking.utils import read_file  # hypothetical helper

# Load a web page (HTML source)

html_source = read_file("example.html")   # <-- replace with your own loader

# Create the pipeline object

chunker = DomRepresentation(
    MAX_NODE_REPR_LENGTH=5000,                # max characters per chunk

    website_code=html_source,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

# Run the full processing pipeline (verbose for debugging if desired)

chunker.start(verbose=True, maximal_verbose=False)

# Access rendered outputs

html_chunks = chunker.render_system.html_render_roi     # {0: "<div>...</div>", 1: "..."}

text_chunks = chunker.render_system.text_render_roi     # {0: "Plain text …", 1: "…"}

# Example: print the first chunk pair

print("HTML chunk 0:\n", html_chunks[0])
print("\nPlain-text chunk 0:\n", text_chunks[0])

Key implementation details in this workflow:

• DomRepresentation.start() triggers the three-stage pipeline: tree building, ROI detection, and rendering via RenderSystem.
• The render_system attribute exposes html_render_roi and text_render_roi, which contain parallel outputs indexed by ROI.

Summary

• The RenderSystem class in betterhtmlchunking/render_system.py generates dual outputs by processing each XPath through both BeautifulSoup's prettify() and parsel_text.get_bs4_soup_text() during a single traversal.
• Four dictionaries manage the parallel pipeline: html_render_with_pos_xpath and text_render_with_pos_xpath store per-XPath results, while html_render_roi and text_render_roi hold the final aggregated chunks.
• The render() method orchestrates the process by iterating over sorted_roi_by_pos_xpath from the TreeRegionsSystem, ensuring that HTML and text outputs maintain identical structural ordering.
• Final outputs are accessed through render_system.html_render_roi and render_system.text_render_roi after invoking the pipeline via DomRepresentation.start().

Frequently Asked Questions

Why does the system generate both formats simultaneously instead of converting HTML to text later?

Generating both representations during the same traversal eliminates the need to re-parse or re-walk the BeautifulSoup tree, which significantly improves performance when processing large documents. According to the implementation in render_system.py, both prettify() and get_bs4_soup_text() operate on the same bs4_elem instance within the same loop iteration (lines 91-104), ensuring that XPath ordering and structural context remain perfectly synchronized between the two outputs.

What is the difference between the per-XPath and per-ROI output dictionaries?

The html_render_with_pos_xpath and text_render_with_pos_xpath dictionaries store individual string representations for each specific XPath within a region of interest, allowing granular access to specific DOM nodes. In contrast, html_render_roi and text_render_roi contain the final aggregated strings where all XPaths within a single ROI have been concatenated with newlines, providing complete chunks suitable for downstream processing or storage.

How does the minimal formatter in BeautifulSoup affect the HTML output?

The formatter="minimal" parameter passed to prettify() in lines 91-96 of render_system.py ensures that BeautifulSoup applies only essential formatting—adding consistent indentation and line breaks for readability—without aggressively escaping HTML entities or altering attribute quoting. This produces clean, human-readable HTML that preserves the original document's structure while ensuring that each XPath's contribution to the ROI is clearly delineated in the final aggregated chunk.

Can I customize the text extraction logic beyond what parsel provides?

While the current implementation in render_system.py hardcodes the use of parsel_text.get_bs4_soup_text() for text extraction (lines 99-104), the modular architecture allows for extension by subclassing RenderSystem and overriding the text rendering logic within the XPath processing loop. However, to maintain synchronization with the HTML output, any custom text extraction method must accept a BeautifulSoup element and return a string, operating within the same iteration structure to preserve the parallel generation guarantee.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}