# How the RenderSystem Generates Both HTML and Plain Text Outputs in Parallel

> Learn how the RenderSystem efficiently generates parallel HTML and plain text outputs using BeautifulSoup and Parsel for faster content processing. Explore the betterhtmlchunking repository.

- Repository: [Carlos A. Planchón/betterhtmlchunking](https://github.com/carlosplanchon/betterhtmlchunking)
- Tags: internals
- Published: 2026-02-26

---

**The `RenderSystem` class in `betterhtmlchunking` processes each region of interest (ROI) by simultaneously applying BeautifulSoup's `prettify()` for HTML and `parsel_text.get_bs4_soup_text()` for plain text, storing results in parallel dictionaries before aggregating them into final per-ROI chunks.**

The `RenderSystem` is a core component of the `carlosplanchon/betterhtmlchunking` repository that transforms identified tree regions into dual-format outputs. When you need to generate both structured HTML and readable plain text from the same document regions, this system handles the parallel rendering process efficiently within a single traversal of the regions of interest.

## Core Architecture of the RenderSystem

### Initialization and Dependencies

The `RenderSystem` is initialized with two critical dependencies: a `TreeRegionsSystem` (which contains `sorted_roi_by_pos_xpath`) and a `DOMTreeRepresentation`. According to the source code in [`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py), the constructor triggers the entire rendering pipeline immediately through `__attrs_post_init__`, which calls `render()` at lines 30-32.

### Data Containers for Dual Output

To support parallel generation, the system maintains four distinct dictionaries that are initialized at the start of the `render()` method (lines 68-74):

```python
self.html_render_with_pos_xpath: dict[int, RegionOfInterestRenderT] = {}
self.text_render_with_pos_xpath: dict[int, RegionOfInterestRenderT] = {}
self.html_render_roi: dict[int, str] = {}
self.text_render_roi: dict[int, str] = {}

```

The `*_with_pos_xpath` dictionaries store individual XPath renderings, while the `*_roi` dictionaries hold the final aggregated chunks for each region of interest.

## Step-by-Step Parallel Rendering Process

### Step 1: Clearing and Initializing Containers

Inside `render()`, the system first clears any existing data and initializes fresh containers for both HTML and plain-text outputs. This ensures that multiple calls to `render()` produce consistent results without state pollution.

### Step 2: Iterating Over Regions of Interest

The system walks through each ROI using the order supplied by `TreeRegionsSystem.sorted_roi_by_pos_xpath` (lines 81-84):

```python
for roi_idx, roi in self.tree_regions_system.sorted_roi_by_pos_xpath.items():
    ...

```

This iteration ensures that both HTML and text outputs maintain the same regional structure and ordering.

### Step 3: Generating HTML with BeautifulSoup Prettify

For every `pos_xpath` within an ROI, the corresponding `bs4_elem` (BeautifulSoup node) is converted to a compact HTML string using the minimal formatter (lines 91-96):

```python
prettified_pos_xpath_html: str = \
    self.tree_regions_system.tree_representation.xpaths_metadata[
        pos_xpath].bs4_elem.prettify(formatter="minimal")

```

The `formatter="minimal"` option ensures the output is clean without aggressive entity escaping, preserving the original HTML structure while adding consistent indentation.

### Step 4: Extracting Plain Text with Parsel

Simultaneously, the same BeautifulSoup element is processed by `parsel_text.get_bs4_soup_text()` to extract readable text content (lines 99-104):

```python
pos_xpath_text: str = parsel_text.get_bs4_soup_text(
    bs4_soup=self.tree_regions_system.tree_representation.xpaths_metadata[
        pos_xpath].bs4_elem
)

```

This function walks the element tree, concatenates visible text nodes, and normalizes whitespace, producing a clean plain-text representation that corresponds exactly to the HTML content just generated.

### Step 5: Aggregating Per-ROI Outputs

After processing all XPaths within an ROI, the system stores the individual results (lines 106-109):

```python
self.html_render_with_pos_xpath[roi_idx][pos_xpath] = prettified_pos_xpath_html
self.text_render_with_pos_xpath[roi_idx][pos_xpath] = pos_xpath_text

```

Finally, the system aggregates these per-XPath dictionaries into single strings for each ROI using `get_roi_html_render_with_pos_xpath()` and `get_roi_text_render_with_pos_xpath()` (lines 113-120). These helper methods simply join the dictionary values with newlines, preserving the original XPath order to maintain document structure.

## Accessing the Final Outputs

After `render()` completes, the `RenderSystem` instance exposes two ready-to-use dictionaries:

- `render_system.html_render_roi` – Maps `roi_index` to a complete HTML chunk containing all XPaths in that region
- `render_system.text_render_roi` – Maps `roi_index` to the corresponding plain-text chunk

These dictionaries are consumed by the top-level `DomRepresentation` class through `compute_render_system()` in [`main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/main.py), providing the final dual-format output to users.

## Complete Usage Example

The following example demonstrates how to process an HTML document and retrieve both representations using the high-level API:

```python
from betterhtmlchunking.main import DomRepresentation
from betterhtmlchunking.utils import read_file  # hypothetical helper

# Load a web page (HTML source)

html_source = read_file("example.html")   # <-- replace with your own loader

# Create the pipeline object

chunker = DomRepresentation(
    MAX_NODE_REPR_LENGTH=5000,                # max characters per chunk

    website_code=html_source,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

# Run the full processing pipeline (verbose for debugging if desired)

chunker.start(verbose=True, maximal_verbose=False)

# Access rendered outputs

html_chunks = chunker.render_system.html_render_roi     # {0: "<div>...</div>", 1: "..."}

text_chunks = chunker.render_system.text_render_roi     # {0: "Plain text …", 1: "…"}

# Example: print the first chunk pair

print("HTML chunk 0:\n", html_chunks[0])
print("\nPlain-text chunk 0:\n", text_chunks[0])

```

Key implementation details in this workflow:

- `DomRepresentation.start()` triggers the three-stage pipeline: tree building, ROI detection, and rendering via `RenderSystem`.
- The `render_system` attribute exposes `html_render_roi` and `text_render_roi`, which contain **parallel** outputs indexed by ROI.

## Summary

- The `RenderSystem` class in [`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py) generates dual outputs by processing each XPath through both BeautifulSoup's `prettify()` and `parsel_text.get_bs4_soup_text()` during a single traversal.
- Four dictionaries manage the parallel pipeline: `html_render_with_pos_xpath` and `text_render_with_pos_xpath` store per-XPath results, while `html_render_roi` and `text_render_roi` hold the final aggregated chunks.
- The `render()` method orchestrates the process by iterating over `sorted_roi_by_pos_xpath` from the `TreeRegionsSystem`, ensuring that HTML and text outputs maintain identical structural ordering.
- Final outputs are accessed through `render_system.html_render_roi` and `render_system.text_render_roi` after invoking the pipeline via `DomRepresentation.start()`.

## Frequently Asked Questions

### Why does the system generate both formats simultaneously instead of converting HTML to text later?

Generating both representations during the same traversal eliminates the need to re-parse or re-walk the BeautifulSoup tree, which significantly improves performance when processing large documents. According to the implementation in [`render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/render_system.py), both `prettify()` and `get_bs4_soup_text()` operate on the same `bs4_elem` instance within the same loop iteration (lines 91-104), ensuring that XPath ordering and structural context remain perfectly synchronized between the two outputs.

### What is the difference between the per-XPath and per-ROI output dictionaries?

The `html_render_with_pos_xpath` and `text_render_with_pos_xpath` dictionaries store individual string representations for each specific XPath within a region of interest, allowing granular access to specific DOM nodes. In contrast, `html_render_roi` and `text_render_roi` contain the final aggregated strings where all XPaths within a single ROI have been concatenated with newlines, providing complete chunks suitable for downstream processing or storage.

### How does the minimal formatter in BeautifulSoup affect the HTML output?

The `formatter="minimal"` parameter passed to `prettify()` in lines 91-96 of [`render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/render_system.py) ensures that BeautifulSoup applies only essential formatting—adding consistent indentation and line breaks for readability—without aggressively escaping HTML entities or altering attribute quoting. This produces clean, human-readable HTML that preserves the original document's structure while ensuring that each XPath's contribution to the ROI is clearly delineated in the final aggregated chunk.

### Can I customize the text extraction logic beyond what parsel provides?

While the current implementation in [`render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/render_system.py) hardcodes the use of `parsel_text.get_bs4_soup_text()` for text extraction (lines 99-104), the modular architecture allows for extension by subclassing `RenderSystem` and overriding the text rendering logic within the XPath processing loop. However, to maintain synchronization with the HTML output, any custom text extraction method must accept a BeautifulSoup element and return a string, operating within the same iteration structure to preserve the parallel generation guarantee.