How to Implement Custom Jinja Template Processors in Apache Superset

Implement custom Jinja template processors by subclassing BaseTemplateProcessor or engine-specific variants, overriding process_template, and registering the class in the CUSTOM_TEMPLATE_PROCESSORS configuration dictionary.

Apache Superset uses a sandboxed Jinja environment to render SQL Lab queries dynamically. The rendering pipeline relies on extensible template processor classes defined in superset/jinja_context.py, allowing you to inject organization-specific macros, custom date functions, or proprietary syntax extensions without modifying core Superset code.

Understanding the Template Processor Hierarchy

Superset organizes template processing into a hierarchy of classes that determine how Jinja contexts are built and rendered:

• BaseTemplateProcessor – Establishes the core sandbox and registers generic filters like where_in and to_datetime. It provides the base implementation of process_template that evaluates Jinja expressions securely.
• JinjaTemplateProcessor – Extends the base class to inject Superset-specific helper functions including url_param, current_user_id, and metric.
• Engine-specific subclasses (PrestoTemplateProcessor, HiveTemplateProcessor, etc.) – Namespace engine-specific macros under the engine name (e.g., {{ presto.latest_partition(...) }}).
• NoOpTemplateProcessor – A pass-through implementation used when the ENABLE_TEMPLATE_PROCESSING feature flag is disabled, returning raw SQL unchanged.

How Superset Selects a Template Processor

Superset discovers and instantiates processors through the get_template_processor function in superset/jinja_context.py. The selection logic follows this priority:

# superset/jinja_context.py

def get_template_processor(database, table=None, query=None, **kwargs):
    if feature_flag_manager.is_feature_enabled("ENABLE_TEMPLATE_PROCESSING"):
        # Look up custom processors first, then defaults

        template_processor = get_template_processors().get(
            database.backend, JinjaTemplateProcessor
        )
    else:
        template_processor = NoOpTemplateProcessor
    return template_processor(database=database, table=table, query=query, **kwargs)

The function maps database.backend (the engine name string, such as "presto" or "hive") to a processor class. If CUSTOM_TEMPLATE_PROCESSORS contains a matching entry, Superset uses your custom class; otherwise it falls back to the defaults defined in DEFAULT_PROCESSORS.

Creating a Custom Template Processor

Follow these steps to extend Superset's Jinja rendering capabilities with custom macros or syntax.

Step 1: Subclass an Existing Processor

Create a Python module that imports and extends the processor corresponding to your target database engine. Most implementations extend PrestoTemplateProcessor or JinjaTemplateProcessor.

# my_custom_processors.py

from superset.jinja_context import PrestoTemplateProcessor
from datetime import datetime, timedelta
from functools import partial
from typing import Any, Dict

def DATE(ts: datetime, day_offset: int = 0, hour_offset: int = 0) -> str:
    """Custom macro returning ISO-formatted date with optional offsets."""
    target = ts + timedelta(days=day_offset, hours=hour_offset)
    return target.date().isoformat()

Step 2: Override process_template or set_context

Override process_template to inject your custom macros into the rendering context. The method receives the raw SQL string and must return the processed string.

import re

class CustomPrestoProcessor(PrestoTemplateProcessor):
    """Processor that adds $-style DATE macros to Presto queries."""
    engine = "presto"  # Overrides the built-in Presto processor

    def process_template(self, sql: str, **kwargs) -> str:
        # Prepare macro dictionary: $DATE(...) expands to the helper above

        macros: Dict[str, Any] = {
            "DATE": partial(DATE, datetime.utcnow())
        }
        # Merge default context, kwargs and user-provided macros

        macros.update(self._context)
        macros.update(kwargs)

        def replacer(match):
            macro_name, args_str = match.groups()
            args = [a.strip() for a in args_str.split(",") if a.strip()]
            return macros[macro_name[1:]](*args)

        macro_names = ["$" + name for name in macros.keys()]
        pattern = r"(%s)\s*\(([^()]*)\)" % "|".join(map(re.escape, macro_names))
        return re.sub(pattern, replacer, sql)

The test suite in tests/integration_tests/superset_test_custom_template_processors.py demonstrates this same pattern for integrating non-Jinja syntax alongside standard Jinja rendering.

Step 3: Register in superset_config.py

Map your custom processor to the appropriate engine name in your Superset configuration file.

# superset_config.py

from my_custom_processors import CustomPrestoProcessor

CUSTOM_TEMPLATE_PROCESSORS = {
    "presto": CustomPrestoProcessor,   # Replace default Presto processor

}

Step 4: Enable the Feature Flag

Ensure template processing is active in your environment. In development configurations, explicitly enable the flag:

from superset.extensions import feature_flag_manager
feature_flag_manager.set_feature_flag("ENABLE_TEMPLATE_PROCESSING", True)

Step 5: Use Custom Macros in SQL Lab

With the processor registered, write queries using your custom syntax:

SELECT *
FROM events
WHERE event_date = $DATE(0)
  AND event_hour = $DATE(0, -1)

The $DATE macro expands to the current UTC date (or with specified offsets) before the Jinja sandbox processes the remainder of the template.

Complete Working Example

Combine the components into a production-ready implementation:

# my_custom_processors.py

from superset.jinja_context import PrestoTemplateProcessor
from datetime import datetime, timedelta
from functools import partial
from typing import Any, Dict
import re

def DATE(ts: datetime, day_offset: int = 0, hour_offset: int = 0) -> str:
    target = ts + timedelta(days=day_offset, hours=hour_offset)
    return target.date().isoformat()

class CustomPrestoProcessor(PrestoTemplateProcessor):
    """Extends Presto processing with $DATE macro support."""
    engine = "presto"

    def process_template(self, sql: str, **kwargs) -> str:
        macros: Dict[str, Any] = {"DATE": partial(DATE, datetime.utcnow())}
        macros.update(self._context)
        macros.update(kwargs)

        def replacer(match):
            macro_name, args_str = match.groups()
            args = [a.strip() for a in args_str.split(",") if a.strip()]
            return macros[macro_name[1:]](*args)

        macro_names = ["$" + name for name in macros.keys()]
        pattern = r"(%s)\s*\(([^()]*)\)" % "|".join(map(re.escape, macro_names))
        return re.sub(pattern, replacer, sql)

# superset_config.py

from my_custom_processors import CustomPrestoProcessor

CUSTOM_TEMPLATE_PROCESSORS = {
    "presto": CustomPrestoProcessor,
}

Key Source Files and Implementation Details

Understanding these specific locations in the Apache Superset codebase helps when debugging or extending processors:

• superset/jinja_context.py – Contains the processor hierarchy, DEFAULT_PROCESSORS mapping, and get_template_processors() logic (lines 666-681).
• superset/jinja_context.py (lines 66-81) – Defines BaseTemplateProcessor with sandbox configuration and default filters.
• superset/jinja_context.py (lines 88-102) – Implements JinjaTemplateProcessor adding Superset-specific context variables.
• tests/integration_tests/superset_test_custom_template_processors.py – Reference implementation demonstrating custom macro expansion patterns used in the test suite (lines 35-59).

Summary

• Template processors in Superset are Python classes that prepare the Jinja environment and render SQL templates safely.
• Extend functionality by subclassing PrestoTemplateProcessor, JinjaTemplateProcessor, or BaseTemplateProcessor in superset/jinja_context.py.
• Override process_template to inject custom macros or implement alternative syntax like $DATE().
• Register custom processors via the CUSTOM_TEMPLATE_PROCESSORS dictionary in superset_config.py, mapping engine names to class implementations.
• Superset checks ENABLE_TEMPLATE_PROCESSING before invoking custom logic; otherwise it uses NoOpTemplateProcessor to return raw SQL unchanged.

Frequently Asked Questions

How does Superset know which processor to use for a specific database?

Superset inspects database.backend (a string like "presto" or "hive") and looks up the corresponding class in CUSTOM_TEMPLATE_PROCESSORS. If no custom mapping exists, it falls back to DEFAULT_PROCESSORS defined in superset/jinja_context.py. This lookup occurs inside get_template_processor before instantiating the class with database, table, and query context.

Can I add custom functions without replacing the entire processor?

Yes. Instead of completely overriding process_template, you can override set_context to add variables or functions to self._context. The base class merges this dictionary into the Jinja environment. For example, append your helper to self._context['my_helper'] = my_function during set_context, then reference it in SQL as {{ my_helper() }} using standard Jinja syntax.

What is the difference between BaseTemplateProcessor and JinjaTemplateProcessor?

BaseTemplateProcessor provides the sandboxed Jinja environment and core security filters but minimal context variables. JinjaTemplateProcessor extends the base class to include Superset-specific globals like current_user_id, current_username, url_param, and metric. Use BaseTemplateProcessor for lightweight customizations or when you want to exclude Superset-specific variables; use JinjaTemplateProcessor to retain standard Superset functionality while adding your own.

Why is my custom processor not being invoked?

First, verify that ENABLE_TEMPLATE_PROCESSING is enabled in your feature flags, as disabled flags force the use of NoOpTemplateProcessor. Second, confirm that CUSTOM_TEMPLATE_PROCESSORS maps the exact engine name string (case-sensitive) to your class. Finally, ensure your processor module is importable from superset_config.py and that the engine class attribute matches the database backend identifier if you are overriding an engine-specific processor.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}