# Multi-Level Label Architectures for Task Routing in Apache Linkis: Load Balancing and Traffic Control

> Discover Apache Linkis's six-level label architecture for intelligent task routing. Achieve precise load balancing and resource isolation in distributed clusters.

- Repository: [The Apache Software Foundation/linkis](https://github.com/apache/linkis)
- Tags: architecture
- Published: 2026-02-24

---

**Apache Linkis employs a hierarchical six-level label system that combines user identity, engine specifications, and traffic-control metadata to route tasks intelligently, enabling precise load balancing and resource isolation across distributed compute clusters.**

Apache Linkis utilizes a sophisticated multi-level label architecture for task routing to transform job metadata into a structured routing language. This hierarchical system attaches contextual labels to every request via the `GatewayContext` or `MarkReq`, allowing the gateway router and engine connection manager to make intelligent decisions about service discovery, engine allocation, and resource quotas. By composing labels ranging from `UserCreatorLabel` to `LoadBalanceLabel`, Linkis achieves granular traffic control and optimal load distribution across heterogeneous computational workloads.

## The Six-Level Label Hierarchy

Linkis organizes routing metadata into six distinct levels, each represented by specific Java classes in the `org.apache.linkis.manager.label.entity` package. These labels collectively form a composite routing key stored in the `linkis_cg_manager_label` database table.

### Level 1: User and Creator Identification

The `UserCreatorLabel` captures the submitting user and the creator service that generated the request. Located at `org.apache.linkis.manager.label.entity.engine.UserCreatorLabel`, this label ensures tasks are associated with specific principals and enables user-based resource quotas. It serves as the foundation for the routing key, distinguishing requests from different sources even when targeting identical engine types.

### Level 2: Engine Type Specification

The `EngineTypeLabel` determines which engine implementation executes the task, specifying both the engine type and version (e.g., Spark-2.4.3, Hive-3.1.3, or Trino). Found in `org.apache.linkis.manager.label.entity.engine.EngineTypeLabel`, this label directs requests to compatible engine instances and prevents version mismatches during task dispatch.

### Level 3: Combined Route Label

Linkis synthesizes the user and engine levels into a `combined_userCreator_engineType` label stored in the `linkis_cg_manager_label` table. This **route label** acts as the primary lookup key for service discovery, enabling the gateway to locate the correct service instance among potentially hundreds of nodes. The `RouteLabel` interface defines this combined identifier used by `DefaultLabelGatewayRouter` during instance selection.

### Level 4: Load Balancing Configuration

The `LoadBalanceLabel` controls parallel instance creation through its `capacity` and `groupId` fields. Defined in `org.apache.linkis.manager.label.entity.entrance.LoadBalanceLabel`, this label specifies the maximum number of concurrent engine instances permitted for a particular label combination. The `LoadBalanceLabelEngineConnManager` interprets this capacity to determine how many *marks* (engine slots) to create and cache for incoming requests.

### Level 5: Job-Group Binding

`BindEngineLabel` ensures stateful pipeline execution by binding all tasks in a job group to the same engine instance. Located in `org.apache.linkis.manager.label.entity.entrance.BindEngineLabel`, this label uses `isJobGroupHead`, `isJobGroupEnd`, and `jobGroupId` fields to cache the selected engine mark for the entire job duration. This prevents race conditions in multi-stage workflows where subsequent tasks must reuse the same computational context.

### Level 6: Runtime and Isolation Controls

Fine-grained execution parameters are managed through labels like `EngineConnRuntimeModeLabel` and `ReuseExclusionLabel`. These control engine reuse policies, YARN deployment modes (cluster vs. local), and isolation boundaries. When `ReuseExclusionLabel` is present, the system prevents oversubscription by excluding already-allocated engines from the candidate pool during load balancing decisions.

## The Task Routing Flow

The routing process transforms HTTP requests into engine allocations through a multi-stage pipeline involving the gateway, metadata database, and orchestrator.

### Gateway Label Parsing

When the gateway receives an HTTP request, it constructs a `GatewayContext` containing the label list via `gatewayContext.getGatewayRoute.getLabels`. The `DefaultLabelGatewayRouter` (located at [`linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-server-support/src/main/scala/org/apache/linkis/gateway/ujes/route/DefaultLabelGatewayRouter.scala`](https://github.com/apache/linkis/blob/main/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-server-support/src/main/scala/org/apache/linkis/gateway/ujes/route/DefaultLabelGatewayRouter.scala)) parses these labels through a chain of `RouteLabelParser` implementations:

```scala
// DefaultLabelGatewayRouter.selectInstance (excerpt)
val routeLabels = parseToRouteLabels(gatewayContext) // ← uses parsers

```

### Service Instance Selection

The router queries the metadata database for service instances carrying the matching `combined_userCreator_engineType` label:

```sql
SELECT * FROM linkis_cg_manager_label
WHERE label_key='combined_userCreator_engineType';

```

After retrieving eligible instances, the router performs a **roulette** (random) selection among healthy nodes to distribute traffic evenly across the cluster.

### Engine Mark Allocation

The `LoadBalanceLabelEngineConnManager` (found in [`linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/ecm/LoadBalanceLabelEngineConnManager.scala`](https://github.com/apache/linkis/blob/main/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/ecm/LoadBalanceLabelEngineConnManager.scala)) processes the `MarkReq` to allocate engine connections. It extracts the `LoadBalanceLabel` from the request's label map:

```scala
val loadBalanceLabel = MarkReq.getLabelBuilderFactory.createLabel[LoadBalanceLabel](
  LabelKeyConstant.LOAD_BALANCE_KEY,
  markReq.getLabels.get(LabelKeyConstant.LOAD_BALANCE_KEY)
)

```

## Load Balancing Mechanisms

Linkis implements sophisticated load distribution through three complementary mechanisms that operate at different stages of the task lifecycle.

### Capacity-Based Mark Creation

The `LoadBalanceLabel.getCapacity()` value determines the maximum number of concurrent engine instances created for a specific label combination. When `applyMark` is invoked, the `LoadBalanceLabelEngineConnManager` creates and caches up to *N* marks based on this capacity, effectively throttling parallel execution for that workload type. This prevents any single user or engine type from monopolizing cluster resources.

### Job-Group Affinity

When `BindEngineLabel.isJobGroupHead` is true, the system selects a random available mark and caches it for the entire job group. Subsequent tasks with the same `jobGroupId` retrieve this cached mark rather than requesting a new engine, ensuring pipeline consistency. The mark is removed from cache when `isJobGroupEnd` signals the final task completion, releasing the engine for reuse.

### Reuse Exclusion Policies

The `ReuseExclusionLabel` prevents oversubscription by marking engines already allocated to other active marks. During the `applyMark` phase, the engine connection manager filters out excluded instances, ensuring load balancing distributes work across truly available resources rather than overloading busy engines.

## Traffic Control Implementation

Beyond load distribution, Linkis enforces hard limits on concurrent execution through the `UserRunningNumber` tracker located in [`linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/execution/impl/UserRunningNumber.scala`](https://github.com/apache/linkis/blob/main/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/execution/impl/UserRunningNumber.scala).

### Per-User and Per-Engine Quotas

The system constructs a unique key combining `UserCreatorLabel` and `EngineTypeLabel` values:

```scala
val key = userCreatorLabel.getStringValue + SPLIT + engineTypeLabel.getStringValue

```

This key increments a counter for every submitted task and decrements it upon completion. When the counter reaches the configured threshold for that key, subsequent requests are queued or rejected, preventing individual users or engine types from exhausting cluster capacity. This mechanism operates independently of the load balancing capacity controls, providing a secondary safety net against resource starvation.

## Practical Implementation: Submitting a Load-Balanced Spark Job

The following Scala example demonstrates how to construct a `MarkReq` with multi-level labels for a Spark-IDE workload with capacity limits and job-group binding:

```scala
import org.apache.linkis.manager.label.builder.factory.LabelBuilderFactoryContext
import org.apache.linkis.manager.label.constant.LabelKeyConstant
import org.apache.linkis.manager.label.entity.entrance.{LoadBalanceLabel, BindEngineLabel}
import org.apache.linkis.manager.label.entity.engine.{UserCreatorLabel, EngineTypeLabel}
import org.apache.linkis.orchestrator.ecm.entity.MarkReq

// 1️⃣ Build the required labels via the factory
val labelFactory = LabelBuilderFactoryContext.getLabelBuilderFactory

val userCreator = labelFactory.createLabel(classOf[UserCreatorLabel])
  .setUser("alice")
  .setCreator("linkis")

val engineType = labelFactory.createLabel(classOf[EngineTypeLabel])
  .setEngineType("spark")
  .setVersion("2.4.3")

val loadBalance = labelFactory.createLabel(classOf[LoadBalanceLabel])
  .setCapacity("5")                     // Maximum 5 concurrent Spark-IDE engines
  .setGroupId("spark-ide-group")

val bindEngine = labelFactory.createLabel(classOf[BindEngineLabel])
  .setJobGroupId("group-001")
  .setIsJobGroupHead(true)

// 2️⃣ Assemble the MarkReq with all labels
val markReq = new MarkReq()
markReq.getLabels.put(LabelKeyConstant.USERCREATOR_KEY, userCreator.getValue)
markReq.getLabels.put(LabelKeyConstant.ENGINETYPE_KEY, engineType.getValue)
markReq.getLabels.put(LabelKeyConstant.LOAD_BALANCE_KEY, loadBalance.getValue)
markReq.getLabels.put(LabelKeyConstant.BIND_ENGINE_KEY, bindEngine.getValue)

// 3️⃣ Apply a mark – triggers load balancing and engine allocation
val engineConnManager = new LoadBalanceLabelEngineConnManager()
val mark = engineConnManager.applyMark(markReq)   // ← Load balancing occurs here

```

This configuration ensures that up to five Spark-IDE engines can run concurrently for the specified user, while all tasks belonging to `group-001` share the same engine instance for stateful execution.

## Summary

Apache Linkis implements multi-level label architectures for task routing through a composable hierarchy of six label types. Key technical implementations include:

- **Hierarchical composition**: User and engine labels combine into `combined_userCreator_engineType` route labels for service discovery in `linkis_cg_manager_label`
- **Gateway routing**: `DefaultLabelGatewayRouter` parses labels and performs roulette selection across matching service instances
- **Capacity management**: `LoadBalanceLabelEngineConnManager` respects `LoadBalanceLabel.capacity` to limit concurrent engine instances
- **Stateful pipelines**: `BindEngineLabel` caches engine marks for job groups, ensuring consistent execution context across multi-stage workflows
- **Quota enforcement**: `UserRunningNumber` tracks concurrent tasks keyed by `UserCreatorLabel` and `EngineTypeLabel` combinations to prevent resource exhaustion

## Frequently Asked Questions

### What is the primary purpose of the combined_userCreator_engineType label?

The `combined_userCreator_engineType` label serves as the fundamental routing key that merges user identity with engine specifications. Stored in the `linkis_cg_manager_label` database table, this composite label enables the `DefaultLabelGatewayRouter` to map incoming requests to specific service instances capable of executing the workload. Without this combined label, the system could not distinguish between different users requesting the same engine type or route traffic appropriately across the cluster.

### How does Linkis prevent a single user from overwhelming the cluster?

Linkis implements dual-layer protection through the `UserRunningNumber` tracker and `LoadBalanceLabel` capacity controls. The `UserRunningNumber` class maintains atomic counters keyed by the concatenation of `UserCreatorLabel` and `EngineTypeLabel` values, rejecting new submissions when per-user limits are reached. Simultaneously, the `LoadBalanceLabelEngineConnManager` enforces maximum engine instance counts per label combination, ensuring that even legitimate requests cannot spawn unlimited resources beyond configured `capacity` thresholds.

### What is the difference between LoadBalanceLabel and BindEngineLabel?

`LoadBalanceLabel` controls **horizontal** scaling by specifying how many concurrent engine instances (`capacity`) may exist for a particular workload type, enabling parallel execution across multiple nodes. In contrast, `BindEngineLabel` controls **vertical** consistency by forcing all tasks within a specific `jobGroupId` to reuse the same engine instance. While `LoadBalanceLabel` promotes distribution and throughput, `BindEngineLabel` ensures stateful operations and data locality by preventing engine switching mid-pipeline.

### Where does the actual load balancing decision occur in the Linkis codebase?

The core load balancing logic resides in [`LoadBalanceLabelEngineConnManager.scala`](https://github.com/apache/linkis/blob/main/LoadBalanceLabelEngineConnManager.scala) within the `linkis-computation-orchestrator` module. When `applyMark` is called, this class reads the `LoadBalanceLabel` from the `MarkReq`, checks existing mark caches, and either reuses available engine connections or creates new ones up to the specified capacity. For gateway-level distribution, `DefaultLabelGatewayRouter` performs initial roulette selection among healthy service instances matching the route label before the orchestrator applies finer-grained engine-level balancing.