# What Is an Executor in Spark Standalone Mode? Key Differences from Workers and Cores

> Understand the executor in Spark standalone mode. Learn how executors differ from workers and cores, and how they run tasks and cache data.

- Repository: [The Apache Software Foundation/spark](https://github.com/apache/spark)
- Tags: deep-dive
- Published: 2026-02-21

---

**An executor in Spark is a per-application JVM process launched by a Worker node to run tasks and cache data, distinct from the long-running Worker daemon that manages node resources and the CPU cores that represent the actual compute slots allocated to that executor by the Master.**

When deploying Apache Spark on a standalone cluster, understanding the hierarchy between workers, executors, and cores is essential for tuning performance and resource utilization. In the `apache/spark` repository, these components are implemented across specific source files that handle resource advertisement, process lifecycle management, and task execution scheduling. This article examines the actual implementation to clarify how these three elements differ and collaborate within the Standalone cluster manager.

## Spark Standalone Architecture Overview

In the Standalone deployment mode, three distinct entities manage compute resources: the **Worker**, the **Executor**, and **CPU cores**. Each plays a specific role in the cluster resource management lifecycle.

### Worker Node

The **Worker** is a long-running daemon process that runs on each machine in the cluster. According to [`core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala), the Worker registers with the Master via the `RegisterWorker` message (lines 58‑71), advertises the total number of **CPU cores** and **memory** available on its host, and awaits instructions to launch executor processes. The Worker maintains a web UI accessible at `http://<host>:8081` and tracks `coresUsed` and `memoryUsed` to enforce local resource limits.

### Executor Process

An **Executor** is a JVM process started by a Worker specifically for one Spark application. As implemented in [`core/src/main/scala/org/apache/spark/executor/Executor.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/executor/Executor.scala) (constructor at lines 48‑55), the Executor creates a thread pool to run tasks, holds the application’s cached RDDs and DataFrames in memory, and reports task progress and heartbeats back to the Driver. When the application finishes, the Worker terminates the executor and frees its allocated resources. Multiple executors from different applications can coexist on the same Worker node, or a single application can span multiple Workers.

### CPU Cores

**Cores** represent the atomic unit of compute capacity that the Master allocates to applications. Workers report their total cores via the `--cores` or `spark.worker.cores` configuration (documented in [`docs/spark-standalone.md`](https://github.com/apache/spark/blob/main/docs/spark-standalone.md), lines 71‑73). When the Master schedules an application, it determines how many cores each executor receives through `spark.executor.cores`. If this value is omitted, the executor consumes **all** free cores on the Worker, which can lead to resource monopolization.

## How Workers, Executors, and Cores Interact

The lifecycle from cluster startup to task execution follows a strict choreography defined in the Standalone master and worker implementations:

1. **Worker Registration** – Starting a Worker via [`sbin/start-worker.sh`](https://github.com/apache/spark/blob/main/sbin/start-worker.sh) initiates the daemon, which contacts the Master at `spark://HOST:PORT` and sends its total resource capacity (cores and memory) as stored in [`WorkerInfo.scala`](https://github.com/apache/spark/blob/main/WorkerInfo.scala).

2. **Application Submission** – When you run `spark-submit --master spark://HOST:PORT`, the Master evaluates the request against available resources using constraints like `spark.cores.max` (total cores for the application) and `spark.deploy.defaultCores` (fallback allocation).

3. **Executor Launch** – The Master issues a `LaunchExecutor` RPC to a selected Worker. The Worker spawns an `ExecutorRunner` process, which instantiates the `Executor` object from [`Executor.scala`](https://github.com/apache/spark/blob/main/Executor.scala) and configures its working directory and log files.

4. **Task Execution** – Inside the Executor, a thread pool runs application tasks. Each active task consumes one **core slot** from the executor’s allocation (configured via `spark.executor.cores`). The Executor sends heartbeats and metrics to the Driver while maintaining local storage for shuffle and cached data.

5. **Resource Cleanup** – Upon application completion or failure, the Worker terminates the Executor process and decrements its `coresUsed` and `memoryUsed` counters, making those resources available for subsequent applications as tracked in [`ExecutorInfo.scala`](https://github.com/apache/spark/blob/main/ExecutorInfo.scala).

## Key Differences Between Workers and Executors

Understanding the distinction between these components prevents common configuration errors like confusing node-level settings with application-level resource requests.

| Aspect | Worker | Executor |
|--------|--------|----------|
| **Lifecycle** | Starts once per node and persists for the cluster’s uptime unless manually stopped. | Created per application when scheduled by the Master; terminated when the application finishes or fails. |
| **Responsibility** | Manages node resources, registers with Master, launches and monitors executor processes, and cleans up local directories. | Executes tasks, stores cached data blocks, performs serialization and garbage collection, and reports status to the Driver. |
| **Resource Granularity** | Holds the *total* CPU cores and memory of the physical host. | Holds a *subset* of the Worker’s cores (determined by `spark.executor.cores`) and a slice of memory (`spark.executor.memory`). |
| **Visibility** | Appears in the Master UI under the **Workers** tab with links to the Worker’s web interface. | Appears in the **Executors** tab of the specific application’s UI with individual log links and task statistics. |

## Configuring Executors and Cores in Standalone Mode

Controlling the relationship between Workers, Executors, and cores requires setting both node-level and application-level parameters.

### Starting a Worker Node

Launch the Worker daemon on each cluster node to advertise resources to the Master:

```bash
./sbin/start-worker.sh spark://master-host:7077

```

The Worker automatically detects local hardware or uses explicit flags to define its total resource offer to the cluster.

### Submitting an Application with Resource Controls

Specify how the Master should slice Worker resources into Executor processes:

```bash
./bin/spark-submit \
  --master spark://master-host:7077 \
  --total-executor-cores 4 \
  --executor-cores 2 \
  --executor-memory 2g \
  --class org.example.MyApp \
  my-app.jar

```

This configuration requests two executors (each with 2 cores), which the Master may place on one Worker or distribute across two different Workers depending on availability.

### Inspecting Executors Programmatically

From within your Spark application, you can query the Driver’s view of allocated executors:

```scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .master("spark://master-host:7077")
  .appName("ResourceCheck")
  .getOrCreate()

spark.sparkContext.getExecutorMemoryStatus.foreach { 
  case (hostPort, (maxMem, remainingMem)) =>
    println(s"Executor $hostPort – max: $maxMem, free: $remainingMem")
}

```

This returns the memory status for each active executor process currently running on the Workers.

### Default Configuration Properties

Set cluster-wide defaults in [`conf/spark-defaults.conf`](https://github.com/apache/spark/blob/main/conf/spark-defaults.conf) to control resource allocation behavior:

```properties
spark.executor.cores=4
spark.cores.max=12
spark.deploy.defaultCores=2

```

These properties determine how the Master divides a Worker’s available cores among competing applications when explicit values are not provided at submit time.

## Summary

- **Workers** are persistent daemons that manage node resources and spawn executor processes as directed by the Master.
- **Executors** are transient JVM processes created per application to execute tasks and maintain in-memory storage, implemented in [`core/src/main/scala/org/apache/spark/executor/Executor.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/executor/Executor.scala).
- **Cores** define the parallel execution capacity; the Master allocates them to executors based on `spark.executor.cores`, while Workers track availability via `coresUsed`.
- The Standalone Master schedules executors by matching application resource requests against Worker advertisements stored in `WorkerInfo` and `ExecutorInfo` data structures.

## Frequently Asked Questions

### How many executors can run on a single Worker node?

A Worker can host multiple executors simultaneously as long as it has sufficient free cores and memory. The Master schedules executors from different applications or multiple executors from the same application (if `spark.executor.cores` is set lower than the Worker’s total cores) until the Worker’s `coresUsed` and `memoryUsed` reach their limits.

### What happens if I do not specify `spark.executor.cores`?

If `spark.executor.cores` is omitted, the Standalone Master allocates **all** available cores on the target Worker to that single executor. This can prevent other applications from running on that node until the executor terminates, effectively monopolizing the Worker’s compute capacity for one application.

### Can multiple Spark applications share the same executor process?

No. Executors are strictly bound to a single application. Each application submission triggers the Master to request new executor processes from Workers. When an application completes, its executors are terminated and their resources returned to the Worker pool for allocation to subsequent applications.

### How does the Master decide which Worker hosts a new executor?

The Master evaluates resource availability using the `WorkerInfo` class to compare advertised cores and memory against current `coresUsed` and `memoryUsed` values. It prioritizes Workers with sufficient capacity to satisfy the application’s `spark.executor.cores` and `spark.executor.memory` requirements, often favoring data locality when HDFS or other storage topology hints are provided.