# What Is an RDD in Spark? Core Characteristics and Architecture Explained

> Discover RDD in Spark, the core abstraction for big data processing. Learn its immutable, partitioned nature, lazy evaluation, fault tolerance, and parallel operations.

- Repository: [The Apache Software Foundation/spark](https://github.com/apache/spark)
- Tags: deep-dive
- Published: 2026-02-16

---

**A Resilient Distributed Dataset (RDD) is Spark's fundamental abstraction for big data processing—an immutable, partitioned collection of elements that supports lazy evaluation, fault-tolerant lineage, and parallel operations across a cluster.**

The Apache Spark repository (`apache/spark`) implements the RDD as the foundational data structure upon which all higher-level APIs—DataFrames, Datasets, and Structured Streaming—are built. Understanding what an RDD in Spark represents and its core characteristics is essential for optimizing distributed data pipelines and debugging performance bottlenecks at the source code level.

## What Is an RDD in Spark?

An **RDD** (Resilient Distributed Dataset) is a read-only, partitioned collection of records distributed across a cluster that can be operated on in parallel. According to the source code in [`core/src/main/scala/org/apache/spark/rdd/RDD.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/rdd/RDD.scala), the abstract class declaration `abstract class RDD[T: ClassTag]` establishes RDDs as generic, typed data structures that serve as the primary interface for Spark's execution engine.

Unlike traditional in-memory collections, RDDs maintain metadata about their computational lineage—the sequence of transformations used to derive them—enabling automatic recovery from node failures without requiring replicated storage.

## Core Characteristics of RDDs in Spark

### Immutability and Functional Transformations

Once created, an RDD cannot be modified in place. As noted at [line 57 of [`RDD.scala`](https://github.com/apache/spark/blob/main/RDD.scala)](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L57), transformations such as `map()` and `filter()` always generate new RDD instances rather than mutating the original dataset. This immutability ensures that the lineage graph remains consistent and enables safe sharing of RDD references across multiple threads or applications.

### Partitioned Parallel Processing

RDDs are physically divided into **partitions**—logical chunks of data that can be processed independently on different executors. The source code at [lines 71-76 of [`RDD.scala`](https://github.com/apache/spark/blob/main/RDD.scala)](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L71-L76) defines the partitioning interface, allowing the scheduler to distribute tasks across the cluster based on data locality and available resources.

### Lazy Evaluation and Execution Planning

Spark employs **lazy evaluation** for RDD transformations. When you call `map()` or `filter()`, Spark does not immediately execute the computation; instead, it records the operation in a **Directed Acyclic Graph (DAG)** of dependencies. The actual computation triggers only when an **action** (such as `collect()` or `count()`) is invoked. This deferred execution allows the Catalyst optimizer and the scheduler to pipeline operations and minimize data shuffling.

### Fault Tolerance Through Lineage

The **resilience** in Resilient Distributed Dataset comes from lineage-based fault tolerance. Each RDD maintains references to its parent RDDs through the `dependencies` method and `getNarrowAncestors` logic in [`RDD.scala`](https://github.com/apache/spark/blob/main/RDD.scala). If a partition is lost due to node failure, Spark recomputes only that specific partition by replaying the transformations recorded in the dependency graph—no data replication to disk is required unless explicitly checkpointed.

### Persistence and Caching Strategies

RDDs support **in-memory persistence** to accelerate iterative machine learning algorithms and interactive data exploration. The `persist()` and `cache()` methods (defined around lines 60-67 in [`RDD.scala`](https://github.com/apache/spark/blob/main/RDD.scala)) allow users to materialize datasets in memory, on disk, or both, with configurable storage levels. Once persisted, subsequent actions on the RDD read directly from the cached partitions rather than recomputing from source.

### Type Safety with ClassTag

Unlike the untyped DataFrame API, RDDs are **strongly typed** using Scala generics (`RDD[T]`) and carry a `ClassTag` for efficient serialization across the JVM boundary. This compile-time type safety catches errors during development but requires careful handling of custom classes to ensure serializability for network transfer.

### Dependency Types: Narrow vs. Wide

Spark distinguishes between two dependency patterns that determine data movement costs:

- **Narrow dependencies** (e.g., `map`, `filter`): Each child partition depends on a small, known set of parent partitions (often one-to-one). These operations require no data shuffling and are pipelined efficiently.
- **Wide dependencies** (e.g., `shuffle`, `groupByKey`): Child partitions depend on data from all parent partitions, necessitating a **shuffle** operation that moves data across the network.

The `Dependency` trait hierarchy in [`core/src/main/scala/org/apache/spark/rdd/Dependency.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/rdd/Dependency.scala) formalizes these relationships, with `NarrowDependency` and `ShuffleDependency` subclasses driving the scheduler's task placement decisions.

## RDD Operations: Transformations and Actions

Spark operations divide into two categories that govern execution timing:

**Transformations** are lazy operations that return a new RDD, including `map()`, `filter()`, `flatMap()`, and `reduceByKey()`. These methods, typically implemented via `withScope` wrappers that instantiate classes like `MapPartitionsRDD`, merely record the computation in the lineage graph.

**Actions** trigger actual job execution and return results to the driver program or write data to external storage. Common actions include `collect()`, `count()`, `reduce()`, `saveAsTextFile()`, and `foreach()`. When an action is called, the `DAGScheduler` uses the RDD's partition and dependency metadata to stage tasks across the cluster.

## Source Code Architecture

The RDD implementation spans several critical files in the `apache/spark` repository:

- **[`core/src/main/scala/org/apache/spark/rdd/RDD.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/rdd/RDD.scala)**: The core abstract class defining the public API, partitioning logic, persistence mechanisms, and lineage tracking.
- **[`core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala)**: Concrete implementation produced by narrow transformations such as `map` and `filter`, handling the actual data processing logic.
- **[`core/src/main/scala/org/apache/spark/rdd/Dependency.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/rdd/Dependency.scala)**: Defines the `Dependency` trait hierarchy that captures parent-child relationships and determines shuffle requirements.
- **[`core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala`](https://github.com/apache/spark/blob/main/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala)**: Provides checkpointing support to truncate lineage graphs for long-running iterative jobs, saving materialized data to reliable storage.

## Practical Example: Creating and Using RDDs in Spark

The following Scala example demonstrates RDD creation, transformations, persistence, and actions using the Spark Core API:

```scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("RDD Demo")
  .master("local[*]")
  .getOrCreate()
val sc = spark.sparkContext

// Create an RDD from a local collection with 4 partitions
val numbers = sc.parallelize(1 to 10, numSlices = 4)

// Transformations (lazy evaluation - no computation yet)
val evens = numbers.filter(_ % 2 == 0)
val squares = evens.map(x => x * x)

// Persist in memory to avoid recomputation for multiple actions
squares.persist()

// Actions trigger execution and return results to driver
val resultArray = squares.collect()
println(s"Squares of even numbers: ${resultArray.mkString(", ")}")

val count = squares.count()
println(s"Number of elements: $count")

```

This example illustrates **immutability** (each transformation produces a new RDD), **lazy evaluation** (computation occurs only at `collect()` and `count()`), **partitioning** (four slices distributed across threads), and **fault tolerance** (if a partition fails during execution, Spark recomputes it using the recorded lineage from `numbers` to `squares`).

## Summary

- An **RDD in Spark** is an immutable, partitioned, distributed collection that serves as the engine's fundamental data abstraction.
- **Lazy evaluation** defers execution until actions are called, enabling optimization of the underlying execution plan.
- **Lineage tracking** provides fault tolerance by allowing Spark to recompute lost partitions from parent RDDs rather than replicating data.
- **Narrow and wide dependencies** determine whether operations can be pipelined locally or require expensive shuffle operations across the network.
- **Persistence methods** like `cache()` and `persist()` materialize RDDs in memory or disk to accelerate iterative workloads.

## Frequently Asked Questions

### What is the difference between an RDD and a DataFrame in Spark?

An RDD is a low-level, typed collection of JVM objects with compile-time type safety but no schema awareness, while a DataFrame is a distributed table of rows with named columns and a schema that enables Catalyst optimizer optimizations. DataFrames generally offer better performance due to code generation and optimized physical plans, whereas RDDs provide finer control over data partitioning and custom data types.

### How does Spark recover lost RDD partitions?

Spark recovers lost partitions using **lineage-based recomputation**. Each RDD maintains a reference to its parent RDDs through the `dependencies` method defined in [`RDD.scala`](https://github.com/apache/spark/blob/main/RDD.scala). When a node fails, the scheduler identifies the missing partitions and re-executes only the specific transformations required to reconstruct them from the original source data or cached checkpoints, rather than restoring from replicated backups.

### When should I use persist() or cache() on an RDD?

Use `persist()` or `cache()` when an RDD will be accessed multiple times in iterative algorithms or interactive queries, as materializing the dataset avoids recomputing the lineage from source for each action. Choose `cache()` for simple in-memory storage, or `persist(StorageLevel)` to specify disk-only, memory-only, or memory-and-disk replication strategies based on your memory constraints and access patterns.

### What are narrow and wide dependencies in Spark RDDs?

**Narrow dependencies** occur when each child partition depends on a limited, known set of parent partitions (e.g., `map`, `filter`), allowing pipelined execution without data movement. **Wide dependencies** occur when child partitions require data from all parent partitions (e.g., `groupByKey`, `reduceByKey`), necessitating a shuffle operation that redistributes data across the cluster. The dependency type is determined by the `Dependency` trait implementation and directly impacts task scheduling and network traffic.