# How to Read and Write HDF5 Files in Nelson: Complete Guide to .nh5 Workspace Format

> Learn to read and write HDF5 files in Nelson using load and save functions. Explore the .nh5 workspace format and HDFS integration. Get direct access to your data.

- Repository: [The Nelson Programming Language/nelson](https://github.com/nelson-lang/nelson)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Nelson natively reads and writes HDF5-based `.nh5` workspace files using the `load()` and `save()` functions, but requires external tools or FUSE mounts to access data stored on HDFS (Hadoop Distributed File System) clusters.**

Nelson is an open-source numerical computing environment that persists workspaces in the HDF5-based `.nh5` format by default. While the software provides robust native support for reading and writing these HDF5 files, it is important to distinguish between HDF5 (the file format) and HDFS (the distributed filesystem), as Nelson does not include native Hadoop client integration.

## Understanding Nelson's HDF5-Based .nh5 Format

Nelson stores workspaces in the **HDF5-based** format `.nh5` (the default) and also supports MATLAB's `.mat` format for interoperability. The `.nh5` extension indicates a standard HDF5 file structure that contains serialized Nelson variables, including matrices, structs, and other data types.

The core implementation resides in the stream manager module, specifically in [`modules/stream_manager/builtin/cpp/saveBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/stream_manager/builtin/cpp/saveBuiltin.cpp) for writing files and [`modules/stream_manager/builtin/cpp/loadBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/stream_manager/builtin/cpp/loadBuiltin.cpp) for reading them. These functions interface directly with the HDF5 library using standard filesystem APIs.

## Native HDF5 Operations: Using load() and save()

Nelson provides straightforward built-in functions for HDF5 I/O operations on local files.

### Saving Workspaces to .nh5

The `save()` function writes variables to an HDF5 file. When you specify the `.nh5` extension, Nelson uses its native HDF5 format.

```nelson
% Save all workspace variables to HDF5
save('myworkspace.nh5');

% Save specific variables only
save('data.nh5', 'var1', 'var2');

% Save with explicit format (though .nh5 is default)
save('backup.nh5', '-nh5');

```

### Loading .nh5 Workspaces

The `load()` function restores variables from HDF5 files into the current workspace.

```nelson
% Load all variables from an HDF5 workspace
load('myworkspace.nh5');

% Load specific variables only
load('data.nh5', 'var1', 'var2');

% Load into a structure instead of workspace
S = load('myworkspace.nh5');

```

## Critical Distinction: No Native HDFS Support

Despite the similar acronyms, **HDF5** (Hierarchical Data Format 5) and **HDFS** (Hadoop Distributed File System) are entirely different technologies. Nelson does not contain any HDFS integration—a search for "HDFS", "hdfs://" or related symbols in the nelson-lang/nelson codebase returns no relevant implementation.

The [`saveBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/saveBuiltin.cpp) and [`loadBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/loadBuiltin.cpp) files in `modules/stream_manager/builtin/cpp/` rely on standard POSIX or Windows filesystem APIs and do not link against Hadoop client libraries (such as libhdfs). Consequently, any path supplied to `load()` or `save()` must be reachable by the operating system as a regular local file. You cannot pass an `hdfs://` URI directly to these functions.

## Workaround: Accessing HDFS-Hosted Files

To work with Nelson workspaces stored on HDFS, you must use an external bridge to transfer files between the distributed filesystem and local storage.

### Method 1: Hadoop CLI Bridge

Use the Hadoop command-line interface to copy files to a temporary local location, process them in Nelson, then push them back to HDFS.

```nelson
% Step 1: Copy HDFS file to local temp
system('hdfs dfs -copyToLocal hdfs:///user/me/project/workspace.nh5 /tmp/workspace.nh5');

% Step 2: Load into Nelson workspace
load('/tmp/workspace.nh5');
% ... perform analysis and modify variables ...
a = a + 1;

% Step 3: Save modified workspace locally
save('/tmp/workspace.nh5');

% Step 4: Copy back to HDFS (force overwrite)
system('hdfs dfs -copyFromLocal -f /tmp/workspace.nh5 hdfs:///user/me/project/workspace.nh5');

```

### Method 2: FUSE Mount

If your system administrator has configured an HDFS FUSE mount (such as `hdfs-fuse` or `hadoop-fuse-dfs`), the HDFS cluster appears as a local directory. Nelson can then read and write `.nh5` files directly through the mount point.

```nelson
% Access HDFS through FUSE mount at /mnt/hdfs
load('/mnt/hdfs/user/me/project/workspace.nh5');

% ... modify data ...

% Write back through FUSE
save('/mnt/hdfs/user/me/project/workspace.nh5');

```

This approach eliminates manual copy operations but requires proper FUSE configuration and may have performance implications for large datasets.

## Key Source Files and Implementation Details

Understanding the underlying source code clarifies why HDFS support requires external tools:

- **[`modules/stream_manager/builtin/cpp/saveBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/stream_manager/builtin/cpp/saveBuiltin.cpp)** – Implements the `save` function, handling file extension detection and HDF5 library operations for `.nh5` files.
- **[`modules/stream_manager/builtin/cpp/loadBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/stream_manager/builtin/cpp/loadBuiltin.cpp)** – Implements the `load` function, detecting `.nh5` or `.mat` formats and restoring variables via the HDF5 library.
- **`modules/hdf5/tests/*`** – Comprehensive test suite verifying HDF5 read/write operations.
- **[`modules/stream_manager/help/en_US/xml/save.xml`](https://github.com/nelson-lang/nelson/blob/main/modules/stream_manager/help/en_US/xml/save.xml) & [`load.xml`](https://github.com/nelson-lang/nelson/blob/main/load.xml)** – Official documentation specifying `.nh5` as the default HDF5-based format.

These implementations rely on standard file descriptors and do not include Hadoop client code (such as libhdfs), confirming that `hdfs://` URIs cannot be passed directly to Nelson's I/O functions.

## Summary

- Nelson natively supports **HDF5** through the `.nh5` workspace format using `load()` and `save()` functions.
- **HDFS** (Hadoop Distributed File System) is not natively supported; Nelson requires local file paths.
- To process HDFS-hosted data, use external Hadoop CLI tools (`hdfs dfs -copyToLocal` / `-copyFromLocal`) or configure an HDFS FUSE mount.
- Core I/O implementation resides in [`modules/stream_manager/builtin/cpp/saveBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/stream_manager/builtin/cpp/saveBuiltin.cpp) and [`loadBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/loadBuiltin.cpp), utilizing standard filesystem APIs without Hadoop integration.

## Frequently Asked Questions

### Can Nelson read files directly from an HDFS cluster using hdfs:// URIs?

No. Nelson does not include Hadoop client libraries. The `load()` and `save()` functions implemented in [`modules/stream_manager/builtin/cpp/loadBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/stream_manager/builtin/cpp/loadBuiltin.cpp) and [`saveBuiltin.cpp`](https://github.com/nelson-lang/nelson/blob/main/saveBuiltin.cpp) operate only on local filesystem paths. To access HDFS data, you must first copy files to local storage using `hdfs dfs` commands or mount HDFS via FUSE.

### What is the difference between .nh5 and .mat formats in Nelson?

The `.nh5` format is Nelson's native HDF5-based workspace format, optimized for Nelson-specific data types and the default for `save()` operations. The `.mat` format provides compatibility with MATLAB. Both are handled by the same underlying I/O functions in the stream manager module, but `.nh5` preserves Nelson-specific features more completely.

### Is there a performance difference between using Hadoop CLI copies versus FUSE mounts?

Yes. Using `hdfs dfs -copyToLocal` and `-copyFromLocal` (Method 1) involves explicit data transfer and temporary local storage, which may be slower for iterative workflows but offers better reliability. FUSE mounts (Method 2) allow direct access but may introduce latency for random access patterns and depend on the stability of the FUSE daemon. For large `.nh5` files, explicit copies are generally more robust.

### Will future Nelson releases add native HDFS support?

Currently, the codebase contains no Hadoop integration. Adding native HDFS support would require linking against libhdfs and implementing virtual file drivers for the HDF5 library to recognize `hdfs://` URIs. Until such an extension is added to the stream manager module, the recommended approach remains using external HDFS tools or FUSE mounts as described above.