How to Read and Write HDF5 Files in Nelson: Complete Guide to .nh5 Workspace Format

Nelson natively reads and writes HDF5-based .nh5 workspace files using the load() and save() functions, but requires external tools or FUSE mounts to access data stored on HDFS (Hadoop Distributed File System) clusters.

Nelson is an open-source numerical computing environment that persists workspaces in the HDF5-based .nh5 format by default. While the software provides robust native support for reading and writing these HDF5 files, it is important to distinguish between HDF5 (the file format) and HDFS (the distributed filesystem), as Nelson does not include native Hadoop client integration.

Understanding Nelson's HDF5-Based .nh5 Format

Nelson stores workspaces in the HDF5-based format .nh5 (the default) and also supports MATLAB's .mat format for interoperability. The .nh5 extension indicates a standard HDF5 file structure that contains serialized Nelson variables, including matrices, structs, and other data types.

The core implementation resides in the stream manager module, specifically in modules/stream_manager/builtin/cpp/saveBuiltin.cpp for writing files and modules/stream_manager/builtin/cpp/loadBuiltin.cpp for reading them. These functions interface directly with the HDF5 library using standard filesystem APIs.

Native HDF5 Operations: Using load() and save()

Nelson provides straightforward built-in functions for HDF5 I/O operations on local files.

Saving Workspaces to .nh5

The save() function writes variables to an HDF5 file. When you specify the .nh5 extension, Nelson uses its native HDF5 format.

% Save all workspace variables to HDF5
save('myworkspace.nh5');

% Save specific variables only
save('data.nh5', 'var1', 'var2');

% Save with explicit format (though .nh5 is default)
save('backup.nh5', '-nh5');

Loading .nh5 Workspaces

The load() function restores variables from HDF5 files into the current workspace.

% Load all variables from an HDF5 workspace
load('myworkspace.nh5');

% Load specific variables only
load('data.nh5', 'var1', 'var2');

% Load into a structure instead of workspace
S = load('myworkspace.nh5');

Critical Distinction: No Native HDFS Support

Despite the similar acronyms, HDF5 (Hierarchical Data Format 5) and HDFS (Hadoop Distributed File System) are entirely different technologies. Nelson does not contain any HDFS integration—a search for "HDFS", "hdfs://" or related symbols in the nelson-lang/nelson codebase returns no relevant implementation.

The saveBuiltin.cpp and loadBuiltin.cpp files in modules/stream_manager/builtin/cpp/ rely on standard POSIX or Windows filesystem APIs and do not link against Hadoop client libraries (such as libhdfs). Consequently, any path supplied to load() or save() must be reachable by the operating system as a regular local file. You cannot pass an hdfs:// URI directly to these functions.

Workaround: Accessing HDFS-Hosted Files

To work with Nelson workspaces stored on HDFS, you must use an external bridge to transfer files between the distributed filesystem and local storage.

Method 1: Hadoop CLI Bridge

Use the Hadoop command-line interface to copy files to a temporary local location, process them in Nelson, then push them back to HDFS.

% Step 1: Copy HDFS file to local temp
system('hdfs dfs -copyToLocal hdfs:///user/me/project/workspace.nh5 /tmp/workspace.nh5');

% Step 2: Load into Nelson workspace
load('/tmp/workspace.nh5');
% ... perform analysis and modify variables ...
a = a + 1;

% Step 3: Save modified workspace locally
save('/tmp/workspace.nh5');

% Step 4: Copy back to HDFS (force overwrite)
system('hdfs dfs -copyFromLocal -f /tmp/workspace.nh5 hdfs:///user/me/project/workspace.nh5');

Method 2: FUSE Mount

If your system administrator has configured an HDFS FUSE mount (such as hdfs-fuse or hadoop-fuse-dfs), the HDFS cluster appears as a local directory. Nelson can then read and write .nh5 files directly through the mount point.

% Access HDFS through FUSE mount at /mnt/hdfs
load('/mnt/hdfs/user/me/project/workspace.nh5');

% ... modify data ...

% Write back through FUSE
save('/mnt/hdfs/user/me/project/workspace.nh5');

This approach eliminates manual copy operations but requires proper FUSE configuration and may have performance implications for large datasets.

Key Source Files and Implementation Details

Understanding the underlying source code clarifies why HDFS support requires external tools:

• modules/stream_manager/builtin/cpp/saveBuiltin.cpp – Implements the save function, handling file extension detection and HDF5 library operations for .nh5 files.
• modules/stream_manager/builtin/cpp/loadBuiltin.cpp – Implements the load function, detecting .nh5 or .mat formats and restoring variables via the HDF5 library.
• modules/hdf5/tests/* – Comprehensive test suite verifying HDF5 read/write operations.
• modules/stream_manager/help/en_US/xml/save.xml & load.xml – Official documentation specifying .nh5 as the default HDF5-based format.

These implementations rely on standard file descriptors and do not include Hadoop client code (such as libhdfs), confirming that hdfs:// URIs cannot be passed directly to Nelson's I/O functions.

Summary

• Nelson natively supports HDF5 through the .nh5 workspace format using load() and save() functions.
• HDFS (Hadoop Distributed File System) is not natively supported; Nelson requires local file paths.
• To process HDFS-hosted data, use external Hadoop CLI tools (hdfs dfs -copyToLocal / -copyFromLocal) or configure an HDFS FUSE mount.
• Core I/O implementation resides in modules/stream_manager/builtin/cpp/saveBuiltin.cpp and loadBuiltin.cpp, utilizing standard filesystem APIs without Hadoop integration.

Frequently Asked Questions

Can Nelson read files directly from an HDFS cluster using hdfs:// URIs?

No. Nelson does not include Hadoop client libraries. The load() and save() functions implemented in modules/stream_manager/builtin/cpp/loadBuiltin.cpp and saveBuiltin.cpp operate only on local filesystem paths. To access HDFS data, you must first copy files to local storage using hdfs dfs commands or mount HDFS via FUSE.

What is the difference between .nh5 and .mat formats in Nelson?

The .nh5 format is Nelson's native HDF5-based workspace format, optimized for Nelson-specific data types and the default for save() operations. The .mat format provides compatibility with MATLAB. Both are handled by the same underlying I/O functions in the stream manager module, but .nh5 preserves Nelson-specific features more completely.

Is there a performance difference between using Hadoop CLI copies versus FUSE mounts?

Yes. Using hdfs dfs -copyToLocal and -copyFromLocal (Method 1) involves explicit data transfer and temporary local storage, which may be slower for iterative workflows but offers better reliability. FUSE mounts (Method 2) allow direct access but may introduce latency for random access patterns and depend on the stability of the FUSE daemon. For large .nh5 files, explicit copies are generally more robust.

Will future Nelson releases add native HDFS support?

Currently, the codebase contains no Hadoop integration. Adding native HDFS support would require linking against libhdfs and implementing virtual file drivers for the HDF5 library to recognize hdfs:// URIs. Until such an extension is added to the stream manager module, the recommended approach remains using external HDFS tools or FUSE mounts as described above.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}