# How to Set Up SSH Remote Storage for Distributed ML Pipelines in CMF

> Set up SSH remote storage for distributed ML pipelines in CMF using cmf init sshremote. Configure DVC and Git for artifact versioning across nodes easily.

- Repository: [Hewlett Packard Enterprise/cmf](https://github.com/hewlettpackard/cmf)
- Tags: how-to-guide
- Published: 2026-03-03

---

**To configure SSH remote storage in CMF (Continuous Machine-Learning Flow), run `cmf init sshremote` with your SSH credentials and path; this initializes a Git repository, configures DVC with a Paramiko-backed SSH remote, and stores connection details in `.dvc/config` for artifact versioning across distributed nodes.**

CMF (Continuous Machine-Learning Flow) by Hewlett Packard Enterprise enables teams to treat any remote Linux host as a secure artifact repository for distributed machine learning pipelines. By leveraging SSH remote storage, you can maintain a centralized, file-system-like store for large model artifacts and datasets while keeping code versioned in Git. This guide walks through the exact implementation details found in the `hewlettpackard/cmf` source code.

## Prerequisites and Installation

Before initializing SSH storage, ensure your environment meets the following requirements:

- **Python 3.7+** with CMF installed (installs Paramiko via `dvc[ssh]` extras as defined in [`setup.py`](https://github.com/hewlettpackard/cmf/blob/main/setup.py))
- **SSH access** to a remote Linux host with SFTP enabled
- **Git** installed locally for repository initialization
- **DVC** (Data Version Control) bundled with CMF or installed separately

The SSH backend relies on the **Paramiko** library to handle SFTP connections. When you install CMF, this dependency is automatically pulled in through the `dvc[ssh,s3]` extras specified in the project configuration.

## Initializing SSH Remote Storage

The `cmf init sshremote` command orchestrates the entire setup process across multiple configuration layers. According to the implementation in [`cmflib/commands/init/sshremote.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/commands/init/sshremote.py), the command performs Git bootstrap, DVC initialization, and credential storage in a single operation.

### Verify Clean State

First, confirm that CMF is not already configured in your current directory:

```bash
cmf init show

```

If unconfigured, this returns `'cmf' is not configured.` If initialized already, the command will display existing settings to prevent accidental overwrites.

### Run the Initialization Command

Execute the initialization with your SSH remote details. The `--path` must use the `ssh://` protocol format:

```bash
cmf init sshremote \
    --path ssh://127.0.0.1/home/user/ssh-storage \
    --user myuser \
    --port 22 \
    --password mypassword \
    --git-remote-url https://github.com/myorg/myrepo.git

```

**Required arguments:**
- `--path`: SSH URL pointing to the remote storage directory (e.g., `ssh://host/path/to/storage`)
- `--user`: SSH username for authentication
- `--port`: SSH server port (typically 22)
- `--password`: SSH password or private key content
- `--git-remote-url`: URL for the Git remote to track pipeline code

### Optional Neo4j Metadata Configuration

To simultaneously configure the CMF metadata server and Neo4j backend, append these arguments to the same command:

```bash
cmf init sshremote \
    --path ssh://127.0.0.1/home/user/ssh-storage \
    --user myuser \
    --port 22 \
    --password mypassword \
    --git-remote-url https://github.com/myorg/myrepo.git \
    --cmf-server-url http://127.0.0.1:80 \
    --neo4j-user neo4j \
    --neo4j-password secret \
    --neo4j-uri bolt://localhost:7687

```

This writes Neo4j connection details to the `neo4j` section of `.cmfconfig` while setting up the SSH storage backend.

## How SSH Remote Storage Works Under the Hood

Understanding the internal workflow helps debug connection issues and customize deployments. The initialization process follows a strict sequence defined in the source code.

### The Init Command Implementation

In [`cmflib/commands/init/sshremote.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/commands/init/sshremote.py), the `CmdInitSSHRemote.run` method (lines 53–100) performs four critical operations:

1. **Argument validation**: Extracts `--path`, `--user`, `--port`, and `--password`, raising `MissingArgument` or `DuplicateArgumentNotAllowed` if required parameters are absent (lines 53–59).

2. **Configuration file generation**: Writes the CMF server URL to the `cmf` section of `.cmfconfig`, using the `CmfConfig.write_config` method (lines 63–64). If Neo4j parameters exist, they populate the `neo4j` section (lines 66–73).

3. **Git repository bootstrap**: Checks for an existing Git repo via `is_git_repo()` (line 81). If none exists, CMF creates a new repository, checks out the `master` branch, makes an initial commit, and adds the remote Git URL (lines 84–89).

4. **DVC initialization and remote registration**: 
   - `dvc_quiet_init()` creates the `.dvc` directory (line 93)
   - `dvc_add_remote_repo('ssh-storage', path)` registers the remote in `.dvc/config` (lines 94–95)
   - `dvc_add_attribute` stores SSH credentials (`user`, `password`, `port`) in the DVC config under `remote.ssh-storage.*` (lines 98–100)

### The Storage Backend Implementation

When pipeline commands like `cmf pull` or `cmf push` execute, they invoke the **SSHremoteArtifacts** class defined in [`cmflib/storage_backends/sshremote_artifacts.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/storage_backends/sshremote_artifacts.py). This class implements the actual file transfer logic using Paramiko.

The initialization process (lines 25–27) reads credentials from the DVC config parsed by `DvcConfig.get_dvc_config()` (found in [`cmflib/utils/dvc_config.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/utils/dvc_config.py), lines 22–35):

```python

# DVC config structure generated by init

[remote "ssh-storage"]
url = ssh://127.0.0.1/home/user/ssh-storage
user = XXXXX
password = example@123
port = 22

```

**File upload process**: The `download_file` method (lines 37–68) creates a `paramiko.SSHClient`, authenticates with the stored credentials, establishes an SFTP session, creates necessary subdirectories recursively, and transfers files using `sftp.put`. It verifies successful transfer by comparing file sizes between local and remote.

**Directory handling**: For `.dir` artifacts (DVC directory placeholders), the `download_directory` method (lines 75–164) parses the `.dir` file to extract file lists, then iteratively uploads each component while preserving the DVC-style directory layout (`md5/xx/<hash>.dir`).

## Managing Artifacts with SSH Remote Storage

Once initialized, interact with the remote storage using standard CMF commands:

```bash

# Pull a specific artifact by ID

cmf pull --artifact-id 12345

# Pull all artifacts for an experiment

cmf pull --experiment my_experiment

# Push local artifacts to SSH remote

cmf push --artifact-id 12345

```

Under the hood, these commands route through the storage backend in [`cmflib/storage_backends/sshremote_artifacts.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/storage_backends/sshremote_artifacts.py), which authenticates via Paramiko and transfers files over SFTP using the credentials stored in `.dvc/config`.

## Troubleshooting Common SSH Connection Issues

| Symptom | Root Cause | Solution |
|---------|------------|----------|
| `ssh: connect to host … port 22: Connection refused` | Remote SSH service unreachable or firewall blocked | Verify SSH service status on the remote host and check `iptables` or cloud security group rules for inbound port 22 |
| `Authentication failed` | Invalid credentials in DVC config | Check stored credentials with `dvc remote list` and modify using `dvc remote modify ssh-storage user <username>` |
| `Permission denied (publickey)` | Remote requires key-based authentication | Update the `--password` parameter to contain your private key content, or modify [`sshremote_artifacts.py`](https://github.com/hewlettpackard/cmf/blob/main/sshremote_artifacts.py) to accept key file paths |
| Artifact download stalls | Network latency or insufficient remote disk space | Test SFTP connectivity manually with `sftp user@host` and verify available storage in the target directory |
| `CmfInitFailed` during initialization | Malformed SSH URL or non-existent remote path | Ensure `--path` uses valid `ssh://` format and points to an existing directory on the remote host |

## Summary

- **CMF SSH remote storage** enables secure artifact versioning on remote Linux hosts using Paramiko-based SFTP transfers
- The `cmf init sshremote` command automates Git repository creation, DVC initialization, and credential storage in [`cmflib/commands/init/sshremote.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/commands/init/sshremote.py)
- Connection parameters are stored in `.dvc/config` under the `remote.ssh-storage` section and parsed by [`cmflib/utils/dvc_config.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/utils/dvc_config.py)
- **SSHremoteArtifacts** in [`cmflib/storage_backends/sshremote_artifacts.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/storage_backends/sshremote_artifacts.py) handles actual file transfers with integrity verification
- Artifacts follow DVC's `md5/xx/<hash>.dir` layout on the remote server, ensuring compatibility with existing DVC ecosystems

## Frequently Asked Questions

### How does CMF authenticate with the SSH remote server?

CMF authenticates using the **Paramiko** library. During initialization, `cmf init sshremote` stores your username, password, and port in the DVC config (`remote.ssh-storage.*`). When transferring artifacts, the `SSHremoteArtifacts` class reads these values via `DvcConfig.get_dvc_config()` and creates an SSH client connection. Authentication is performed automatically without manual key management, though you can modify the backend to use private keys by adjusting the credentials passed to the Paramiko client.

### What files does CMF create during SSH remote initialization?

After running `cmf init sshremote`, your project directory contains four key components: `.git/` (versioned source code), `.dvc/` (DVC metadata and cache), `.dvc/config` (containing the SSH remote definition and credentials), and `.cmfconfig` (optional CMF server and Neo4j settings). The `.dvc/config` file specifically stores the SSH URL, user, password, and port required for subsequent connections.

### Can I use SSH key-based authentication instead of passwords?

The current implementation in [`cmflib/storage_backends/sshremote_artifacts.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/storage_backends/sshremote_artifacts.py) primarily supports password authentication via the `--password` argument. However, you can adapt the code to support key-based authentication by modifying the `SSHremoteArtifacts.__init__` method to accept a private key path and using Paramiko's `SSHClient.connect()` with the `key_filename` or `pkey` parameter instead of the `password` parameter.

### Why does my artifact pull fail with "Connection refused" errors?

This error indicates the Paramiko client cannot establish a TCP connection to the remote host on the specified port. Verify that the remote server is running an SSH daemon (sshd), the port specified in `--port` matches the server's listening port, and no firewalls block the connection. Additionally, ensure the path specified in `--path` uses the correct `ssh://` URL format and that the remote directory exists before initialization.