How to Set Up SSH Remote Storage for Distributed ML Pipelines in CMF

To configure SSH remote storage in CMF (Continuous Machine-Learning Flow), run cmf init sshremote with your SSH credentials and path; this initializes a Git repository, configures DVC with a Paramiko-backed SSH remote, and stores connection details in .dvc/config for artifact versioning across distributed nodes.

CMF (Continuous Machine-Learning Flow) by Hewlett Packard Enterprise enables teams to treat any remote Linux host as a secure artifact repository for distributed machine learning pipelines. By leveraging SSH remote storage, you can maintain a centralized, file-system-like store for large model artifacts and datasets while keeping code versioned in Git. This guide walks through the exact implementation details found in the hewlettpackard/cmf source code.

Prerequisites and Installation

Before initializing SSH storage, ensure your environment meets the following requirements:

• Python 3.7+ with CMF installed (installs Paramiko via dvc[ssh] extras as defined in setup.py)
• SSH access to a remote Linux host with SFTP enabled
• Git installed locally for repository initialization
• DVC (Data Version Control) bundled with CMF or installed separately

The SSH backend relies on the Paramiko library to handle SFTP connections. When you install CMF, this dependency is automatically pulled in through the dvc[ssh,s3] extras specified in the project configuration.

Initializing SSH Remote Storage

The cmf init sshremote command orchestrates the entire setup process across multiple configuration layers. According to the implementation in cmflib/commands/init/sshremote.py, the command performs Git bootstrap, DVC initialization, and credential storage in a single operation.

Verify Clean State

First, confirm that CMF is not already configured in your current directory:

cmf init show

If unconfigured, this returns 'cmf' is not configured. If initialized already, the command will display existing settings to prevent accidental overwrites.

Run the Initialization Command

Execute the initialization with your SSH remote details. The --path must use the ssh:// protocol format:

cmf init sshremote \
    --path ssh://127.0.0.1/home/user/ssh-storage \
    --user myuser \
    --port 22 \
    --password mypassword \
    --git-remote-url https://github.com/myorg/myrepo.git

Required arguments:

• --path: SSH URL pointing to the remote storage directory (e.g., ssh://host/path/to/storage)
• --user: SSH username for authentication
• --port: SSH server port (typically 22)
• --password: SSH password or private key content
• --git-remote-url: URL for the Git remote to track pipeline code

Optional Neo4j Metadata Configuration

To simultaneously configure the CMF metadata server and Neo4j backend, append these arguments to the same command:

cmf init sshremote \
    --path ssh://127.0.0.1/home/user/ssh-storage \
    --user myuser \
    --port 22 \
    --password mypassword \
    --git-remote-url https://github.com/myorg/myrepo.git \
    --cmf-server-url http://127.0.0.1:80 \
    --neo4j-user neo4j \
    --neo4j-password secret \
    --neo4j-uri bolt://localhost:7687

This writes Neo4j connection details to the neo4j section of .cmfconfig while setting up the SSH storage backend.

How SSH Remote Storage Works Under the Hood

Understanding the internal workflow helps debug connection issues and customize deployments. The initialization process follows a strict sequence defined in the source code.

The Init Command Implementation

In cmflib/commands/init/sshremote.py, the CmdInitSSHRemote.run method (lines 53–100) performs four critical operations:

1. Argument validation: Extracts --path, --user, --port, and --password, raising MissingArgument or DuplicateArgumentNotAllowed if required parameters are absent (lines 53–59).

2. Configuration file generation: Writes the CMF server URL to the cmf section of .cmfconfig, using the CmfConfig.write_config method (lines 63–64). If Neo4j parameters exist, they populate the neo4j section (lines 66–73).

3. Git repository bootstrap: Checks for an existing Git repo via is_git_repo() (line 81). If none exists, CMF creates a new repository, checks out the master branch, makes an initial commit, and adds the remote Git URL (lines 84–89).

4. DVC initialization and remote registration:

   • dvc_quiet_init() creates the .dvc directory (line 93)
   • dvc_add_remote_repo('ssh-storage', path) registers the remote in .dvc/config (lines 94–95)
   • dvc_add_attribute stores SSH credentials (user, password, port) in the DVC config under remote.ssh-storage.* (lines 98–100)

The Storage Backend Implementation

When pipeline commands like cmf pull or cmf push execute, they invoke the SSHremoteArtifacts class defined in cmflib/storage_backends/sshremote_artifacts.py. This class implements the actual file transfer logic using Paramiko.

The initialization process (lines 25–27) reads credentials from the DVC config parsed by DvcConfig.get_dvc_config() (found in cmflib/utils/dvc_config.py, lines 22–35):

# DVC config structure generated by init

[remote "ssh-storage"]
url = ssh://127.0.0.1/home/user/ssh-storage
user = XXXXX
password = example@123
port = 22

File upload process: The download_file method (lines 37–68) creates a paramiko.SSHClient, authenticates with the stored credentials, establishes an SFTP session, creates necessary subdirectories recursively, and transfers files using sftp.put. It verifies successful transfer by comparing file sizes between local and remote.

Directory handling: For .dir artifacts (DVC directory placeholders), the download_directory method (lines 75–164) parses the .dir file to extract file lists, then iteratively uploads each component while preserving the DVC-style directory layout (md5/xx/<hash>.dir).

Managing Artifacts with SSH Remote Storage

Once initialized, interact with the remote storage using standard CMF commands:

# Pull a specific artifact by ID

cmf pull --artifact-id 12345

# Pull all artifacts for an experiment

cmf pull --experiment my_experiment

# Push local artifacts to SSH remote

cmf push --artifact-id 12345

Under the hood, these commands route through the storage backend in cmflib/storage_backends/sshremote_artifacts.py, which authenticates via Paramiko and transfers files over SFTP using the credentials stored in .dvc/config.

Troubleshooting Common SSH Connection Issues

Symptom	Root Cause	Solution
ssh: connect to host … port 22: Connection refused	Remote SSH service unreachable or firewall blocked	Verify SSH service status on the remote host and check iptables or cloud security group rules for inbound port 22
Authentication failed	Invalid credentials in DVC config	Check stored credentials with dvc remote list and modify using dvc remote modify ssh-storage user <username>
Permission denied (publickey)	Remote requires key-based authentication	Update the --password parameter to contain your private key content, or modify sshremote_artifacts.py to accept key file paths
Artifact download stalls	Network latency or insufficient remote disk space	Test SFTP connectivity manually with sftp user@host and verify available storage in the target directory
CmfInitFailed during initialization	Malformed SSH URL or non-existent remote path	Ensure --path uses valid ssh:// format and points to an existing directory on the remote host

Summary

• CMF SSH remote storage enables secure artifact versioning on remote Linux hosts using Paramiko-based SFTP transfers
• The cmf init sshremote command automates Git repository creation, DVC initialization, and credential storage in cmflib/commands/init/sshremote.py
• Connection parameters are stored in .dvc/config under the remote.ssh-storage section and parsed by cmflib/utils/dvc_config.py
• SSHremoteArtifacts in cmflib/storage_backends/sshremote_artifacts.py handles actual file transfers with integrity verification
• Artifacts follow DVC's md5/xx/<hash>.dir layout on the remote server, ensuring compatibility with existing DVC ecosystems

Frequently Asked Questions

How does CMF authenticate with the SSH remote server?

CMF authenticates using the Paramiko library. During initialization, cmf init sshremote stores your username, password, and port in the DVC config (remote.ssh-storage.*). When transferring artifacts, the SSHremoteArtifacts class reads these values via DvcConfig.get_dvc_config() and creates an SSH client connection. Authentication is performed automatically without manual key management, though you can modify the backend to use private keys by adjusting the credentials passed to the Paramiko client.

What files does CMF create during SSH remote initialization?

After running cmf init sshremote, your project directory contains four key components: .git/ (versioned source code), .dvc/ (DVC metadata and cache), .dvc/config (containing the SSH remote definition and credentials), and .cmfconfig (optional CMF server and Neo4j settings). The .dvc/config file specifically stores the SSH URL, user, password, and port required for subsequent connections.

Can I use SSH key-based authentication instead of passwords?

The current implementation in cmflib/storage_backends/sshremote_artifacts.py primarily supports password authentication via the --password argument. However, you can adapt the code to support key-based authentication by modifying the SSHremoteArtifacts.__init__ method to accept a private key path and using Paramiko's SSHClient.connect() with the key_filename or pkey parameter instead of the password parameter.

Why does my artifact pull fail with "Connection refused" errors?

This error indicates the Paramiko client cannot establish a TCP connection to the remote host on the specified port. Verify that the remote server is running an SSH daemon (sshd), the port specified in --port matches the server's listening port, and no firewalls block the connection. Additionally, ensure the path specified in --path uses the correct ssh:// URL format and that the remote directory exists before initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}