# Configuring sdata for Cloud Storage Backends: MinIO, S3, and OwnCloud Integration Guide

> Integrate MinIO, S3, and OwnCloud cloud storage with sdata. This guide shows how to configure wrapper classes for seamless backend integration using environment variables or explicit settings.

- Repository: [lepy/sdata](https://github.com/lepy/sdata)
- Tags: how-to-guide
- Published: 2026-03-05

---

**sdata provides ready-to-use wrapper classes—`MinioBucket` for S3-compatible storage and `OwnCloudVault` for WebDAV—that enable seamless cloud storage integration through environment variables or explicit configuration.**

The `lepy/sdata` library simplifies data serialization by offering native support for popular cloud storage protocols. Configuring sdata for cloud storage backends allows you to persist large binary assets and datasets directly to MinIO, Amazon S3-compatible services, or OwnCloud shares without modifying your core application logic. Both implementations follow a thin-wrapper pattern that delegates heavy lifting to underlying libraries while exposing Pythonic APIs for `Path`, `bytes`, and `BinaryIO` types.

## MinIO and S3-Compatible Storage Configuration

The `MinioBucket` class in [`sdata/iolib/minio.py`](https://github.com/lepy/sdata/blob/main/sdata/iolib/minio.py) provides a complete abstraction over MinIO and other S3-compatible object stores, including Amazon S3, Wasabi, and DigitalOcean Spaces.

### Authentication and Client Initialization

The `MinioBucket.__init__` method establishes the client connection and guarantees bucket existence upon instantiation. You can supply credentials explicitly or rely on environment variables for secure deployment.

**Configuration options:**

- **Explicit**: Pass `access_key` and `secret_key` directly to the constructor
- **Environment variables**: Set `MINIO_ACCESS_KEY_ID` and `MINIO_SECRET_ACCESS_KEY` in your environment; the class automatically falls back to these when credentials are omitted
- **Endpoint**: Specify hostname and port (e.g., `"minio.mycompany.com:9000"`)
- **Secure flag**: Set `secure=False` for HTTP connections to local or internal MinIO instances

```python
from sdata.iolib.minio import MinioBucket

# Initialize with explicit credentials or env var fallback

minio = MinioBucket(
    endpoint="minio.mycompany.com:9000",
    bucket="my-data",
    access_key="my-key",      # Optional: falls back to MINIO_ACCESS_KEY_ID

    secret_key="my-secret",   # Optional: falls back to MINIO_SECRET_ACCESS_KEY

    secure=False
)

```

As implemented in the source code, the constructor automatically checks `bucket_exists` and creates the bucket if missing, ensuring the storage location is ready for immediate use.

### Core Storage Operations

The class provides high-level methods that translate between Python types and the storage client API. According to [`sdata/iolib/minio.py`](https://github.com/lepy/sdata/blob/main/sdata/iolib/minio.py), key operations include:

- **`objectnames()`**: Returns a list of object names in the bucket
- **`list_df()`**: Returns a pandas DataFrame with filename, size, modification time, and suffix (source: lines 81-85)
- **`upload()`**: Uploads local files with optional `force` parameter to control overwrite behavior
- **`upload_bytes()`**: Streams binary data directly without temporary files (source: lines 97-108)
- **`download()`** and **`delete()`**: Retrieve and remove objects by key

```python
from pathlib import Path

# List objects as DataFrame for inspection

print(minio.list_df().head())

# Upload local file (force=False skips existing files)

minio.upload(Path("samples/data.csv"), force=False)

# Stream large in-memory payloads directly

payload = b"binary data" * 100_000
minio.upload_bytes(payload, "large_blob.bin", force=True)

# Download to specific path

minio.download("data.csv", Path("downloads/local_copy.csv"))

# Clean up

minio.delete("large_blob.bin")

```

## OwnCloud WebDAV Integration

For OwnCloud integration, sdata provides `OwnCloudVault` in [`sdata/iolib/owncloudfs.py`](https://github.com/lepy/sdata/blob/main/sdata/iolib/owncloudfs.py), which connects to public shares via the WebDAV protocol.

### Connecting to Public Shares

The `OwnCloudVault.__init__` method (source: lines 19-33) establishes the connection using three parameters: `hostname`, `share_id`, and optional `password`. Unlike MinIO, this backend requires no bucket creation step, as the public share URL is immutable.

```python
from sdata.iolib.owncloudfs import OwnCloudVault

# Connect to a public OwnCloud share

vault = OwnCloudVault(
    hostname="https://cloud.example.com",
    share_id="abcdef1234567890",  # Token from the public link

    password=""                  # Leave empty if the share has no password

)

```

### File Synchronization

The implementation supports bidirectional folder synchronization and single-file operations:

- **`upload_file()`**: Uploads a single local file to a remote path
- **`download_file()`**: Retrieves a remote file to a local path
- **`sync_local_to_remote()`**: Recursively uploads an entire local directory tree (source: lines 64-74)
- **`sync_remote_to_local()`**: Downloads the complete remote folder structure to a local directory

```python
from pathlib import Path

# Inspect remote structure

print(vault.list_remote("/"))

# Single file operations

vault.upload_file(Path("report.pdf"), "/reports/annual.pdf")
vault.download_file("/reports/annual.pdf", Path("local_report.pdf"))

# Batch synchronization

vault.sync_local_to_remote(Path("datasets/"), remote_folder="/backup/datasets/")
vault.sync_remote_to_local(remote_folder="/backup/datasets/", local_folder=Path("restored_data/"))

```

## Integrating Cloud Backends with sdata Objects

Higher-level sdata objects accept a `storage_backend` parameter, enabling seamless persistence without protocol-specific code. The `storage_backend` hook is defined in the core classes (see [`sdata/base.py`](https://github.com/lepy/sdata/blob/main/sdata/base.py)), allowing objects like `Workbook` to offload binary asset storage to the configured cloud provider.

```python
from sdata import Workbook
from sdata.iolib.minio import MinioBucket
import pandas as pd

# Create workbook backed by MinIO

wb = Workbook(
    storage_backend=MinioBucket("minio.mycompany.com:9000", "workbook-bucket")
)

# Add data (DataFrame stored locally; large binaries routed to bucket)

df = pd.DataFrame({"x": range(1000), "y": range(1000)})
wb.add_table(df, name="large_dataset")

# Persist—automatically uploads to configured backend

wb.save("analysis.sdata")

```

This pattern allows you to switch between MinIO and OwnCloud (or any custom backend implementing the storage interface) by changing a single line of configuration code.

## Summary

- **`MinioBucket`** ([`sdata/iolib/minio.py`](https://github.com/lepy/sdata/blob/main/sdata/iolib/minio.py)) wraps S3-compatible storage with automatic bucket creation, environment variable credential fallback (`MINIO_ACCESS_KEY_ID` / `MINIO_SECRET_ACCESS_KEY`), and pandas-friendly listing methods.
- **`OwnCloudVault`** ([`sdata/iolib/owncloudfs.py`](https://github.com/lepy/sdata/blob/main/sdata/iolib/owncloudfs.py)) provides WebDAV access to public OwnCloud shares with bidirectional recursive synchronization capabilities.
- Both classes handle translation between Python native types (`Path`, `bytes`, `BinaryIO`) and underlying library APIs, re-raising `S3Error` or WebDAV exceptions as standard Python `Exception` objects.
- Integration requires only passing the configured backend instance to sdata objects via the `storage_backend` parameter, enabling cloud-native data serialization without application changes.

## Frequently Asked Questions

### How do I configure MinIO credentials securely in sdata?

You can pass `access_key` and `secret_key` directly to `MinioBucket()`, or set the `MINIO_ACCESS_KEY_ID` and `MINIO_SECRET_ACCESS_KEY` environment variables. The constructor automatically checks for environment variables when explicit credentials are omitted, allowing secure deployment in containerized environments without hardcoded secrets.

### Does sdata support Amazon S3 directly, or only MinIO?

The `MinioBucket` class uses the standard MinIO client library, which is fully compatible with Amazon S3 and other S3-compatible services. Configure your AWS endpoint (e.g., `"s3.amazonaws.com"`), region, and IAM credentials exactly as you would for MinIO—no code changes are required to switch between providers.

### Can I synchronize entire directories with OwnCloud using sdata?

Yes. The `OwnCloudVault.sync_local_to_remote()` method recursively uploads a local directory tree to the remote share, while `sync_remote_to_local()` performs the reverse operation. These methods handle directory traversal and file comparison automatically, as implemented in [`sdata/iolib/owncloudfs.py`](https://github.com/lepy/sdata/blob/main/sdata/iolib/owncloudfs.py) lines 64-74.

### What error handling does sdata implement for cloud storage operations?

Both wrapper classes catch underlying library exceptions—`S3Error` for MinIO and WebDAV-specific errors for OwnCloud—and re-raise them as generic `Exception` instances with descriptive messages. This allows consistent try/except blocks in your application regardless of which storage backend is configured.