Authentication and Authorization Methods for Pathway Connectors (S3, Azure): A Complete Guide

Pathway supports credential-based authentication for Amazon S3 via explicit access keys, AWS profiles, session tokens, environment variables, and anonymous public access, while Azure Blob Storage requires explicit account name and key authentication.

Pathway is an open-source stream processing framework that enables real-time data pipelines with persistent state storage. Understanding the authentication and authorization methods for Pathway connectors is essential for securely connecting to cloud storage backends like Amazon S3 and Azure Blob Storage. This guide examines the credential-based authentication system implemented in the pathwaycom/pathway repository, detailing how user-provided credentials flow from the Python API to the Rust-based persistence backends.

Supported Authentication Methods by Connector

Pathway’s storage connectors use credential-based authentication exposed through the Python API. The framework currently supports two primary storage backends with distinct authentication mechanisms.

Amazon S3 Authentication Options

The S3 connector in Pathway supports multiple credential strategies, defined in AwsS3Settings within src/python_api.rs (lines 4161–4240):

• Explicit credentials: access_key paired with secret_access_key
• AWS Profile: profile name referencing the AWS shared config file
• Temporary STS credentials: session_token for short-lived access
• Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN
• Public/anonymous access: No credentials provided for publicly readable buckets

Azure Blob Storage Authentication Options

The Azure connector uses a simpler authentication model defined in AzureBlobStorageSettings (src/python_api.rs, lines 4134–4156):

• Account key authentication: account name paired with password (storage account key)
• Explicit only: Unlike S3, Pathway does not support AZURE_STORAGE_CONNECTION_STRING environment variables; the client is always built from explicit keys provided in the settings object

Implementation Details: From Python API to Storage Backend

Understanding how authentication and authorization methods for Pathway connectors work requires tracing the credential flow from user code to the underlying Rust SDK clients.

Python API Layer

When you instantiate AwsS3Settings or AzureBlobStorageSettings in Python, you create a PyO3-wrapped Rust struct. These objects are defined in src/python_api.rs:

• AwsS3Settings (lines 4161–4185): Validates and stores S3 configuration parameters
• AzureBlobStorageSettings (lines 4134–4156): Holds Azure account credentials and container name

These settings objects are stored as Py<AwsS3Settings> or AzureBlobStorageSettings within the table definition and accessed by the engine when opening data sources (lines 4568–5176 for S3, lines 4588–5200 for Azure).

Persistence Backend Layer

The actual authentication occurs when the persistence backend initializes:

S3 Backend (src/persistence/backends/s3.rs, lines 16–44): The S3KVStorage receives AwsS3Settings and calls either construct_private_bucket (for authenticated access) or construct_public_bucket (for anonymous access). The construct_private_bucket method creates an aws_sdk_s3::Bucket using AwsCredentials built from the provided keys, profile, or environment variables. This Bucket object, which embeds the authentication context, is stored inside S3KVStorage and handles all I/O operations (list, get, put, delete).

Azure Backend (src/persistence/backends/azure.rs, lines 28–48): The AzureKVStorage constructor receives credentials from AzureBlobStorageSettings.credentials(), which returns an azure_storage::StorageCredentials::access_key. The backend creates a BlobClient for each operation using these credentials. All blob-level operations (list_blobs, get, put_block_blob, delete) use this authenticated client.

Configuration Enumeration

The PersistentStorageConfig enum in src/persistence/config.rs (lines 48–58) enumerates the supported storage backends and carries the configuration structs to the runtime, ensuring type-safe credential propagation from Python to the Rust engine.

Code Examples

Authenticating with Amazon S3 (Private Bucket)

Use explicit credentials or profile-based authentication for private S3 buckets:

from pathway.engine import AwsS3Settings, Table

# Explicit credentials

s3_cfg = AwsS3Settings(
    bucket_name="my-data-bucket",
    access_key="AKIA************",
    secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCY********",
    region="us-east-1",
    with_path_style=False,
)

tbl = Table(
    name="my_table",
    path="s3://my-data-bucket/path/to/data/",
    aws_s3_settings=s3_cfg,
)

The AwsS3Settings object triggers construct_private_bucket, which creates an s3::Bucket with the provided AwsCredentials (see lines 4215–4240 in src/python_api.rs). The bucket backs the S3KVStorage persistence layer (see src/persistence/backends/s3.rs).

Accessing Public S3 Buckets

Omit credentials for anonymous access to public buckets:

from pathway.engine import AwsS3Settings, Table

s3_cfg = AwsS3Settings(
    bucket_name="public-bucket",
    region="us-west-2",
    # No access_key / secret_access_key → public access

)

tbl = Table(
    name="public_tbl",
    path="s3://public-bucket/dataset/",
    aws_s3_settings=s3_cfg,
)

Because access_key and secret_access_key are omitted, AwsS3Settings.construct_bucket falls back to construct_public_bucket (lines 4285–4289). The resulting public Bucket needs no authentication.

Authenticating with Azure Blob Storage

Azure requires explicit account name and key:

from pathway.engine import AzureBlobStorageSettings, Table

azure_cfg = AzureBlobStorageSettings(
    account="myaccountname",
    password="myaccountkey==",
    container="mycontainer",
)

tbl = Table(
    name="azure_tbl",
    path="azure://mycontainer/prefix/",
    azure_blob_storage_settings=azure_cfg,
)

AzureBlobStorageSettings.credentials() (lines 4154–4156) builds an AzureStorageCredentials::access_key. The AzureKVStorage backend receives these credentials (see src/persistence/backends/azure.rs lines 28–48) and creates a BlobClient for each blob operation.

Key Source Files

File	Role	Link
src/python_api.rs – AwsS3Settings definition & constructors	Public API for S3 authentication	https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs#L4161-L4240
src/python_api.rs – AzureBlobStorageSettings definition	Public API for Azure authentication	https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs#L4134-L4156
src/persistence/backends/s3.rs – S3KVStorage implementation	Persists data to S3 using the authenticated bucket	https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/s3.rs#L16-L44
src/persistence/backends/azure.rs – AzureKVStorage implementation	Persists data to Azure Blob Storage using the supplied credentials	https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/azure.rs#L28-L48
src/persistence/config.rs – PersistentStorageConfig enum	Enumerates supported storage back-ends and carries the config structs	https://github.com/pathwaycom/pathway/blob/main/src/persistence/config.rs#L48-L58
src/connectors/scanner/s3.rs – S3Scanner	Uses the authenticated S3Bucket to read objects	https://github.com/pathwaycom/pathway/blob/main/src/connectors/scanner/s3.rs#L70-L84

Summary

• Pathway connectors use credential-based authentication exposed through Python API settings objects.
• Amazon S3 supports multiple authentication methods: explicit access keys, AWS profiles, session tokens, environment variables, and anonymous public access.
• Azure Blob Storage requires explicit account name and key authentication; environment variables like AZURE_STORAGE_CONNECTION_STRING are not supported.
• Credentials flow from AwsS3Settings and AzureBlobStorageSettings in src/python_api.rs to the persistence backends S3KVStorage and AzureKVStorage, where they initialize SDK-specific clients (s3::Bucket and BlobClient).
• The PersistentStorageConfig enum ensures type-safe transport of authentication configuration from Python to the Rust runtime.

Frequently Asked Questions

Does Pathway support IAM role-based authentication for S3?

Pathway does not directly expose IAM role authentication in the AwsS3Settings API. However, you can use the profile parameter to reference an AWS profile configured with IAM role assumptions in your shared AWS config file (~/.aws/config), or set the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables populated by your IAM role provider.

Can I use Azure managed identities with Pathway connectors?

Currently, Pathway only supports explicit account key authentication for Azure Blob Storage via the account and password parameters in AzureBlobStorageSettings. The AZURE_STORAGE_CONNECTION_STRING environment variable is not utilized by the runtime. Support for Azure managed identities would require extending the AzureBlobStorageSettings struct and the AzureKVStorage backend implementation.

How does Pathway handle credential rotation for long-running pipelines?

Pathway initializes the storage client (s3::Bucket or BlobClient) with credentials at the time the table is opened. For long-running pipelines, credential rotation requires restarting the Pathway process with updated credentials, as the current implementation in S3KVStorage and AzureKVStorage does not support dynamic credential refresh or token rotation during runtime.

Is it possible to use S3-compatible storage like MinIO with Pathway?

Yes, Pathway can connect to S3-compatible storage systems like MinIO by using the AwsS3Settings with the with_path_style=True parameter (virtual-hosted-style vs path-style buckets) and specifying the custom endpoint in your environment or SDK configuration. The construct_private_bucket method in src/python_api.rs supports these configurations when building the s3::Bucket client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}