# Authentication and Authorization Methods for Pathway Connectors (S3, Azure): A Complete Guide > Explore Pathway's authentication and authorization methods for S3 and Azure connectors. Learn about credential-based access for S3 and account key authentication for Azure Blob Storage. - Repository: [Pathway/pathway](https://github.com/pathwaycom/pathway) - Tags: how-to-guide - Published: 2026-03-06 --- **Pathway supports credential-based authentication for Amazon S3 via explicit access keys, AWS profiles, session tokens, environment variables, and anonymous public access, while Azure Blob Storage requires explicit account name and key authentication.** Pathway is an open-source stream processing framework that enables real-time data pipelines with persistent state storage. Understanding the authentication and authorization methods for Pathway connectors is essential for securely connecting to cloud storage backends like Amazon S3 and Azure Blob Storage. This guide examines the credential-based authentication system implemented in the `pathwaycom/pathway` repository, detailing how user-provided credentials flow from the Python API to the Rust-based persistence backends. ## Supported Authentication Methods by Connector Pathway’s storage connectors use **credential-based authentication** exposed through the Python API. The framework currently supports two primary storage backends with distinct authentication mechanisms. ### Amazon S3 Authentication Options The S3 connector in Pathway supports multiple credential strategies, defined in `AwsS3Settings` within [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) (lines 4161–4240): - **Explicit credentials**: `access_key` paired with `secret_access_key` - **AWS Profile**: `profile` name referencing the AWS shared config file - **Temporary STS credentials**: `session_token` for short-lived access - **Environment variables**: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` - **Public/anonymous access**: No credentials provided for publicly readable buckets ### Azure Blob Storage Authentication Options The Azure connector uses a simpler authentication model defined in `AzureBlobStorageSettings` ([`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs), lines 4134–4156): - **Account key authentication**: `account` name paired with `password` (storage account key) - **Explicit only**: Unlike S3, Pathway does not support `AZURE_STORAGE_CONNECTION_STRING` environment variables; the client is always built from explicit keys provided in the settings object ## Implementation Details: From Python API to Storage Backend Understanding how authentication and authorization methods for Pathway connectors work requires tracing the credential flow from user code to the underlying Rust SDK clients. ### Python API Layer When you instantiate `AwsS3Settings` or `AzureBlobStorageSettings` in Python, you create a PyO3-wrapped Rust struct. These objects are defined in [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs): - **`AwsS3Settings`** (lines 4161–4185): Validates and stores S3 configuration parameters - **`AzureBlobStorageSettings`** (lines 4134–4156): Holds Azure account credentials and container name These settings objects are stored as `Py` or `AzureBlobStorageSettings` within the table definition and accessed by the engine when opening data sources (lines 4568–5176 for S3, lines 4588–5200 for Azure). ### Persistence Backend Layer The actual authentication occurs when the persistence backend initializes: **S3 Backend ([`src/persistence/backends/s3.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/s3.rs), lines 16–44):** The `S3KVStorage` receives `AwsS3Settings` and calls either `construct_private_bucket` (for authenticated access) or `construct_public_bucket` (for anonymous access). The `construct_private_bucket` method creates an `aws_sdk_s3::Bucket` using `AwsCredentials` built from the provided keys, profile, or environment variables. This `Bucket` object, which embeds the authentication context, is stored inside `S3KVStorage` and handles all I/O operations (`list`, `get`, `put`, `delete`). **Azure Backend ([`src/persistence/backends/azure.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/azure.rs), lines 28–48):** The `AzureKVStorage` constructor receives credentials from `AzureBlobStorageSettings.credentials()`, which returns an `azure_storage::StorageCredentials::access_key`. The backend creates a `BlobClient` for each operation using these credentials. All blob-level operations (`list_blobs`, `get`, `put_block_blob`, `delete`) use this authenticated client. ### Configuration Enumeration The `PersistentStorageConfig` enum in [`src/persistence/config.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/config.rs) (lines 48–58) enumerates the supported storage backends and carries the configuration structs to the runtime, ensuring type-safe credential propagation from Python to the Rust engine. ## Code Examples ### Authenticating with Amazon S3 (Private Bucket) Use explicit credentials or profile-based authentication for private S3 buckets: ```python from pathway.engine import AwsS3Settings, Table # Explicit credentials s3_cfg = AwsS3Settings( bucket_name="my-data-bucket", access_key="AKIA************", secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCY********", region="us-east-1", with_path_style=False, ) tbl = Table( name="my_table", path="s3://my-data-bucket/path/to/data/", aws_s3_settings=s3_cfg, ) ``` The `AwsS3Settings` object triggers `construct_private_bucket`, which creates an `s3::Bucket` with the provided `AwsCredentials` (see lines 4215–4240 in [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs)). The bucket backs the `S3KVStorage` persistence layer (see [`src/persistence/backends/s3.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/s3.rs)). ### Accessing Public S3 Buckets Omit credentials for anonymous access to public buckets: ```python from pathway.engine import AwsS3Settings, Table s3_cfg = AwsS3Settings( bucket_name="public-bucket", region="us-west-2", # No access_key / secret_access_key → public access ) tbl = Table( name="public_tbl", path="s3://public-bucket/dataset/", aws_s3_settings=s3_cfg, ) ``` Because `access_key` and `secret_access_key` are omitted, `AwsS3Settings.construct_bucket` falls back to `construct_public_bucket` (lines 4285–4289). The resulting public `Bucket` needs no authentication. ### Authenticating with Azure Blob Storage Azure requires explicit account name and key: ```python from pathway.engine import AzureBlobStorageSettings, Table azure_cfg = AzureBlobStorageSettings( account="myaccountname", password="myaccountkey==", container="mycontainer", ) tbl = Table( name="azure_tbl", path="azure://mycontainer/prefix/", azure_blob_storage_settings=azure_cfg, ) ``` `AzureBlobStorageSettings.credentials()` (lines 4154–4156) builds an `AzureStorageCredentials::access_key`. The `AzureKVStorage` backend receives these credentials (see [`src/persistence/backends/azure.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/azure.rs) lines 28–48) and creates a `BlobClient` for each blob operation. ## Key Source Files | File | Role | Link | |------|------|------| | [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) – `AwsS3Settings` definition & constructors | Public API for S3 authentication | | | [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) – `AzureBlobStorageSettings` definition | Public API for Azure authentication | | | [`src/persistence/backends/s3.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/s3.rs) – `S3KVStorage` implementation | Persists data to S3 using the authenticated bucket | | | [`src/persistence/backends/azure.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/azure.rs) – `AzureKVStorage` implementation | Persists data to Azure Blob Storage using the supplied credentials | | | [`src/persistence/config.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/config.rs) – `PersistentStorageConfig` enum | Enumerates supported storage back-ends and carries the config structs | | | [`src/connectors/scanner/s3.rs`](https://github.com/pathwaycom/pathway/blob/main/src/connectors/scanner/s3.rs) – `S3Scanner` | Uses the authenticated `S3Bucket` to read objects | | ## Summary - Pathway connectors use **credential-based authentication** exposed through Python API settings objects. - **Amazon S3** supports multiple authentication methods: explicit access keys, AWS profiles, session tokens, environment variables, and anonymous public access. - **Azure Blob Storage** requires explicit account name and key authentication; environment variables like `AZURE_STORAGE_CONNECTION_STRING` are not supported. - Credentials flow from `AwsS3Settings` and `AzureBlobStorageSettings` in [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) to the persistence backends `S3KVStorage` and `AzureKVStorage`, where they initialize SDK-specific clients (`s3::Bucket` and `BlobClient`). - The `PersistentStorageConfig` enum ensures type-safe transport of authentication configuration from Python to the Rust runtime. ## Frequently Asked Questions ### Does Pathway support IAM role-based authentication for S3? Pathway does not directly expose IAM role authentication in the `AwsS3Settings` API. However, you can use the `profile` parameter to reference an AWS profile configured with IAM role assumptions in your shared AWS config file (`~/.aws/config`), or set the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` environment variables populated by your IAM role provider. ### Can I use Azure managed identities with Pathway connectors? Currently, Pathway only supports explicit account key authentication for Azure Blob Storage via the `account` and `password` parameters in `AzureBlobStorageSettings`. The `AZURE_STORAGE_CONNECTION_STRING` environment variable is not utilized by the runtime. Support for Azure managed identities would require extending the `AzureBlobStorageSettings` struct and the `AzureKVStorage` backend implementation. ### How does Pathway handle credential rotation for long-running pipelines? Pathway initializes the storage client (`s3::Bucket` or `BlobClient`) with credentials at the time the table is opened. For long-running pipelines, credential rotation requires restarting the Pathway process with updated credentials, as the current implementation in `S3KVStorage` and `AzureKVStorage` does not support dynamic credential refresh or token rotation during runtime. ### Is it possible to use S3-compatible storage like MinIO with Pathway? Yes, Pathway can connect to S3-compatible storage systems like MinIO by using the `AwsS3Settings` with the `with_path_style=True` parameter (virtual-hosted-style vs path-style buckets) and specifying the custom endpoint in your environment or SDK configuration. The `construct_private_bucket` method in [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) supports these configurations when building the `s3::Bucket` client.