How Pathway Handles Authentication with Google Drive, SharePoint, and S3 Data Sources

Pathway authenticates with Google Drive via service-account JSON keys, with SharePoint via Azure AD client certificates, and with S3 using AWS credentials, with each connector automatically handling token refresh through the underlying libraries.

The pathwaycom/llm-app repository demonstrates how Pathway's unified I/O API abstracts authentication complexities for external storage systems. Each connector accepts explicit parameters or environment variables—never requiring hard-coded secrets—while managing OAuth and signature workflows internally.

Google Drive Authentication

Pathway connects to Google Drive through the pw.io.gdrive.read connector, which implements server-to-server authentication using Google Cloud service accounts.

Service Account JSON Credentials

Authentication requires a service-account JSON key file containing the private key and client email. You provide the filesystem path via the service_user_credentials_file parameter, typically sourced from the GOOGLE_CREDS environment variable as shown in templates/drive_alert/app.py:

files = pw.io.gdrive.read(
    object_id=object_id,
    service_user_credentials_file=service_user_credentials_file,
    refresh_interval=30,
)

Pathway reads the JSON file and uses the Google client library to obtain OAuth access tokens, refreshing them automatically whenever they expire. The service account must have appropriate Drive API permissions and shared folder access configured in the Google Cloud Console.

SharePoint Authentication

For SharePoint integration, Pathway utilizes the !pw.xpacks.connectors.sharepoint.read connector, configured declaratively through YAML rather than Python code.

Azure AD Client Certificate Flow

SharePoint authentication requires Azure AD application credentials using the OAuth2 client-certificate flow. According to templates/question_answering_rag/README.md, you must supply the SharePoint site URL, Azure AD tenant ID, application client ID, certificate path, and certificate thumbprint in a sources_configuration.yaml file:

sources:
  - name: sharepoint_source
    connector: !pw.xpacks.connectors.sharepoint.read
    url: "https://mycompany.sharepoint.com/sites/docs"
    tenant: "12345678-90ab-cdef-1234-567890abcdef"
    client_id: "abcdef12-3456-7890-abcd-ef1234567890"
    cert_path: "/secrets/sharepoint_cert.pem"
    thumbprint: "A1B2C3D4E5F60789ABCD1234567890EF"
    root_path: "/Shared Documents/ProjectX"

The connector uses these parameters to authenticate with Azure AD, obtains an access token for SharePoint Online, and automatically refreshes the token before expiration. The PEM-encoded certificate must be registered in your Azure AD application registration.

Amazon S3 Authentication

Pathway's S3 connector (pw.io.s3.read) supports standard AWS authentication mechanisms through the boto3 library.

AWS Credentials via Environment Variables or Explicit Parameters

You can supply credentials through standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) or pass them explicitly via the aws_credentials dictionary. As indicated by the placeholder comment at line 162 of templates/drive_alert/app.py, explicit configuration follows this pattern:

files = pw.io.s3.read(
    bucket="my-bucket",
    prefix="data/",
    aws_credentials={
        "access_key_id": os.getenv("AWS_ACCESS_KEY_ID"),
        "secret_access_key": os.getenv("AWS_SECRET_ACCESS_KEY"),
        "session_token": os.getenv("AWS_SESSION_TOKEN"),
    },
)

Pathway forwards these values to boto3, which handles request signing and automatic refresh for temporary session tokens. When running on AWS infrastructure, omitting the aws_credentials parameter allows the connector to inherit IAM role permissions through the default credential chain.

Summary

• Google Drive: Requires a service-account JSON file via the service_user_credentials_file parameter in pw.io.gdrive.read; Pathway manages OAuth token refresh via the Google client library.
• SharePoint: Uses Azure AD client-certificate authentication configured in YAML with tenant, client_id, cert_path, and thumbprint parameters for the !pw.xpacks.connectors.sharepoint.read connector.
• S3: Accepts standard AWS credentials via environment variables or the aws_credentials dictionary in pw.io.s3.read, delegating signature and refresh logic to boto3.
• All connectors integrate with Pathway's reactive streaming engine and automatically handle credential expiration without pipeline interruption.

Frequently Asked Questions

How does Pathway store Google Drive service account credentials securely?

Pathway does not store credentials internally; it reads the service-account JSON file path from the service_user_credentials_file parameter, which production deployments typically populate from the GOOGLE_CREDS environment variable defined in files like templates/drive_alert/.env.example. This approach keeps private keys out of version control and allows integration with secrets managers.

What Azure AD permissions are required for the SharePoint connector?

The Azure AD application requires Microsoft Graph or SharePoint API permissions such as Sites.Read.All to access document libraries. Additionally, the client certificate specified in cert_path must be uploaded to the Azure AD application registration, and the application must be granted consent by a SharePoint administrator to access the target site collections.

Can Pathway use IAM roles instead of access keys for S3 authentication?

Yes. When executing on AWS infrastructure such as EC2, ECS, or Lambda, Pathway's S3 connector automatically retrieves IAM role credentials through boto3's default credential provider chain. You can omit the aws_credentials parameter entirely in these environments, and the connector will inherit the instance or task role permissions without explicit key management.

Does Pathway support OAuth 2.0 user authentication for Google Drive?

The current implementation in pathwaycom/llm-app exclusively supports service-account authentication via pw.io.gdrive.read. Service accounts are designed for automated, server-to-server access without interactive user consent, which aligns with Pathway's architecture for continuous data streaming. User-based OAuth 2.0 flows requiring browser-based consent are not supported by this connector.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}