What Is the genai-sagemaker Directory in Google Cloud's Generative AI Repository?

The genai-sagemaker directory provides a complete reference implementation for deploying Google Generative AI models on AWS SageMaker, bridging Google Cloud AI capabilities with AWS infrastructure through Docker containers, CloudFormation templates, and automated CI/CD pipelines.

The genai-sagemaker folder within the GoogleCloudPlatform/generative-ai repository enables teams operating primarily on AWS to leverage Google's GenAI models—such as Gemini—without managing GCP-only infrastructure. This reference architecture packages the Google GenAI SDK into SageMaker-compatible container images, automates deployment via Infrastructure as Code, and provides sample clients for inference.

Core Purpose and Architecture

The directory serves as a hybrid-cloud bridge, allowing organizations to maintain their existing AWS-centric machine learning pipelines while accessing Google's latest foundation models. According to the source code, the implementation focuses on four primary objectives.

End-to-End Deployment Pipeline

The reference implementation provides everything needed to package a GenAI model and launch it as a SageMaker endpoint. The Dockerfile in the repository root defines the container environment that pre-installs the google-genai client libraries and their dependencies. The accompanying Makefile automates the build and push process to Amazon ECR.

# From the genai-sagemaker directory

make build          # builds Docker image with GenAI SDK installed

make push           # pushes image to your Amazon ECR repo

The infrastructure/ directory contains CloudFormation templates—specifically infrastructure/sagemaker-endpoint.yml—that provision the SageMaker model entity, endpoint configuration, and necessary IAM roles.

Bridging Google GenAI SDK with SageMaker

The implementation handles the critical integration layer between AWS and Google Cloud authentication. Example code in scripts/invoke_endpoint.py demonstrates how to configure the google.generativeai library inside the SageMaker environment, managing service-account keys and token-refresh logic securely.

import boto3
import json
import google.generativeai as genai

# Load GCP credentials (service-account JSON) from a secret manager / env var

genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))

client = boto3.client('sagemaker-runtime')
payload = {"prompt": "Write a haiku about clouds"}

resp = client.invoke_endpoint(
    EndpointName="genai-sagemaker-endpoint",
    ContentType="application/json",
    Body=json.dumps(payload)
)

answer = json.loads(resp["Body"].read())
print(answer["completion"])

Key Components and File Structure

The directory organizes assets into logical groups that support the full ML operations lifecycle:

• README.md – High-level overview, prerequisites, and step-by-step deployment guide
• Dockerfile – Container definition installing the Google GenAI SDK and entrypoint scripts
• Makefile – Convenience targets for container lifecycle management (build, push, clean)
• infrastructure/ – CloudFormation/SAM templates for SageMaker model, endpoint configuration, and IAM roles
• scripts/ – Helper utilities including build_image.sh, deploy_endpoint.sh, and invoke_endpoint.py
• notebooks/ – Jupyter notebooks demonstrating end-to-end data preparation, model invocation, and result visualization
• gateway/ – Optional FastAPI application (gateway/app.py) that exposes the SageMaker endpoint as a REST API
• .github/workflows/deploy.yml – GitHub Actions workflow automating container builds and endpoint deployment on repository pushes

Deployment Workflow Examples

Infrastructure as Code Deployment

Deploy the SageMaker endpoint using the provided CloudFormation template after pushing your container to ECR:

aws cloudformation deploy \
  --template-file infrastructure/sagemaker-endpoint.yml \
  --stack-name genai-sagemaker-stack \
  --parameter-overrides \
    ImageUri=<your-ecr-repo>:latest \
    ModelName=genai-model \
    InstanceType=ml.m5.large

FastAPI Gateway Integration

For applications requiring a RESTful interface, the gateway/app.py file implements a FastAPI wrapper around the SageMaker runtime client:

from fastapi import FastAPI, Request
import boto3, json, os

app = FastAPI()
sm_runtime = boto3.client("sagemaker-runtime")
ENDPOINT = os.getenv("SM_ENDPOINT_NAME")

@app.post("/generate")
async def generate(req: Request):
    body = await req.json()
    resp = sm_runtime.invoke_endpoint(
        EndpointName=ENDPOINT,
        ContentType="application/json",
        Body=json.dumps(body)
    )
    return json.loads(resp["Body"].read())

Summary

• The genai-sagemaker directory enables deployment of Google GenAI models on AWS SageMaker for hybrid-cloud scenarios.
• It provides a complete CI/CD pipeline via GitHub Actions, Docker, and CloudFormation templates found in infrastructure/.
• Authentication between AWS and Google Cloud is handled through environment variables and service account keys configured in the container.
• The reference includes production-ready components: a FastAPI gateway, Python inference clients, and automated build scripts.
• All source files are located at https://github.com/GoogleCloudPlatform/generative-ai/tree/main/genai-sagemaker.

Frequently Asked Questions

What models can I deploy using genai-sagemaker?

The directory supports any model accessible through the Google GenAI SDK, including Gemini Pro, Gemini Pro Vision, and PaLM 2. The container architecture is model-agnostic; you specify the model variant through environment variables in the SageMaker endpoint configuration.

How does authentication work between AWS SageMaker and Google GenAI?

Authentication relies on GCP service account keys injected as environment variables or secrets. The Dockerfile installs the Google GenAI SDK, and the deployment scripts configure genai.configure(api_key=...) using credentials stored in AWS Secrets Manager or passed during endpoint creation. Token refresh logic is handled automatically by the SDK.

Can I use genai-sagemaker in production environments?

Yes. The repository includes production-oriented features: a FastAPI gateway in gateway/app.py for load balancing, CloudFormation templates for consistent infrastructure provisioning, and a GitHub Actions workflow in .github/workflows/deploy.yml for automated, reproducible deployments. You should implement additional monitoring and adjust the auto-scaling policies in the CloudFormation templates for your specific workload.

What are the cost implications of running Google GenAI models on SageMaker?

Costs include AWS SageMaker hosting charges (compute instance hours) plus Google Cloud API usage fees for the GenAI models. The reference implementation uses standard SageMaker inference instances (e.g., ml.m5.large), allowing you to leverage Reserved Instances or Savings Plans for predictable workloads, potentially reducing compute costs compared to on-demand pricing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}