# How to Run genai-sagemaker Examples on Amazon SageMaker

> Adapt generative AI examples for Amazon SageMaker using the sagemaker Python SDK and Hugging Face inference. Learn how to run models effectively on AWS.

- Repository: [Google Cloud Platform/generative-ai](https://github.com/GoogleCloudPlatform/generative-ai)
- Tags: how-to-guide
- Published: 2026-03-09

---

**The `genai-sagemaker` directory does not exist in the GoogleCloudPlatform/generative-ai repository, but you can adapt the existing Vertex AI and open-models examples to run on Amazon SageMaker using the `sagemaker` Python SDK and Hugging Face inference containers.**

The GoogleCloudPlatform/generative-ai repository is the authoritative source for generative AI implementations on Google Cloud, containing end-to-end notebooks for **Vertex AI** and **Gemini**. While developers searching for native `genai-sagemaker` examples will not find a dedicated folder in the current `main` branch, the architectural patterns in the `open-models` and `gemini` directories provide a complete blueprint for deploying these same workloads on Amazon SageMaker with minimal modifications.

## Why the Repository Lacks a genai-sagemaker Directory

The repository's primary focus is Google Cloud's Generative AI services. According to the source code structure, SageMaker-specific notebooks are not part of the public `main` tree, which is why a `genai-sagemaker` directory cannot be found through repository search.

However, the **open-models** notebooks demonstrate framework-agnostic patterns for loading models from the Hugging Face hub and deploying them to managed endpoints. These patterns translate directly to SageMaker's deployment workflow.

## Adapting Vertex AI Patterns for SageMaker Deployment

You can recreate the SageMaker workflow by replacing Google Cloud SDK calls with AWS equivalents while preserving the model loading and inference logic.

### Step 1: Configure Dependencies and Environment

Start by installing the required Python SDKs in your SageMaker notebook instance. While the repository does not ship a SageMaker-specific [`requirements.txt`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/requirements.txt), you can adapt the generic dependencies found in [`search/web-app/requirements.txt`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/search/web-app/requirements.txt) and add AWS-specific packages:

- `sagemaker`
- `boto3`
- `transformers`

### Step 2: Select a Pretrained Model

Choose a model from the Hugging Face hub, following the pattern in `open-models/use-cases/vertex_ai_deepseek_smolagents.ipynb`. This notebook illustrates how to load models like `google/gemma-2b` using the `transformers` library, which works identically on SageMaker.

### Step 3: Create the SageMaker Model Object

Use the `sagemaker` Python SDK to define a `Model` object that points to a Docker container capable of running your chosen framework. This parallels the Vertex AI model creation logic in `open-models/serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb`.

For Hugging Face models, use the official `huggingface-pytorch-inference` container from the AWS ECR registry:

```python
container_image = (
    "763104351884.dkr.ecr.us-west-2.amazonaws.com"
    "/huggingface-pytorch-inference:1.13.1-transformers4.30.2-gpu-py39-cu118-ubuntu20.04"
)

```

### Step 4: Deploy the Endpoint

Call `model.deploy()` with your desired instance type, mirroring the deployment sections in the open-models serving notebooks. This provisions the inference infrastructure and exposes a runtime endpoint.

### Step 5: Invoke the Endpoint

Use the SageMaker runtime client or the `Predictor` object to send JSON payloads containing your prompts. The request-response handling follows the same structure as the Gemini Chat API demonstrated in `gemini/getting-started/intro_gemini_chat.ipynb`, using `inputs` and `parameters` keys.

### Step 6: Optional REST API Wrapper

The repository includes a minimal Flask application in [`search/web-app/main.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/search/web-app/main.py) that wraps Vertex AI predictions. You can reuse this structure to expose your SageMaker endpoint as a REST API by replacing the Vertex AI client initialization with `boto3.client('runtime.sagemaker')` calls.

### Step 7: Clean Up Resources

Delete the endpoint when finished to avoid ongoing charges using `sagemaker.Session().delete_endpoint(endpoint_name)`. This cleanup pattern appears implicitly in every deployment notebook in the repository.

## Complete SageMaker Deployment Script

Below is a minimal, self-contained script that implements the full workflow on SageMaker, adapted from the patterns in `open-models/serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb` and `gemini/getting-started/intro_gemini_chat.ipynb`.

```python

# Install dependencies (run once in your SageMaker notebook)

# !pip install -q sagemaker boto3 transformers

import json
import boto3
import sagemaker
from sagemaker import Model
from sagemaker.session import Session
from sagemaker.predictor import Predictor

# -------------------------------------------------

# 1️⃣  Choose a pre-trained model from Hugging Face

# -------------------------------------------------

model_id = "google/gemma-2b"  # replace with any HF model id

# -------------------------------------------------

# 2️⃣  Define the SageMaker container (Hugging Face)

# -------------------------------------------------

container_image = (
    "763104351884.dkr.ecr.us-west-2.amazonaws.com"
    "/huggingface-pytorch-inference:1.13.1-transformers4.30.2-gpu-py39-cu118-ubuntu20.04"
)

# -------------------------------------------------

# 3️⃣  Create the SageMaker Model object

# -------------------------------------------------

role = sagemaker.get_execution_role()  # SageMaker notebook role

sess = Session()

hf_model = Model(
    image_uri=container_image,
    model_data=None,  # No model tarball – HF container pulls from hub at runtime

    role=role,
    sagemaker_session=sess,
    env={"HF_MODEL_ID": model_id, "HF_TASK": "text-generation"},
)

# -------------------------------------------------

# 4️⃣  Deploy an endpoint

# -------------------------------------------------

predictor: Predictor = hf_model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.large",
    endpoint_name="gemma-2b-endpoint",
)

# -------------------------------------------------

# 5️⃣  Invoke the endpoint

# -------------------------------------------------

payload = {
    "inputs": "Write a short poem about clouds.",
    "parameters": {"max_new_tokens": 64, "temperature": 0.7},
}
response = predictor.predict(json.dumps(payload))
generated = json.loads(response.decode("utf-8"))
print(generated[0]["generated_text"])

# -------------------------------------------------

# 6️⃣  Clean up (delete endpoint when done)

# -------------------------------------------------

# predictor.delete_endpoint()

```

## Key Files to Reference for SageMaker Migration

When building SageMaker examples, consult these specific files from the GoogleCloudPlatform/generative-ai repository:

- [`search/web-app/requirements.txt`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/search/web-app/requirements.txt) – Lists core Python packages; add `sagemaker` and `boto3` to adapt for AWS.

- `open-models/use-cases/vertex_ai_deepseek_smolagents.ipynb` – Demonstrates loading models from the Hugging Face hub and preparing them for managed endpoints.

- `open-models/serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb` – Contains the detailed deployment workflow (model object creation, endpoint configuration, and invocation) that maps to SageMaker patterns.

- `gemini/getting-started/intro_gemini_chat.ipynb` – Shows the JSON payload structure for LLM chat requests, compatible with Hugging Face inference containers.

- [`search/web-app/main.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/search/web-app/main.py) – Provides a Flask wrapper structure useful for exposing SageMaker endpoints via REST API.

## Summary

- The `genai-sagemaker` directory is not present in the repository's `main` branch.
- Adaptation requires replacing **Vertex AI SDK** calls with **SageMaker SDK** (`sagemaker` and `boto3`).
- Use **Hugging Face inference containers** for deploying open models from the Hub.
- Request/response JSON structures from the Gemini examples transfer directly to SageMaker endpoints.
- Always delete endpoints after use to avoid AWS infrastructure charges.

## Frequently Asked Questions

### Does the GoogleCloudPlatform/generative-ai repository contain a genai-sagemaker directory?

No. The repository focuses exclusively on Google Cloud services, and the `genai-sagemaker` folder does not exist in the public `main` branch. You must manually adapt the Vertex AI and open-models examples for Amazon SageMaker.

### Which repository file shows the best pattern for SageMaker endpoint deployment?

The `open-models/serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb` notebook provides the most relevant blueprint. It demonstrates model object creation, endpoint deployment, and invocation patterns that translate directly to SageMaker's `Model.deploy()` workflow.

### Can I use the Gemini chat examples on Amazon SageMaker?

Yes. While you cannot use the Gemini API itself on SageMaker, the request formatting logic in `gemini/getting-started/intro_gemini_chat.ipynb` (using structured JSON with `inputs` and `parameters` keys) is compatible with Hugging Face inference containers deployed on SageMaker, requiring minimal modification to your application code.

### How do I handle authentication when adapting these examples for AWS?

Replace the Google Cloud authentication (`google.auth`) with standard AWS credential chains. In a SageMaker notebook environment, use `sagemaker.get_execution_role()` to obtain the IAM role, as shown in the deployment script above. For local development, configure the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables or use the `boto3` credential provider chain.