# How to Deploy a Generative AI Model on SageMaker Using the GoogleCloudPlatform/generative-ai Repository

> Deploy a generative AI model on SageMaker by leveraging the GoogleCloudPlatform/generative-ai repository. Learn to containerize, publish to ECR, and deploy with SageMaker SDK.

- Repository: [Google Cloud Platform/generative-ai](https://github.com/GoogleCloudPlatform/generative-ai)
- Tags: how-to-guide
- Published: 2026-03-09

---

**You can deploy a generative AI model on SageMaker by extracting the Docker containerization logic from the repository's Vertex AI notebooks, publishing the image to Amazon ECR, and deploying via the SageMaker Python SDK.**

The **GoogleCloudPlatform/generative-ai** repository is a comprehensive collection of notebooks and deployment recipes optimized for **Vertex AI**. While it does not ship native SageMaker manifests, the container packaging patterns are cloud-agnostic. By adapting the custom handler logic found in the repository's serving notebooks, you can deploy a generative AI model on SageMaker with minimal modifications to the underlying Docker configuration.

## Extract Container Build Logic from Vertex AI Notebooks

The repository's serving notebooks demonstrate how to package Hugging Face models into production-ready containers. The file `open-models/serving/vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb` contains the critical Docker build instructions that you can repurpose. This notebook constructs a container using the base image `us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-pytorch-inference-cu121.2-3.transformers.4-48.ubuntu2204.py311` and installs additional dependencies like FastAPI for custom inference handlers.

To reuse this for SageMaker, extract the Dockerfile definition from the notebook's `%%bash` cells. The container structure remains identical regardless of the target cloud platform—you only need to change the destination registry from Google Container Registry to Amazon ECR.

### Sample Dockerfile Configuration

```dockerfile
FROM us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-pytorch-inference-cu121.2-3.transformers.4-48.ubuntu2204.py311

RUN pip install --no-cache-dir fastapi uvicorn
COPY inference.py /opt/program/inference.py

ENTRYPOINT ["uvicorn", "inference:app", "--host", "0.0.0.0", "--port", "8080"]

```

## Push the Image to Amazon ECR

Before deploying, you must publish your container to **Amazon Elastic Container Registry (ECR)**. Replace the repository's default Google Artifact Registry workflow with the following AWS CLI commands.

```bash
aws ecr get-login-password --region <region> | \
  docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com

aws ecr create-repository --repository-name my-genai-model

docker build -t my-genai-model:latest .
docker tag my-genai-model:latest <account-id>.dkr.ecr.<region>.amazonaws.com/my-genai-model:latest
docker push <account-id>.dkr.ecr.<region>.amazonaws.com/my-genai-model:latest

```

## Deploy to SageMaker Using the Python SDK

The **SageMaker SDK** provides direct equivalents to the Vertex AI deployment workflow demonstrated in `open-models/serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb`. The logical flow—registering the model, creating an endpoint configuration, and deploying—remains identical, though the class names differ.

```python
import sagemaker
from sagemaker.model import Model
from sagemaker import Session

session = Session()
role = "<your-execution-role-arn>"

image_uri = "<account-id>.dkr.ecr.<region>.amazonaws.com/my-genai-model:latest"
model = Model(
    image_uri=image_uri,
    role=role,
    sagemaker_session=session,
    env={"SAGEMAKER_PROGRAM": "inference.py"}
)

predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.xlarge",
    serializer=sagemaker.serializers.JSONSerializer(),
    deserializer=sagemaker.deserializers.JSONDeserializer()
)

```

### Architecture Comparison

The deployment pipeline maps directly between platforms:

- **Vertex AI (Repository)**: Uses `aiplatform.Model.upload()` followed by `model.deploy()` to create an endpoint.
- **SageMaker**: Uses `sagemaker.model.Model()` constructor followed by `model.deploy()`, which automatically creates both the endpoint configuration and the endpoint.

Both platforms expect the container to expose an HTTP server on port **8080** and accept POST requests to the `/invocations` path.

## Implement the Inference Handler

Your container must implement the **SageMaker inference contract**. The following [`inference.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/inference.py) adapts the custom handler pattern from the repository's PLLuM notebook for FastAPI:

```python
from fastapi import FastAPI, Request
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

app = FastAPI()

model_name = "togethercomputer/RedPajama-INCITE-Chat-3B-v1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

@app.post("/invocations")
async def invoke(request: Request):
    payload = await request.json()
    inputs = payload.get("inputs", "")
    inputs_encoded = tokenizer.encode(inputs, return_tensors="pt")
    outputs = model.generate(inputs_encoded, max_new_tokens=64)
    text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return {"generated_text": text}

```

## Test and Clean Up

After deployment, invoke the endpoint using the predictor object:

```python
request = {"inputs": "Explain the theory of relativity in 2 sentences."}
response = predictor.predict(request)
print(response)

```

When testing is complete, delete the endpoint to avoid ongoing charges:

```python
predictor.delete_endpoint()

```

## Summary

- The **GoogleCloudPlatform/generative-ai** repository contains reusable Docker packaging logic in notebooks like `open-models/serving/vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb`.
- To deploy a generative AI model on SageMaker, extract the Dockerfile, push to **Amazon ECR**, and deploy using the **SageMaker Python SDK**.
- The container must expose port 8080 and handle POST requests to `/invocations`.
- GPU instances like `ml.g5.xlarge` provide the necessary compute for large language model inference.

## Frequently Asked Questions

### Does the GoogleCloudPlatform/generative-ai repository support SageMaker natively?

No. The repository is optimized for Vertex AI and Google Cloud services. However, the Docker containerization patterns and model loading code are platform-agnostic and can be adapted for SageMaker by changing the target container registry and deployment SDK.

### Which instance type should I use for generative AI models on SageMaker?

For models based on the Hugging Face Transformers library, use GPU-enabled instances such as `ml.g5.xlarge` or larger. These instances provide the CUDA support required by the PyTorch inference containers referenced in the repository's notebooks.

### Can I use the repository's notebooks directly on SageMaker Studio?

While you cannot run the Vertex AI-specific cells directly, you can adapt the Docker build and model packaging logic for SageMaker Studio. Extract the shell commands and Python inference handlers from notebooks like `vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb` and execute them in a SageMaker Studio environment with AWS credentials configured.

### What is the difference between Vertex AI and SageMaker deployment in this context?

The primary difference lies in the orchestration layer. Vertex AI uses `aiplatform.Model.upload()` and `model.deploy()`, while SageMaker uses `sagemaker.model.Model()` and `predictor.deploy()`. Both platforms ultimately run the same Docker container, but SageMaker requires the image to be stored in Amazon ECR rather than Google Artifact Registry.