How to Deploy a Generative AI Model on SageMaker Using the GoogleCloudPlatform/generative-ai Repository

You can deploy a generative AI model on SageMaker by extracting the Docker containerization logic from the repository's Vertex AI notebooks, publishing the image to Amazon ECR, and deploying via the SageMaker Python SDK.

The GoogleCloudPlatform/generative-ai repository is a comprehensive collection of notebooks and deployment recipes optimized for Vertex AI. While it does not ship native SageMaker manifests, the container packaging patterns are cloud-agnostic. By adapting the custom handler logic found in the repository's serving notebooks, you can deploy a generative AI model on SageMaker with minimal modifications to the underlying Docker configuration.

Extract Container Build Logic from Vertex AI Notebooks

The repository's serving notebooks demonstrate how to package Hugging Face models into production-ready containers. The file open-models/serving/vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb contains the critical Docker build instructions that you can repurpose. This notebook constructs a container using the base image us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-pytorch-inference-cu121.2-3.transformers.4-48.ubuntu2204.py311 and installs additional dependencies like FastAPI for custom inference handlers.

To reuse this for SageMaker, extract the Dockerfile definition from the notebook's %%bash cells. The container structure remains identical regardless of the target cloud platform—you only need to change the destination registry from Google Container Registry to Amazon ECR.

Sample Dockerfile Configuration

FROM us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-pytorch-inference-cu121.2-3.transformers.4-48.ubuntu2204.py311

RUN pip install --no-cache-dir fastapi uvicorn
COPY inference.py /opt/program/inference.py

ENTRYPOINT ["uvicorn", "inference:app", "--host", "0.0.0.0", "--port", "8080"]

Push the Image to Amazon ECR

Before deploying, you must publish your container to Amazon Elastic Container Registry (ECR). Replace the repository's default Google Artifact Registry workflow with the following AWS CLI commands.

aws ecr get-login-password --region <region> | \
  docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com

aws ecr create-repository --repository-name my-genai-model

docker build -t my-genai-model:latest .
docker tag my-genai-model:latest <account-id>.dkr.ecr.<region>.amazonaws.com/my-genai-model:latest
docker push <account-id>.dkr.ecr.<region>.amazonaws.com/my-genai-model:latest

Deploy to SageMaker Using the Python SDK

The SageMaker SDK provides direct equivalents to the Vertex AI deployment workflow demonstrated in open-models/serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb. The logical flow—registering the model, creating an endpoint configuration, and deploying—remains identical, though the class names differ.

import sagemaker
from sagemaker.model import Model
from sagemaker import Session

session = Session()
role = "<your-execution-role-arn>"

image_uri = "<account-id>.dkr.ecr.<region>.amazonaws.com/my-genai-model:latest"
model = Model(
    image_uri=image_uri,
    role=role,
    sagemaker_session=session,
    env={"SAGEMAKER_PROGRAM": "inference.py"}
)

predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.xlarge",
    serializer=sagemaker.serializers.JSONSerializer(),
    deserializer=sagemaker.deserializers.JSONDeserializer()
)

Architecture Comparison

The deployment pipeline maps directly between platforms:

• Vertex AI (Repository): Uses aiplatform.Model.upload() followed by model.deploy() to create an endpoint.
• SageMaker: Uses sagemaker.model.Model() constructor followed by model.deploy(), which automatically creates both the endpoint configuration and the endpoint.

Both platforms expect the container to expose an HTTP server on port 8080 and accept POST requests to the /invocations path.

Implement the Inference Handler

Your container must implement the SageMaker inference contract. The following inference.py adapts the custom handler pattern from the repository's PLLuM notebook for FastAPI:

from fastapi import FastAPI, Request
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

app = FastAPI()

model_name = "togethercomputer/RedPajama-INCITE-Chat-3B-v1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

@app.post("/invocations")
async def invoke(request: Request):
    payload = await request.json()
    inputs = payload.get("inputs", "")
    inputs_encoded = tokenizer.encode(inputs, return_tensors="pt")
    outputs = model.generate(inputs_encoded, max_new_tokens=64)
    text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return {"generated_text": text}

Test and Clean Up

After deployment, invoke the endpoint using the predictor object:

request = {"inputs": "Explain the theory of relativity in 2 sentences."}
response = predictor.predict(request)
print(response)

When testing is complete, delete the endpoint to avoid ongoing charges:

predictor.delete_endpoint()

Summary

• The GoogleCloudPlatform/generative-ai repository contains reusable Docker packaging logic in notebooks like open-models/serving/vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb.
• To deploy a generative AI model on SageMaker, extract the Dockerfile, push to Amazon ECR, and deploy using the SageMaker Python SDK.
• The container must expose port 8080 and handle POST requests to /invocations.
• GPU instances like ml.g5.xlarge provide the necessary compute for large language model inference.

Frequently Asked Questions

Does the GoogleCloudPlatform/generative-ai repository support SageMaker natively?

No. The repository is optimized for Vertex AI and Google Cloud services. However, the Docker containerization patterns and model loading code are platform-agnostic and can be adapted for SageMaker by changing the target container registry and deployment SDK.

Which instance type should I use for generative AI models on SageMaker?

For models based on the Hugging Face Transformers library, use GPU-enabled instances such as ml.g5.xlarge or larger. These instances provide the CUDA support required by the PyTorch inference containers referenced in the repository's notebooks.

Can I use the repository's notebooks directly on SageMaker Studio?

While you cannot run the Vertex AI-specific cells directly, you can adapt the Docker build and model packaging logic for SageMaker Studio. Extract the shell commands and Python inference handlers from notebooks like vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb and execute them in a SageMaker Studio environment with AWS credentials configured.

What is the difference between Vertex AI and SageMaker deployment in this context?

The primary difference lies in the orchestration layer. Vertex AI uses aiplatform.Model.upload() and model.deploy(), while SageMaker uses sagemaker.model.Model() and predictor.deploy(). Both platforms ultimately run the same Docker container, but SageMaker requires the image to be stored in Amazon ECR rather than Google Artifact Registry.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}