How to Fine-Tune Foundation Models Using the GenAI-SageMaker Directory

The genai-sagemaker directory provides a complete toolchain for supervised fine-tuning of foundation models on Amazon SageMaker using Google's Gen AI SDK, including containerized training environments, YAML-based job configurations, and automated deployment utilities.

The GoogleCloudPlatform/genai-sagemaker repository delivers production-ready infrastructure for fine-tuning large language models on AWS infrastructure. This tooling bridges Google’s Gen AI SDK with Amazon SageMaker's managed training capabilities, enabling you to customize models like Gemma or Llama using proprietary datasets while maintaining full control over hyperparameters and compute resources.

Prerequisites and Installation

Before submitting training jobs, install the helper package and configure your AWS environment.

Install the package from the repository root:

pip install -e .

Ensure your AWS credentials are configured and that you have an IAM Role with AmazonSageMakerFullAccess permissions. The roleArn specified in your training configuration must have permissions to read from S3 and write training artifacts.

Preparing Training Data for Fine-Tuning

The GenAI-SageMaker workflow requires JSON Lines (.jsonl) files where each line contains a prompt and the expected completion. This schema aligns with the Gen AI Supervised-Tuning documentation.

Reference the example structure in data/sample_dataset.jsonl:

{"prompt": "Translate the following English text to French: 'Hello world'", "completion": "Bonjour le monde"}
{"prompt": "Summarize: The quick brown fox...", "completion": "A fox jumps over a lazy dog."}

Upload your dataset to Amazon S3 using the AWS CLI or the provided S3 helper utilities:

aws s3 cp my_training_data.jsonl s3://my-bucket/genai-finetune/input/

The src/utils/s3_helper.py module contains additional utility functions for programmatically uploading and downloading data to S3 buckets.

Configuring the SageMaker Training Job

Define your training configuration in config/training_job.yaml. This file specifies the Docker image, compute resources, hyperparameters, and data locations required by SageMaker's CreateTrainingJob API.

Key configuration parameters include:

• trainingJobName: Unique identifier for the training job
• roleArn: IAM role ARN with SageMaker execution permissions
• inputDataConfig: S3 URI pointing to your training data
• resourceConfig: Instance type (e.g., ml.p3.2xlarge) and count
• hyperParameters: Fine-tuning variables including learning_rate, epochs, and batch_size

Example configuration snippet:

trainingJobName: genai-gemma-finetune-001
roleArn: arn:aws:iam::123456789012:role/SageMakerExecutionRole
inputDataConfig:
  - channelName: train
    dataSource:
      s3DataSource:
        s3Uri: s3://my-bucket/genai-finetune/input/
        s3DataType: S3Prefix
outputDataConfig:
  s3OutputPath: s3://my-bucket/genai-finetune/output/
resourceConfig:
  instanceType: ml.p3.2xlarge
  instanceCount: 1
  volumeSizeInGB: 50
hyperParameters:
  learning_rate: "5e-5"
  epochs: "3"
  batch_size: "8"

The Docker image referenced in the configuration is built from docker/Dockerfile, which bundles the Gen AI SDK, TRL/PEFT libraries, and the training execution script into the Google-GenAI-SageMaker container.

Launching the Fine-Tuning Job

Use src/run_finetune.py to submit the training job to SageMaker. This script acts as a CLI wrapper that translates your YAML configuration into a SageMaker CreateTrainingJob API call.

Launch the job using the CLI entry point:

genai-sagemaker run-finetune --config config/training_job.yaml

Alternatively, import the Python module directly for programmatic execution:

from src.run_finetune import main
main()

The training container automatically pulls the specified foundation model, applies your dataset for supervised fine-tuning, and pushes the resulting checkpoint to the S3 output location defined in outputDataConfig.

Monitoring Training Progress

Track job status through the SageMaker console, CloudWatch logs, or the command-line interface.

Check job status using the CLI:

genai-sagemaker describe-job --name genai-gemma-finetune-001

The helper scripts in src/utils/ provide additional functionality for streaming logs and polling job completion. Monitor metrics such as loss convergence and training throughput via CloudWatch Logs to detect convergence issues or resource constraints during execution.

Deploying the Fine-Tuned Model

Once training completes, the tuned model artifacts reside in the S3 output path specified in your configuration. Use src/deploy_endpoint.py to create a SageMaker endpoint for real-time inference.

Deploy the model using the CLI:

genai-sagemaker deploy-endpoint \
  --model-artifact s3://my-bucket/genai-finetune/output/model.tar.gz \
  --endpoint-name gemma-finetuned-endpoint \
  --instance-type ml.m5.large

This script creates a SageMaker Model object and deploys it to an HTTPS endpoint, making your fine-tuned model available for inference requests through AWS SDKs or boto3.

Summary

• Data Format: Prepare JSON Lines files with prompt and completion fields, referencing data/sample_dataset.jsonl for schema guidance.

• Configuration: Define training parameters in config/training_job.yaml, specifying instance types, hyperparameters, and S3 locations for input/output data.

• Execution: Submit jobs via src/run_finetune.py or the genai-sagemaker run-finetune CLI command, which invokes the SageMaker CreateTrainingJob API.

• Containerization: The training environment uses a custom Docker image from docker/Dockerfile containing the Gen AI SDK and PEFT libraries.

• Deployment: Create inference endpoints using src/deploy_endpoint.py after training artifacts are written to the S3 output path.

Frequently Asked Questions

What data format does genai-sagemaker require for fine-tuning?

The tooling requires JSON Lines (.jsonl) files where each line contains a JSON object with prompt and completion keys. This format matches the Gen AI Supervised-Tuning API schema, allowing you to provide input-output pairs for supervised fine-tuning of foundation models.

How do I customize hyperparameters for my training job?

Edit the hyperParameters section in config/training_job.yaml to adjust values such as learning_rate, epochs, and batch_size. These parameters are passed directly to the training container built from docker/Dockerfile, which uses TRL/PEFT libraries to configure the fine-tuning process.

Where are the fine-tuned model artifacts stored?

Upon job completion, the training container pushes the model checkpoint to the S3 URI specified in outputDataConfig.s3OutputPath within your YAML configuration. You reference this S3 path when deploying the model using genai-sagemaker deploy-endpoint or src/deploy_endpoint.py.

Can I modify the training container for custom dependencies?

Yes. The docker/Dockerfile defines the Google-GenAI-SageMaker container environment. You can extend this Dockerfile to include additional Python packages or custom training scripts, then reference your rebuilt image in the training configuration's Docker image field before submitting jobs via src/run_finetune.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}