How to Configure GenAI Inference in Agent Platform: A Complete Developer Guide

To configure GenAI inference in Agent Platform, enable the Agent Platform API, grant the aiplatform.agent IAM role, deploy a model from the Model Garden to an endpoint, and invoke it using the Google Gen AI SDK with proper project and location configuration.

Configuring GenAI inference in Agent Platform requires integrating Google Cloud authentication, endpoint management, and the unified Gen AI SDK. The google/skills repository provides authoritative skill documentation and code patterns for implementing this workflow across Python, JavaScript, Go, Java, and C#.

Prerequisites and IAM Configuration

Before invoking models, you must enable the appropriate APIs and configure service account permissions. According to the skills/cloud/agent-platform-inference/SKILL.md file, the inference flow requires explicit IAM bindings and environment variables.

Enable APIs and Set Permissions

1. Enable the Agent Platform API in the Google Cloud Console under APIs & Services.
2. Create a service account and assign the roles/aiplatform.agent role, which includes the aiplatform.endpoints.predict permission required for inference.
3. Export environment variables to ensure the SDK locates the correct regional endpoint:

export GOOGLE_CLOUD_PROJECT=your-project-id
export GOOGLE_CLOUD_LOCATION=us-central1  # e.g., us-central1, europe-west1

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json

Deploying Models and Creating Endpoints

Agent Platform utilizes the Model Garden to provide access to Gemini and open-source models. As documented in skills/cloud/agent-platform-deploy/SKILL.md, you must create an endpoint before deploying a model.

Create an Endpoint via gcloud

gcloud ai endpoints create \
    --project=$GOOGLE_CLOUD_PROJECT \
    --region=$GOOGLE_CLOUD_LOCATION \
    --display-name=my-inference-endpoint

Deploy a Model from Model Garden

Deploy a specific model version to your endpoint with accelerator configuration:

gcloud ai endpoints deploy-model $ENDPOINT_ID \
    --model=$MODEL_ID \
    --machine-type=n1-standard-4 \
    --accelerator-type=NVIDIA_T4 \
    --min-replica-count=1 \
    --max-replica-count=5

For Gemini models, you can also use the container image gcr.io/cloud-aiplatform/prediction/gemini:latest when uploading custom models.

Initializing the Gen AI SDK

The skills/cloud/gemini-api/SKILL.md file specifies that the Google Gen AI SDK provides a unified interface for interacting with Agent Platform endpoints. Initialize the client with your project and location context.

Python SDK Configuration

from google import genai
import os

genai.configure(
    project_id=os.getenv("GOOGLE_CLOUD_PROJECT"),
    location=os.getenv("GOOGLE_CLOUD_LOCATION"),
)

JavaScript/TypeScript SDK Configuration

import { GoogleAI } from "@google/genai";

const genai = new GoogleAI({
  projectId: process.env.GOOGLE_CLOUD_PROJECT,
  location: process.env.GOOGLE_CLOUD_LOCATION,
});

Implementing Inference Calls

Once configured, invoke the model using the generate_content method (Python) or generateContent method (other languages). The following examples demonstrate the inference pattern for supported languages.

Python Implementation

As shown in the Gemini API skill documentation, use the genai.generate_content function with structured parameters:

import os
from google import genai

# Configure client

genai.configure(
    project_id=os.getenv("GOOGLE_CLOUD_PROJECT"),
    location=os.getenv("GOOGLE_CLOUD_LOCATION"),
)

# Generate content

response = genai.generate_content(
    model="gemini-1.5-flash",
    prompt="Explain quantum entanglement in simple terms.",
    generation_config=genai.GenerationConfig(
        temperature=0.7,
        max_output_tokens=1024,
    ),
)

print(response.text)

JavaScript/TypeScript Implementation

import { GoogleAI } from "@google/genai";

const genai = new GoogleAI({
  projectId: process.env.GOOGLE_CLOUD_PROJECT!,
  location: process.env.GOOGLE_CLOUD_LOCATION!,
});

const model = genai.getGenerativeModel({ model: "gemini-1.5-flash" });

async function runInference() {
  const result = await model.generateContent(
    "Summarize the benefits of using Agent Platform for LLM inference."
  );
  console.log(result.response.text());
}

runInference();

Go Implementation

The Go SDK resides at cloud.google.com/go/aiplatform/genai:

package main

import (
	"context"
	"fmt"
	"os"

	"cloud.google.com/go/aiplatform/genai"
)

func main() {
	ctx := context.Background()
	projectID := os.Getenv("GOOGLE_CLOUD_PROJECT")
	location := os.Getenv("GOOGLE_CLOUD_LOCATION")

	// Create client
	client, err := genai.NewClient(ctx, projectID, location)
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// Get model and generate
	model := client.GenerativeModel("gemini-1.5-flash")
	resp, err := model.GenerateContent(ctx, genai.Text("Explain why LLMs benefit from caching."))
	if err != nil {
		panic(err)
	}
	
	fmt.Println(resp.Candidates[0].Content.Parts[0].Text)
}

Java Implementation

import com.google.cloud.aiplatform.v1.*;
import com.google.protobuf.Value;

public class GenAIInference {
  public static void main(String[] args) throws Exception {
    String project = System.getenv("GOOGLE_CLOUD_PROJECT");
    String location = System.getenv("GOOGLE_CLOUD_LOCATION");
    
    // Construct request
    GenerateContentRequest request = GenerateContentRequest.newBuilder()
        .setModel("gemini-1.5-flash")
        .addContents(Content.newBuilder()
            .addParts(Part.newBuilder()
                .setText("Give a concise definition of reinforcement learning.")
                .build())
            .build())
        .build();

    // Execute request
    try (PredictionServiceClient client = PredictionServiceClient.create()) {
      GenerateContentResponse response = client.generateContent(request);
      System.out.println(response.getCandidates(0).getContent().getParts(0).getText());
    }
  }
}

C# Implementation

using Google.Cloud.AIPlatform.V1;
using Google.Protobuf.WellKnownTypes;
using System;

class Program {
    static void Main() {
        var project = Environment.GetEnvironmentVariable("GOOGLE_CLOUD_PROJECT");
        var location = Environment.GetEnvironmentVariable("GOOGLE_CLOUD_LOCATION");
        var client = PredictionServiceClient.Create();

        var request = new GenerateContentRequest {
            Model = "gemini-1.5-flash",
            Contents = {
                new Content {
                    Parts = { new Part { Text = "List three advantages of using Agent Platform for LLM inference." } }
                }
            }
        };

        var response = client.GenerateContent(request);
        Console.WriteLine(response.Candidates[0].Content.Parts[0].Text);
    }
}

Advanced Configuration Options

Beyond basic text generation, Agent Platform supports streaming, batch processing, and caching mechanisms documented across the skill files.

Streaming with the Live API

For real-time applications, use the Live API (streaming) instead of synchronous generation. In Python, initiate a chat session using genai.start_chat() followed by chat.send_message(), or use the equivalent streaming methods in other languages as detailed in skills/cloud/gemini-api/SKILL.md.

Batch Prediction

For high-volume workloads, submit a BatchPredictionJob rather than synchronous calls. This approach processes large datasets offline and stores results in Cloud Storage, reducing per-request overhead.

Caching and Performance Optimization

Enable request-level caching via genai.set_cache_policy() (Python) or equivalent configurations to reduce latency and costs for repeated prompts. Configure autoscaling policies with min-replica-count and max-replica-count to balance availability against cost.

Troubleshooting Common Configuration Issues

When configuring GenAI inference in Agent Platform, verify these specific constraints from the source documentation:

• Region mismatch errors: The endpoint location must match the SDK client configuration; the SDK raises explicit errors if GOOGLE_CLOUD_LOCATION differs from the endpoint's region.
• Permission denied: Ensure the service account possesses aiplatform.endpoints.predict via the aiplatform.agent role, not just viewer permissions.
• Quota exhaustion: GPU and TPU endpoints consume project-specific quotas; request quota increases before production deployment.
• Model version compatibility: Verify that your payload structure matches the model version (e.g., multimodal inputs require specific Part configurations).

Summary

• Enable the Agent Platform API and assign the aiplatform.agent IAM role to your service account before deploying resources.
• Create endpoints using gcloud or the Cloud Console, specifying machine types and accelerators (GPU/TPU) based on latency requirements.
• Initialize the Gen AI SDK with project_id and location parameters to ensure requests route to the correct regional endpoint.
• Invoke models using generate_content (Python) or generateContent (other languages), with optional streaming via the Live API for real-time applications.
• Reference skill documentation in google/skills at skills/cloud/agent-platform-inference/SKILL.md and skills/cloud/gemini-api/SKILL.md for authoritative syntax and best practices.

Frequently Asked Questions

What role permissions are required to run GenAI inference in Agent Platform?

The service account requires the roles/aiplatform.agent role, which includes the aiplatform.endpoints.predict permission. This role allows the account to query deployed endpoints but does not grant broader model training or infrastructure modification rights.

How do I select between synchronous inference and batch prediction?

Use synchronous inference (the generate_content method) for real-time, low-latency applications requiring immediate responses. Use batch prediction (submitting a BatchPredictionJob) when processing large datasets where latency is not critical, as this method costs significantly less per token for bulk workloads.

Can I use the same SDK configuration for different programming languages?

Yes, the Google Gen AI SDK maintains consistent configuration patterns across Python, JavaScript/TypeScript, Go, Java, and C#. All languages require the same project_id and location parameters during client initialization, though syntax varies (e.g., genai.configure() in Python versus new GoogleAI() in TypeScript).

Where are the official code examples for Agent Platform inference hosted?

Official code examples and deployment manifests reside in the google/skills repository. Critical files include skills/cloud/agent-platform-inference/SKILL.md for inference workflows, skills/cloud/gemini-api/SKILL.md for SDK usage patterns, and skills/cloud/agent-platform-deploy/SKILL.md for endpoint creation and model lifecycle management.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}