# How to Configure GenAI Inference in Agent Platform: A Complete Developer Guide

> Configure GenAI inference in Agent Platform. This guide details enabling the API, granting roles, deploying models, and invoking them with the Gen AI SDK.

- Repository: [Google/skills](https://github.com/google/skills)
- Tags: how-to-guide
- Published: 2026-09-05

---

**To configure GenAI inference in Agent Platform, enable the Agent Platform API, grant the `aiplatform.agent` IAM role, deploy a model from the Model Garden to an endpoint, and invoke it using the Google Gen AI SDK with proper project and location configuration.**

Configuring GenAI inference in Agent Platform requires integrating Google Cloud authentication, endpoint management, and the unified Gen AI SDK. The `google/skills` repository provides authoritative skill documentation and code patterns for implementing this workflow across Python, JavaScript, Go, Java, and C#.

## Prerequisites and IAM Configuration

Before invoking models, you must enable the appropriate APIs and configure service account permissions. According to the [`skills/cloud/agent-platform-inference/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/agent-platform-inference/SKILL.md) file, the inference flow requires explicit IAM bindings and environment variables.

### Enable APIs and Set Permissions

1. **Enable the Agent Platform API** in the Google Cloud Console under **APIs & Services**.
2. **Create a service account** and assign the **`roles/aiplatform.agent`** role, which includes the `aiplatform.endpoints.predict` permission required for inference.
3. **Export environment variables** to ensure the SDK locates the correct regional endpoint:

```bash
export GOOGLE_CLOUD_PROJECT=your-project-id
export GOOGLE_CLOUD_LOCATION=us-central1  # e.g., us-central1, europe-west1

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json

```

## Deploying Models and Creating Endpoints

Agent Platform utilizes the Model Garden to provide access to Gemini and open-source models. As documented in [`skills/cloud/agent-platform-deploy/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/agent-platform-deploy/SKILL.md), you must create an endpoint before deploying a model.

### Create an Endpoint via gcloud

```bash
gcloud ai endpoints create \
    --project=$GOOGLE_CLOUD_PROJECT \
    --region=$GOOGLE_CLOUD_LOCATION \
    --display-name=my-inference-endpoint

```

### Deploy a Model from Model Garden

Deploy a specific model version to your endpoint with accelerator configuration:

```bash
gcloud ai endpoints deploy-model $ENDPOINT_ID \
    --model=$MODEL_ID \
    --machine-type=n1-standard-4 \
    --accelerator-type=NVIDIA_T4 \
    --min-replica-count=1 \
    --max-replica-count=5

```

For Gemini models, you can also use the container image `gcr.io/cloud-aiplatform/prediction/gemini:latest` when uploading custom models.

## Initializing the Gen AI SDK

The [`skills/cloud/gemini-api/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gemini-api/SKILL.md) file specifies that the **Google Gen AI SDK** provides a unified interface for interacting with Agent Platform endpoints. Initialize the client with your project and location context.

### Python SDK Configuration

```python
from google import genai
import os

genai.configure(
    project_id=os.getenv("GOOGLE_CLOUD_PROJECT"),
    location=os.getenv("GOOGLE_CLOUD_LOCATION"),
)

```

### JavaScript/TypeScript SDK Configuration

```typescript
import { GoogleAI } from "@google/genai";

const genai = new GoogleAI({
  projectId: process.env.GOOGLE_CLOUD_PROJECT,
  location: process.env.GOOGLE_CLOUD_LOCATION,
});

```

## Implementing Inference Calls

Once configured, invoke the model using the `generate_content` method (Python) or `generateContent` method (other languages). The following examples demonstrate the inference pattern for supported languages.

### Python Implementation

As shown in the Gemini API skill documentation, use the `genai.generate_content` function with structured parameters:

```python
import os
from google import genai

# Configure client

genai.configure(
    project_id=os.getenv("GOOGLE_CLOUD_PROJECT"),
    location=os.getenv("GOOGLE_CLOUD_LOCATION"),
)

# Generate content

response = genai.generate_content(
    model="gemini-1.5-flash",
    prompt="Explain quantum entanglement in simple terms.",
    generation_config=genai.GenerationConfig(
        temperature=0.7,
        max_output_tokens=1024,
    ),
)

print(response.text)

```

### JavaScript/TypeScript Implementation

```typescript
import { GoogleAI } from "@google/genai";

const genai = new GoogleAI({
  projectId: process.env.GOOGLE_CLOUD_PROJECT!,
  location: process.env.GOOGLE_CLOUD_LOCATION!,
});

const model = genai.getGenerativeModel({ model: "gemini-1.5-flash" });

async function runInference() {
  const result = await model.generateContent(
    "Summarize the benefits of using Agent Platform for LLM inference."
  );
  console.log(result.response.text());
}

runInference();

```

### Go Implementation

The Go SDK resides at `cloud.google.com/go/aiplatform/genai`:

```go
package main

import (
	"context"
	"fmt"
	"os"

	"cloud.google.com/go/aiplatform/genai"
)

func main() {
	ctx := context.Background()
	projectID := os.Getenv("GOOGLE_CLOUD_PROJECT")
	location := os.Getenv("GOOGLE_CLOUD_LOCATION")

	// Create client
	client, err := genai.NewClient(ctx, projectID, location)
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// Get model and generate
	model := client.GenerativeModel("gemini-1.5-flash")
	resp, err := model.GenerateContent(ctx, genai.Text("Explain why LLMs benefit from caching."))
	if err != nil {
		panic(err)
	}
	
	fmt.Println(resp.Candidates[0].Content.Parts[0].Text)
}

```

### Java Implementation

```java
import com.google.cloud.aiplatform.v1.*;
import com.google.protobuf.Value;

public class GenAIInference {
  public static void main(String[] args) throws Exception {
    String project = System.getenv("GOOGLE_CLOUD_PROJECT");
    String location = System.getenv("GOOGLE_CLOUD_LOCATION");
    
    // Construct request
    GenerateContentRequest request = GenerateContentRequest.newBuilder()
        .setModel("gemini-1.5-flash")
        .addContents(Content.newBuilder()
            .addParts(Part.newBuilder()
                .setText("Give a concise definition of reinforcement learning.")
                .build())
            .build())
        .build();

    // Execute request
    try (PredictionServiceClient client = PredictionServiceClient.create()) {
      GenerateContentResponse response = client.generateContent(request);
      System.out.println(response.getCandidates(0).getContent().getParts(0).getText());
    }
  }
}

```

### C# Implementation

```csharp
using Google.Cloud.AIPlatform.V1;
using Google.Protobuf.WellKnownTypes;
using System;

class Program {
    static void Main() {
        var project = Environment.GetEnvironmentVariable("GOOGLE_CLOUD_PROJECT");
        var location = Environment.GetEnvironmentVariable("GOOGLE_CLOUD_LOCATION");
        var client = PredictionServiceClient.Create();

        var request = new GenerateContentRequest {
            Model = "gemini-1.5-flash",
            Contents = {
                new Content {
                    Parts = { new Part { Text = "List three advantages of using Agent Platform for LLM inference." } }
                }
            }
        };

        var response = client.GenerateContent(request);
        Console.WriteLine(response.Candidates[0].Content.Parts[0].Text);
    }
}

```

## Advanced Configuration Options

Beyond basic text generation, Agent Platform supports streaming, batch processing, and caching mechanisms documented across the skill files.

### Streaming with the Live API

For real-time applications, use the **Live API** (streaming) instead of synchronous generation. In Python, initiate a chat session using `genai.start_chat()` followed by `chat.send_message()`, or use the equivalent streaming methods in other languages as detailed in [`skills/cloud/gemini-api/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gemini-api/SKILL.md).

### Batch Prediction

For high-volume workloads, submit a **BatchPredictionJob** rather than synchronous calls. This approach processes large datasets offline and stores results in Cloud Storage, reducing per-request overhead.

### Caching and Performance Optimization

Enable request-level caching via `genai.set_cache_policy()` (Python) or equivalent configurations to reduce latency and costs for repeated prompts. Configure autoscaling policies with `min-replica-count` and `max-replica-count` to balance availability against cost.

## Troubleshooting Common Configuration Issues

When configuring GenAI inference in Agent Platform, verify these specific constraints from the source documentation:

- **Region mismatch errors**: The endpoint `location` must match the SDK client configuration; the SDK raises explicit errors if `GOOGLE_CLOUD_LOCATION` differs from the endpoint's region.
- **Permission denied**: Ensure the service account possesses `aiplatform.endpoints.predict` via the `aiplatform.agent` role, not just `viewer` permissions.
- **Quota exhaustion**: GPU and TPU endpoints consume project-specific quotas; request quota increases before production deployment.
- **Model version compatibility**: Verify that your payload structure matches the model version (e.g., multimodal inputs require specific `Part` configurations).

## Summary

- **Enable the Agent Platform API** and assign the `aiplatform.agent` IAM role to your service account before deploying resources.
- **Create endpoints** using gcloud or the Cloud Console, specifying machine types and accelerators (GPU/TPU) based on latency requirements.
- **Initialize the Gen AI SDK** with `project_id` and `location` parameters to ensure requests route to the correct regional endpoint.
- **Invoke models** using `generate_content` (Python) or `generateContent` (other languages), with optional streaming via the Live API for real-time applications.
- **Reference skill documentation** in `google/skills` at [`skills/cloud/agent-platform-inference/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/agent-platform-inference/SKILL.md) and [`skills/cloud/gemini-api/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gemini-api/SKILL.md) for authoritative syntax and best practices.

## Frequently Asked Questions

### What role permissions are required to run GenAI inference in Agent Platform?

The service account requires the **`roles/aiplatform.agent`** role, which includes the `aiplatform.endpoints.predict` permission. This role allows the account to query deployed endpoints but does not grant broader model training or infrastructure modification rights.

### How do I select between synchronous inference and batch prediction?

Use **synchronous inference** (the `generate_content` method) for real-time, low-latency applications requiring immediate responses. Use **batch prediction** (submitting a `BatchPredictionJob`) when processing large datasets where latency is not critical, as this method costs significantly less per token for bulk workloads.

### Can I use the same SDK configuration for different programming languages?

Yes, the **Google Gen AI SDK** maintains consistent configuration patterns across Python, JavaScript/TypeScript, Go, Java, and C#. All languages require the same `project_id` and `location` parameters during client initialization, though syntax varies (e.g., `genai.configure()` in Python versus `new GoogleAI()` in TypeScript).

### Where are the official code examples for Agent Platform inference hosted?

Official code examples and deployment manifests reside in the `google/skills` repository. Critical files include [`skills/cloud/agent-platform-inference/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/agent-platform-inference/SKILL.md) for inference workflows, [`skills/cloud/gemini-api/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gemini-api/SKILL.md) for SDK usage patterns, and [`skills/cloud/agent-platform-deploy/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/agent-platform-deploy/SKILL.md) for endpoint creation and model lifecycle management.