# How to Integrate Google's Generative Models Using the generative-ai-gcp Directory

> Integrate Google's generative models using the generative-ai-gcp directory. Learn to build prompts, batch process data, and evaluate outputs with Vertex AI API.

- Repository: [Google Cloud Platform/generative-ai](https://github.com/GoogleCloudPlatform/generative-ai)
- Tags: how-to-guide
- Published: 2026-03-09

---

**You can integrate Google's generative models by using the three helper modules in `tools/llmevalkit/src/`—[`gcp_prompt.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_prompt.py), [`gcp_dataset.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_dataset.py), and [`gcp_evaluation.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_evaluation.py)—to build prompts, batch process data, and evaluate outputs via the Vertex AI API.**

The `generative-ai-gcp` directory within the GoogleCloudPlatform/generative-ai repository provides production-ready Python utilities for calling Gemini and Vertex AI large language models. These lightweight helpers abstract the boilerplate of authentication, prompt construction, and batch inference while maintaining full access to the underlying Vertex AI SDK.

## Overview of the generative-ai-gcp Toolkit

The integration logic is encapsulated in three thin helper modules under `tools/llmevalkit/src/`. Each module handles a distinct stage of the LLM workflow, allowing you to compose solutions without importing the entire Vertex AI SDK boilerplate into your application code.

### Prompt Construction with gcp_prompt.py

The **[`gcp_prompt.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_prompt.py)** module standardizes how you build inputs for Gemini and Vertex AI models. It exposes **`create_prompt()`**, **`add_context()`**, and **`to_message_list()`** to assemble system instructions, few-shot examples, and user queries into the message-list format required by the Gemini API.

### Data Loading with gcp_dataset.py

The **[`gcp_dataset.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_dataset.py)** module handles ingestion from CSV, BigQuery, or JSON sources and prepares data for batch inference. Key functions include **`load_dataset()`** for reading files from Cloud Storage or local paths, and **`batch_iterable()`** for chunking records into batches that respect model token limits.

### Evaluation with gcp_evaluation.py

The **[`gcp_evaluation.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_evaluation.py)** module wraps the Vertex AI Evaluation API to score model outputs. The **`evaluate_responses()`** function computes metrics such as BLEU and Rouge-L, while **`write_to_bq()`** persists results to a BigQuery table defined by the `EVAL_BQ_DATASET` environment variable.

## Authentication and Environment Setup

Before invoking any helper functions, you must authenticate to GCP and configure your environment.

**Application Default Credentials (ADC)** is the recommended approach. Run the following command locally:

```bash
gcloud auth application-default login

```

For production deployments, mount a service-account key and set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. The Quickbot sample application demonstrates this pattern in `gemini/sample-apps/quickbot/backend/Dockerfile.local`:

```dockerfile

# The secret containing the service-account JSON is mounted at /tmp/gcp_adc.json

ENV GOOGLE_APPLICATION_CREDENTIALS=/tmp/gcp_adc.json

```

Define these environment variables in your shell or deployment manifest:

```bash
export GCP_PROJECT_ID="my-gcp-project"
export GCP_REGION="us-central1"
export EVAL_BQ_DATASET="my_eval_dataset"

```

Install the required dependencies:

```bash
pip install -r tools/llmevalkit/requirements.txt

```

## Step-by-Step Integration Workflow

### Building and Sending Prompts

Use `create_prompt()` to assemble your input and `to_message_list()` to convert it into the format expected by the Gemini API.

```python
from tools.llmevalkit.src.gcp_prompt import create_prompt
from vertexai.language_models import TextGenerationModel
import os

project = os.getenv("GCP_PROJECT_ID")
region = os.getenv("GCP_REGION")

# Build a chat-style prompt

prompt = create_prompt(
    system="You are a helpful assistant that always speaks in a friendly tone.",
    user="Explain quantum computing in two sentences."
)

# Initialize the Gemini model

model = TextGenerationModel.from_pretrained(
    "gemini-1.5-pro",
    project=project,
    location=region,
)

# Generate the response

response = model.predict(prompt.to_message_list())
print(response.text)

```

### Processing Large Datasets in Batches

For high-throughput scenarios, use `load_dataset()` to read from Cloud Storage and `batch_iterable()` to chunk inputs.

```python
import pandas as pd
from tools.llmevalkit.src.gcp_dataset import load_dataset, batch_iterable
from tools.llmevalkit.src.gcp_prompt import create_prompt
from vertexai.language_models import TextGenerationModel
import os

df = load_dataset("gs://my-bucket/queries.csv")  # CSV with a column 'question'

batches = batch_iterable(df["question"], batch_size=8)

model = TextGenerationModel.from_pretrained(
    "gemini-1.5-flash",
    project=os.getenv("GCP_PROJECT_ID"),
    location=os.getenv("GCP_REGION"),
)

answers = []
for batch in batches:
    prompts = [create_prompt(user=q).to_message_list() for q in batch]
    resp = model.batch_predict(prompts)
    answers.extend([r.text for r in resp])

df["answer"] = answers
df.to_csv("gs://my-bucket/answers.csv", index=False)

```

### Evaluating Model Outputs

After generating responses, call `evaluate_responses()` to compute metrics and persist them to BigQuery.

```python
from tools.llmevalkit.src.gcp_evaluation import evaluate_responses
import os
import pandas as pd

# Assume df has columns: prompt, model_output, ground_truth

metrics = evaluate_responses(
    dataset=df,
    project=os.getenv("GCP_PROJECT_ID"),
    location=os.getenv("GCP_REGION"),
    bq_dataset=os.getenv("EVAL_BQ_DATASET")
)

print("Overall BLEU:", metrics["bleu"])
print("Average Rouge-L:", metrics["rouge_l"])

```

## Deployment Patterns

### Cloud Functions and Cloud Run

The helper modules are stateless and fit naturally into serverless containers. The Quickbot sample application demonstrates a production-ready Cloud Run deployment. In `gemini/sample-apps/quickbot/backend/Dockerfile.local`, the container mounts a service-account key at [`/tmp/gcp_adc.json`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main//tmp/gcp_adc.json) and sets `GOOGLE_APPLICATION_CREDENTIALS` to enable ADC inside the container.

Your Cloud Function handler can import the same utilities and invoke the model in response to HTTP triggers, reusing the authentication flow described above.

### Vertex AI Notebooks

For interactive development, install the toolkit in a Vertex AI Workbench notebook. The `workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb` notebook demonstrates how to evaluate RAG pipelines using the `gcp_evaluation` module directly within a managed notebook environment.

## Summary

- The **`generative-ai-gcp`** directory provides three minimal helper modules—**[`gcp_prompt.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_prompt.py)**, **[`gcp_dataset.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_dataset.py)**, and **[`gcp_evaluation.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_evaluation.py)**—that wrap the Vertex AI SDK for common LLM workflows.
- **Authentication** relies on Application Default Credentials (ADC) via `gcloud auth application-default login` or a mounted service-account key set via `GOOGLE_APPLICATION_CREDENTIALS`.
- **Prompt construction** uses `create_prompt()` and `to_message_list()` to format inputs for Gemini models like `gemini-1.5-pro` and `gemini-1.5-flash`.
- **Batch processing** leverages `load_dataset()` and `batch_iterable()` to stream large datasets from Cloud Storage and respect token limits.
- **Evaluation** is handled by `evaluate_responses()`, which computes BLEU, Rouge-L, and other metrics and writes results to BigQuery using the `EVAL_BQ_DATASET` environment variable.

## Frequently Asked Questions

### What authentication method does the generative-ai-gcp directory use?

The helper modules rely on **Application Default Credentials (ADC)**. When running locally, execute `gcloud auth application-default login` to populate credentials. In production deployments such as Cloud Run or Cloud Functions, mount a service-account JSON key and set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the key path, as demonstrated in the Quickbot Dockerfile.

### Can I use generative-ai-gcp with custom Vertex AI models instead of Gemini?

Yes. While the examples highlight Gemini models (`gemini-1.5-pro`, `gemini-1.5-flash`), the `TextGenerationModel.from_pretrained()` method accepts any model ID available in your Vertex AI Model Garden. The prompt helpers in [`gcp_prompt.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_prompt.py) generate standard message lists compatible with both Gemini and other Vertex AI LLM endpoints.

### How does batch processing handle model token limits?

The `batch_iterable()` function in [`gcp_dataset.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gcp_dataset.py) chunks input sequences into batches that respect the maximum token limit for the target model. When iterating over a dataset, the helper yields subsets sized to stay within the model's context window, preventing request failures due to payload size while maximizing throughput.

### Where are evaluation metrics stored after running evaluate_responses?

By default, `evaluate_responses()` persists metrics to a **BigQuery** dataset specified by the `EVAL_BQ_DATASET` environment variable. The function writes aggregated scores—such as BLEU and Rouge-L—alongside individual response records to a table within that dataset, enabling downstream analysis and dashboarding in Looker or BigQuery Studio.