LLM Demos for Multiple Providers in the Google Cloud Generative AI Repository

The GoogleCloudPlatform/generative-ai repository does not contain a centralized LLM-demos folder, but instead distributes production-ready code examples for OpenAI, Anthropic Claude, vLLM, and other providers across open-models/, partner-models/, and tools/ directories.

The GoogleCloudPlatform/generative-ai repository serves as the definitive resource for implementing generative AI solutions on Google Cloud. While you will not find a single LLM-demos directory at the repository root, the project contains extensive LLM demos for different providers organized by use case and vendor integration. These notebooks demonstrate authentication patterns, endpoint configuration, and inference calls for major large language model services using the Google Cloud Generative AI SDKs.

Where to Find Provider-Specific LLM Demos

The repository organizes provider examples into three primary locations:

• open-models/ – Contains notebooks for OpenAI-compatible endpoints and open-source model serving via vLLM.
• partner-models/ – Houses Anthropic Claude integration examples and other third-party model demonstrations.
• tools/llmevalkit/ – Provides a Streamlit-based evaluation interface that can target any configured LLM provider.

OpenAI-Compatible Demos on Vertex AI

Google Cloud provides OpenAI-compatible endpoints that allow you to use the official OpenAI Python client against Vertex AI infrastructure. The repository contains two primary patterns for this integration.

Using vLLM with OpenAI Client

The notebook open-models/serving/cloud_run_vllm_gemma3_inference.ipynb demonstrates deploying Gemma models via vLLM on Cloud Run, which exposes an OpenAI-compatible REST API.

# Install the OpenAI client (vLLM provides an OpenAI-compatible server)

%pip install -q openai
from openai import OpenAI

# Vertex AI endpoint that mimics the OpenAI REST API

client = OpenAI(
    base_url="https://us-central1-aiplatform.googleapis.com/v1/endpoints/ENDPOINT_ID:predict",
    api_key="YOUR_TOKEN"
)
response = client.chat.completions.create(
    model="gemma-2b",
    messages=[{"role": "user", "content": "Tell me a joke"}]
)
print(response.choices[0].message.content)

Direct OpenAI SDK Against Vertex AI Endpoints

The open-models/get_started_with_model_garden_sdk.ipynb notebook shows how to use the OpenAI SDK directly with Vertex AI Model Garden endpoints.

%pip install -q openai google-cloud-aiplatform
import openai
import google.auth

creds, _ = google.auth.default()
client = openai.OpenAI(
    base_url="https://vertexai.googleapis.com/v1/projects/PROJECT/locations/us-central1/endpoints/ENDPOINT:predict",
    api_key=creds.token
)
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this paragraph"}]
)
print(resp.choices[0].message.content)

Anthropic Claude Integration Examples

For organizations using Anthropic's Claude models, the repository provides dedicated integration patterns through the partner-models/claude/ directory.

Multimodal Function Calling with Claude

The partner-models/claude/claude_multimodal_function_calling.ipynb notebook demonstrates Claude-3-Sonnet usage, including tool-calling and multimodal inputs.

%pip install -q anthropic[vertex]
from anthropic import AnthropicVertex

# Vertex AI automatically routes to the Claude model you have enabled

client = AnthropicVertex()
response = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Write a haiku about clouds"}]
)
print(response.content)

Agent Engine with Claude

The agents/agent_engine/tutorial_claude_with_adk_on_agent_engine.ipynb file provides an end-to-end agent-engine example that runs Claude via anthropic[vertex].

Open-Source Model Serving with vLLM

For teams deploying open-source models like Gemma, the repository includes vLLM serving examples that expose OpenAI-compatible endpoints.

The open-models/serving/cloud_run_vllm_gemma3_inference.ipynb notebook demonstrates deploying vLLM on Cloud Run.

%pip install -q openai
from openai import OpenAI

# The container runs vLLM exposing an OpenAI-compatible server on port 8000

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.completions.create(
    model="gemma-2b",
    prompt="Explain quantum computing in one sentence."
)
print(resp.choices[0].text)

LLM Evaluation and Migration Tools

Beyond inference examples, the repository provides utilities for evaluating and migrating between providers.

LLM EvalKit Interface

The tools/llmevalkit/index.py file implements a Streamlit application for managing prompts and running evaluations against any configured LLM provider.

import streamlit as st

st.title("LLM EvalKit")

# Prompt management page loads prompts from a CSV, then sends them to the selected LLM provider

if st.button("Run evaluation"):
    # Example using Vertex AI OpenAI-compatible client

    from openai import OpenAI
    client = OpenAI(base_url=VAI_ENDPOINT, api_key=TOKEN)
    result = client.chat.completions.create(...)
    st.success("Evaluation completed")

OpenAI to Gemini Migration

The migration/migrate_from_openai_to_gemini.ipynb notebook provides a structured guide for transitioning code from OpenAI's API to Google's Gemini API, including side-by-side code comparisons.

Summary

• The GoogleCloudPlatform/generative-ai repository does not contain a single LLM-demos directory, but instead organizes LLM demos for different providers across specialized folders.
• OpenAI-compatible examples reside in open-models/serving/ and demonstrate both vLLM serving and direct SDK usage against Vertex AI endpoints.
• Anthropic Claude integrations are located in partner-models/claude/ and include multimodal function calling and Agent Engine tutorials.
• Open-source model serving examples use vLLM to expose OpenAI-compatible APIs for models like Gemma.
• Evaluation and migration tools in tools/llmevalkit/ and migration/ support provider-agnostic testing and API transitions.

Frequently Asked Questions

Is there a single LLM-demos folder in the GoogleCloudPlatform/generative-ai repository?

No. The repository does not contain a top-level LLM-demos directory. Instead, provider-specific examples are distributed across folders like open-models/, partner-models/, and tools/llmevalkit/ based on integration type and use case.

Which LLM providers have dedicated code examples in the repository?

The repository includes examples for OpenAI-compatible endpoints (via Vertex AI and vLLM), Anthropic Claude (via Vertex AI Model Garden), and open-source models served through vLLM. Additionally, migration guides exist for transitioning from OpenAI to Google Gemini APIs.

How do I authenticate with different LLM providers in these examples?

Authentication patterns vary by provider. For OpenAI-compatible Vertex AI endpoints, you typically use Google OAuth credentials via google.auth.default(). For Anthropic Claude via Vertex AI, the anthropic[vertex] package handles authentication automatically using your Google Cloud credentials. Local vLLM deployments often use an empty API key string.

Can I use these demos to evaluate multiple LLM providers simultaneously?

Yes. The tools/llmevalkit/index.py Streamlit application provides a unified interface for loading prompts from CSV files and running evaluations against any configured LLM provider. This allows you to compare performance across OpenAI-compatible endpoints, Claude, and other supported models within a single workflow.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}