# LLM Demos for Multiple Providers in the Google Cloud Generative AI Repository

> Explore LLM demos for OpenAI, Anthropic Claude, and vLLM in Google Cloud's generative AI repository. Find production-ready code examples across multiple directories for seamless integration.

- Repository: [Google Cloud Platform/generative-ai](https://github.com/GoogleCloudPlatform/generative-ai)
- Tags: deep-dive
- Published: 2026-03-09

---

**The GoogleCloudPlatform/generative-ai repository does not contain a centralized `LLM-demos` folder, but instead distributes production-ready code examples for OpenAI, Anthropic Claude, vLLM, and other providers across `open-models/`, `partner-models/`, and `tools/` directories.**

The GoogleCloudPlatform/generative-ai repository serves as the definitive resource for implementing generative AI solutions on Google Cloud. While you will not find a single `LLM-demos` directory at the repository root, the project contains extensive **LLM demos for different providers** organized by use case and vendor integration. These notebooks demonstrate authentication patterns, endpoint configuration, and inference calls for major large language model services using the Google Cloud Generative AI SDKs.

## Where to Find Provider-Specific LLM Demos

The repository organizes provider examples into three primary locations:

- **`open-models/`** – Contains notebooks for OpenAI-compatible endpoints and open-source model serving via vLLM.
- **`partner-models/`** – Houses Anthropic Claude integration examples and other third-party model demonstrations.
- **`tools/llmevalkit/`** – Provides a Streamlit-based evaluation interface that can target any configured LLM provider.

## OpenAI-Compatible Demos on Vertex AI

Google Cloud provides OpenAI-compatible endpoints that allow you to use the official OpenAI Python client against Vertex AI infrastructure. The repository contains two primary patterns for this integration.

### Using vLLM with OpenAI Client

The notebook `open-models/serving/cloud_run_vllm_gemma3_inference.ipynb` demonstrates deploying Gemma models via vLLM on Cloud Run, which exposes an OpenAI-compatible REST API.

```python

# Install the OpenAI client (vLLM provides an OpenAI-compatible server)

%pip install -q openai
from openai import OpenAI

# Vertex AI endpoint that mimics the OpenAI REST API

client = OpenAI(
    base_url="https://us-central1-aiplatform.googleapis.com/v1/endpoints/ENDPOINT_ID:predict",
    api_key="YOUR_TOKEN"
)
response = client.chat.completions.create(
    model="gemma-2b",
    messages=[{"role": "user", "content": "Tell me a joke"}]
)
print(response.choices[0].message.content)

```

### Direct OpenAI SDK Against Vertex AI Endpoints

The `open-models/get_started_with_model_garden_sdk.ipynb` notebook shows how to use the OpenAI SDK directly with Vertex AI Model Garden endpoints.

```python
%pip install -q openai google-cloud-aiplatform
import openai
import google.auth

creds, _ = google.auth.default()
client = openai.OpenAI(
    base_url="https://vertexai.googleapis.com/v1/projects/PROJECT/locations/us-central1/endpoints/ENDPOINT:predict",
    api_key=creds.token
)
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this paragraph"}]
)
print(resp.choices[0].message.content)

```

## Anthropic Claude Integration Examples

For organizations using Anthropic's Claude models, the repository provides dedicated integration patterns through the `partner-models/claude/` directory.

### Multimodal Function Calling with Claude

The `partner-models/claude/claude_multimodal_function_calling.ipynb` notebook demonstrates Claude-3-Sonnet usage, including tool-calling and multimodal inputs.

```python
%pip install -q anthropic[vertex]
from anthropic import AnthropicVertex

# Vertex AI automatically routes to the Claude model you have enabled

client = AnthropicVertex()
response = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Write a haiku about clouds"}]
)
print(response.content)

```

### Agent Engine with Claude

The `agents/agent_engine/tutorial_claude_with_adk_on_agent_engine.ipynb` file provides an end-to-end agent-engine example that runs Claude via `anthropic[vertex]`.

## Open-Source Model Serving with vLLM

For teams deploying open-source models like Gemma, the repository includes vLLM serving examples that expose OpenAI-compatible endpoints.

The `open-models/serving/cloud_run_vllm_gemma3_inference.ipynb` notebook demonstrates deploying vLLM on Cloud Run.

```python
%pip install -q openai
from openai import OpenAI

# The container runs vLLM exposing an OpenAI-compatible server on port 8000

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.completions.create(
    model="gemma-2b",
    prompt="Explain quantum computing in one sentence."
)
print(resp.choices[0].text)

```

## LLM Evaluation and Migration Tools

Beyond inference examples, the repository provides utilities for evaluating and migrating between providers.

### LLM EvalKit Interface

The [`tools/llmevalkit/index.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/tools/llmevalkit/index.py) file implements a Streamlit application for managing prompts and running evaluations against any configured LLM provider.

```python
import streamlit as st

st.title("LLM EvalKit")

# Prompt management page loads prompts from a CSV, then sends them to the selected LLM provider

if st.button("Run evaluation"):
    # Example using Vertex AI OpenAI-compatible client

    from openai import OpenAI
    client = OpenAI(base_url=VAI_ENDPOINT, api_key=TOKEN)
    result = client.chat.completions.create(...)
    st.success("Evaluation completed")

```

### OpenAI to Gemini Migration

The `migration/migrate_from_openai_to_gemini.ipynb` notebook provides a structured guide for transitioning code from OpenAI's API to Google's Gemini API, including side-by-side code comparisons.

## Summary

- The **GoogleCloudPlatform/generative-ai** repository does not contain a single `LLM-demos` directory, but instead organizes **LLM demos for different providers** across specialized folders.
- **OpenAI-compatible** examples reside in `open-models/serving/` and demonstrate both vLLM serving and direct SDK usage against Vertex AI endpoints.
- **Anthropic Claude** integrations are located in `partner-models/claude/` and include multimodal function calling and Agent Engine tutorials.
- **Open-source model serving** examples use vLLM to expose OpenAI-compatible APIs for models like Gemma.
- **Evaluation and migration tools** in `tools/llmevalkit/` and `migration/` support provider-agnostic testing and API transitions.

## Frequently Asked Questions

### Is there a single LLM-demos folder in the GoogleCloudPlatform/generative-ai repository?

No. The repository does not contain a top-level `LLM-demos` directory. Instead, provider-specific examples are distributed across folders like `open-models/`, `partner-models/`, and `tools/llmevalkit/` based on integration type and use case.

### Which LLM providers have dedicated code examples in the repository?

The repository includes examples for OpenAI-compatible endpoints (via Vertex AI and vLLM), Anthropic Claude (via Vertex AI Model Garden), and open-source models served through vLLM. Additionally, migration guides exist for transitioning from OpenAI to Google Gemini APIs.

### How do I authenticate with different LLM providers in these examples?

Authentication patterns vary by provider. For OpenAI-compatible Vertex AI endpoints, you typically use Google OAuth credentials via `google.auth.default()`. For Anthropic Claude via Vertex AI, the `anthropic[vertex]` package handles authentication automatically using your Google Cloud credentials. Local vLLM deployments often use an empty API key string.

### Can I use these demos to evaluate multiple LLM providers simultaneously?

Yes. The [`tools/llmevalkit/index.py`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/tools/llmevalkit/index.py) Streamlit application provides a unified interface for loading prompts from CSV files and running evaluations against any configured LLM provider. This allows you to compare performance across OpenAI-compatible endpoints, Claude, and other supported models within a single workflow.