How to Run Google Cloud LLM Demos Locally: Vertex AI Search and RAG Examples

You can run the Vertex AI Search and RAG dual-LLM demos locally by cloning the GoogleCloudPlatform/generative-ai repository, installing dependencies from the demo-specific requirements.txt, authenticating with gcloud auth application-default login, and launching the Streamlit or FastAPI server.

The GoogleCloudPlatform/generative-ai repository hosts self-contained demonstration applications that showcase Vertex AI capabilities, including a Vertex AI Search web app and a RAG dual-LLM Streamlit demo. You can run these LLM demos locally on your workstation to experiment with generative AI features, test custom prompts, and validate configurations before deploying to production.

Clone the Repository and Select a Demo

First, obtain the source code by cloning the repository. The demos reside in specific subdirectories within the search/ folder.

git clone https://github.com/GoogleCloudPlatform/generative-ai.git
cd generative-ai

The repository contains two primary demonstration applications:

• Vertex AI Search Web App: Located in search/web-app/, this FastAPI application demonstrates enterprise search capabilities using Vertex AI Search.
• RAG Dual-LLM Demo: Located in search/retrieval-augmented-generation/rag_with_dual_llms/, this Streamlit application implements retrieval-augmented generation with two LLMs and an optional judge model for evaluation.

Configure the Python Environment

Each demo requires an isolated Python environment to prevent dependency conflicts. Navigate to your chosen demo directory and create a virtual environment before installing any packages.

For the RAG dual-LLM demo:

cd search/retrieval-augmented-generation/rag_with_dual_llms
python3 -m venv .venv
source .venv/bin/activate

For the Vertex AI Search web app:

cd search/web-app
python3 -m venv .venv
source .venv/bin/activate

Install Demo-Specific Dependencies

Install the required packages using the requirements.txt file provided in each demo folder. This ensures you receive compatible versions of the Vertex AI SDK, Streamlit, LangChain, FastAPI, and other dependencies.

pip install --upgrade pip
pip install -r requirements.txt

The requirements.txt in search/retrieval-augmented-generation/rag_with_dual_llms/ specifically pins versions for Streamlit and the Google Cloud AI Platform client library. The search/web-app/requirements.txt includes FastAPI and Uvicorn for the web server, along with the Google Cloud Discovery Engine client.

Authenticate to Google Cloud

The demos invoke Vertex AI services including Gemini, Search, and Embeddings APIs. You must authenticate using Application Default Credentials (ADC) tied to your GCP project.

Run the following command to open a browser-based authentication flow:

gcloud auth application-default login

After authentication, set the GOOGLE_CLOUD_PROJECT environment variable so the code knows which project to bill and which resources to access:

export GOOGLE_CLOUD_PROJECT=<YOUR_PROJECT_ID>

Launch the Demos Locally

Run the RAG Dual-LLM Streamlit Demo

Navigate to the source directory and launch the Streamlit application. The main entry point is src/vertex_rag_demo_dual_llms.py.

streamlit run src/vertex_rag_demo_dual_llms.py

To enable the judge model that evaluates and scores the two LLM responses, use the with_judge variant and pass the --judge flag:

streamlit run src/vertex_rag_demo_dual_llms_with_judge.py -- --judge

Once running, open your browser to http://localhost:8501 to interact with the RAG interface.

Run the Vertex AI Search Web App

For the FastAPI-based web application, launch the development server using Uvicorn. The entry point is main:app in the search/web-app/ directory.

uvicorn main:app --reload

Alternatively, deploy directly to Cloud Run from source without running a local server:

gcloud run deploy vertex-ai-search-demo --source .

When running locally with Uvicorn, visit http://127.0.0.1:8000 to access the search interface.

Summary

• Clone the GoogleCloudPlatform/generative-ai repository to access demo source code in search/web-app/ and search/retrieval-augmented-generation/rag_with_dual_llms/.
• Create a Python virtual environment and install dependencies from the demo-specific requirements.txt files to ensure compatible versions of Streamlit, FastAPI, and the Vertex AI SDK.
• Authenticate using gcloud auth application-default login and set GOOGLE_CLOUD_PROJECT to enable API access.
• Launch the RAG demo with streamlit run src/vertex_rag_demo_dual_llms.py or the web app with uvicorn main:app --reload.
• Access the interfaces at http://localhost:8501 for Streamlit or http://127.0.0.1:8000 for the FastAPI server.

Frequently Asked Questions

What are the system requirements for running these LLM demos locally?

You need Python 3.9 or higher, approximately 500 MB of disk space for the virtual environment, and a Google Cloud project with the Vertex AI API enabled. The demos run on macOS, Linux, and Windows (with WSL recommended for Windows users).

How do I troubleshoot authentication errors when connecting to Vertex AI?

Ensure you have run gcloud auth application-default login and that the GOOGLE_CLOUD_PROJECT environment variable matches a project where you have Vertex AI Administrator or Editor permissions. Verify the Vertex AI API is enabled in the Google Cloud Console under "APIs & Services > Library".

Can I modify the LLM models used in the demos?

Yes. The README files in each demo folder document how to customize the application, including swapping the LLM model name and providing custom prompts. You can edit the model configuration in src/vertex_rag_demo_dual_llms.py or set environment variables as described in the specific demo's README to point to different Gemini or PaLM models in Vertex AI.

How do I clean up resources after running the demos?

If you deployed the Vertex AI Search web app to Cloud Run using gcloud run deploy, delete the service to avoid ongoing charges by running gcloud run services delete vertex-ai-search-demo. For local development servers, simply deactivate the virtual environment with deactivate and delete the .venv directory if desired.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}