How to Use FreeLLMAPI with LangChain and LlamaIndex: Complete Integration Guide

Yes, FreeLLMAPI integrates seamlessly with LangChain and LlamaIndex by exposing an OpenAI-compatible REST API that requires only a base URL configuration change to redirect requests from OpenAI's servers to your self-hosted instance.

The tashfeenahmed/freellmapi repository provides a self-hosted HTTP interface that mirrors the OpenAI chat-completion and embeddings schema. Because the request/response format follows the de-facto OpenAI specification, you can use FreeLLMAPI with LangChain or LlamaIndex as a drop-in replacement for OpenAI's official API, routing all LLM calls through your own infrastructure.

Why FreeLLMAPI Works with LangChain and LlamaIndex

FreeLLMAPI implements the standard OpenAI REST API contract, making it compatible with any client library expecting that interface. In server/src/services/router.ts, the Express router registers HTTP endpoints including /v1/chat/completions and /v1/embeddings, accepting requests that match OpenAI's payload structure.

Both LangChain and LlamaIndex expose configuration parameters that override the default OpenAI base URL (https://api.openai.com). By pointing these parameters to your FreeLLMAPI instance (e.g., http://localhost:3000/v1), you create a transparent proxy that routes all LLM and embedding requests through your self-hosted backend. No additional code changes are required beyond these initialization settings.

Configuring LangChain to Use FreeLLMAPI

Required Parameters

To redirect LangChain's OpenAI client to FreeLLMAPI, you must provide two key arguments:

• openai_api_base: Set to your FreeLLMAPI endpoint URL (e.g., http://localhost:3000/v1)
• openai_api_key: Any non-empty string works, as server/src/services/auth.ts validates keys against an internal quota system rather than OpenAI's authentication service

Implementation Example

The following snippet demonstrates a complete LangChain integration using the OpenAI LLM class:

from langchain.llms import OpenAI
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

# Configure the client to use FreeLLMAPI

llm = OpenAI(
    temperature=0.7,
    openai_api_base="http://localhost:3000/v1",  # FreeLLMAPI base URL

    openai_api_key="dummy-key"                    # Any non-empty string works

)

# Create and run a chain

prompt = PromptTemplate.from_template("Write a haiku about {topic}.")
chain = LLMChain(llm=llm, prompt=prompt)

print(chain.run({"topic": "autumn"}))

Integrating LlamaIndex with FreeLLMAPI

Connection Setup

LlamaIndex uses similar configuration options to LangChain. When initializing the OpenAI-compatible LLM, specify:

• api_base: Your FreeLLMAPI server address with the /v1 path
• api_key: A placeholder value (validated locally in server/src/services/auth.ts)

Building Indexes with FreeLLMAPI

You can use FreeLLMAPI as the underlying LLM for document indexing and querying:

from llama_index.llms import OpenAI
from llama_index import SimpleDirectoryReader, GPTVectorStoreIndex

# Point to FreeLLMAPI endpoint

llm = OpenAI(
    temperature=0.5,
    model="gpt-3.5-turbo",
    api_base="http://localhost:3000/v1",   # FreeLLMAPI endpoint

    api_key="any-key"
)

# Build index using FreeLLMAPI for LLM operations

documents = SimpleDirectoryReader("data").load_data()
index = GPTVectorStoreIndex.from_documents(documents, llm=llm)

query_engine = index.as_query_engine()
print(query_engine.query("What are the main features of FreeLLMAPI?"))

Working with Embeddings

FreeLLMAPI provides an /v1/embeddings endpoint implemented in server/src/services/embeddings.ts, supporting vector generation for both LangChain and LlamaIndex. Configure the embedding classes the same way you configure the LLM classes:

from langchain.embeddings import OpenAIEmbeddings

emb = OpenAIEmbeddings(
    model="text-embedding-ada-002",
    openai_api_base="http://localhost:3000/v1",
    openai_api_key="dummy"
)

vector = emb.embed_query("FreeLLMAPI architecture")

This compatibility extends to vector stores, retrievers, and agent tools in both frameworks that rely on OpenAI embedding models.

Key FreeLLMAPI Components for Integration

Understanding the server architecture helps troubleshoot integration issues:

• server/src/services/router.ts: Registers the HTTP routes (/v1/chat/completions, /v1/embeddings) that LangChain and LlamaIndex call
• server/src/services/auth.ts: Validates API keys and enforces per-key usage quotas, accepting any non-empty string for local development
• server/src/services/embeddings.ts: Handles embedding requests with the same request/response contract as OpenAI
• server/src/services/model-listing.ts: Returns available models, which both frameworks may query to validate model names
• server/src/services/scoring.ts: Routes chat completion requests to underlying LLM providers (OpenAI, Anthropic, etc.) based on the requested model

Summary

• FreeLLMAPI exposes OpenAI-compatible endpoints via server/src/services/router.ts, making it interoperable with LangChain and LlamaIndex
• Configure LangChain using openai_api_base and openai_api_key parameters to point to your FreeLLMAPI instance
• Configure LlamaIndex using api_base and api_key parameters with the same endpoint URL
• Embeddings are available through the /v1/embeddings endpoint defined in server/src/services/embeddings.ts
• No code changes are required in your LangChain or LlamaIndex logic beyond the client initialization configuration

Frequently Asked Questions

Is FreeLLMAPI fully compatible with LangChain's OpenAI integration?

Yes. Because FreeLLMAPI implements the standard OpenAI REST API schema in server/src/services/router.ts, LangChain's OpenAI classes work without modification when you override the base URL to point to your FreeLLMAPI server.

What authentication is required when connecting LlamaIndex to FreeLLMAPI?

You must provide any non-empty string as the API key. According to server/src/services/auth.ts, FreeLLMAPI validates keys against its internal quota system rather than OpenAI's authentication, allowing dummy keys like "any-key" or "dummy" for local development.

Can I use FreeLLMAPI embeddings with existing LangChain vector stores?

Yes. The /v1/embeddings endpoint implemented in server/src/services/embeddings.ts follows the OpenAI specification exactly. You can use OpenAIEmbeddings with a custom openai_api_base URL to generate embeddings for any LangChain vector store or LlamaIndex index.

Do I need to modify my existing LangChain or LlamaIndex code to switch to FreeLLMAPI?

No. Both frameworks support configuration parameters that override the default OpenAI base URL. You only need to change initialization parameters (openai_api_base for LangChain, api_base for LlamaIndex) without modifying your chain definitions, prompt templates, or query logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}