How to Select the Right Foundation Model for Your Use Case: A 3-Stage Engineering Approach

Selecting the right foundation model requires defining evaluation axes specific to your product, gathering candidates that meet deployment constraints, and running a private benchmark to create a weighted ranking based on latency, cost, and accuracy trade-offs.

Choosing a foundation model is a systematic engineering decision, not guesswork. According to the AI Engineering (AIE) book by Chip Huyen, effective model selection follows a rigorous three-stage workflow that transforms vague intuition into quantitative evidence. This approach, documented in chapter-summaries.md (lines 96-100), treats the model as a black-box component that must satisfy a contract defined by your specific requirements.

The Three-Stage Selection Framework

The AIE book outlines a private "leaderboard" methodology consisting of three tightly-coupled stages.

Stage 1: Define the Evaluation Axes

Before comparing models, list the dimensions that matter for your product. Common axes include data privacy, latency, cost, factual consistency, safety, and domain-specific knowledge. This step guarantees you compare models on the exact trade-offs your stakeholders care about, as outlined in the build-vs-buy decision criteria in chapter-summaries.md (line 96).

Stage 2: Gather Candidate Models

Pull in publicly available models that meet basic constraints such as API access, open-source licensing, size limits, or multimodal support. This creates a manageable shortlist before expensive testing begins. The book notes that while thousands of public benchmarks exist, they should only serve as rough filters due to data contamination risks (documented in chapter-summaries.md around line 100).

Stage 3: Run a Private Benchmark

Build a small, task-specific test set and run each candidate through it. Record scores on every axis, then rank models using a weighted sum or multi-objective optimizer (e.g., Pareto front). This private leaderboard approach surfaces hidden failure modes such as contamination, hallucinations, or latency spikes that public benchmarks miss.

Critical Architectural Considerations

Beyond the three-stage workflow, the repository emphasizes several architectural principles that impact how you select the right foundation model for your use case.

Evaluation-First Mindset

Treat the model as a black-box component that must satisfy a contract defined by your evaluation axes. This mirrors classic software component testing and makes later integration steps—prompt engineering, RAG, or fine-tuning—much safer and more predictable.

Host vs. API Deployment Analysis

Decide whether you will self-host or consume an API before evaluating model quality. This decision influences latency, privacy, and cost, thereby changing the relative importance of your evaluation axes (referenced in chapter-summaries.md line 98). Self-hosted models offer data sovereignty but require infrastructure overhead, while API solutions provide scalability with potential vendor lock-in.

Public Benchmark Limitations

Public leaderboards like HELM or MMLU are useful for rough filtering but can be contaminated because many models have seen the benchmark data during training. Use them only as sanity checks; the decisive ranking must come from your private benchmark on held-out, domain-specific data.

Implementing the Selection Workflow in Code

Here is a complete Python implementation that executes the three-stage workflow for a text-generation use case. The script evaluates OpenAI, Anthropic, and Cohere APIs on latency, cost, and factual consistency:

import time, os, json, requests
from typing import List, Dict

# ----------------------------------------------------------------------

# 1️⃣  Define evaluation axes and their weights (adjust per project)

# ----------------------------------------------------------------------

AXES = {
    "latency_ms": 0.3,      # lower is better

    "cost_per_1k": 0.2,     # lower is better

    "factual_score": 0.5   # higher is better (0‑1)

}

# ----------------------------------------------------------------------

# 2️⃣  Candidate model descriptors (API endpoint + pricing)

# ----------------------------------------------------------------------

CANDIDATES = [
    {
        "name": "gpt‑4‑o",
        "url": "https://api.openai.com/v1/chat/completions",
        "key_env": "OPENAI_API_KEY",
        "cost_per_1k": 0.005,          # USD per 1k tokens (example)

        "model_arg": "gpt-4o"
    },
    {
        "name": "claude‑3‑sonnet",
        "url": "https://api.anthropic.com/v1/messages",
        "key_env": "ANTHROPIC_API_KEY",
        "cost_per_1k": 0.003,
        "model_arg": "claude-3-sonnet-20240229"
    },
    {
        "name": "cohere‑command‑r",
        "url": "https://api.cohere.com/v1/chat",
        "key_env": "COHERE_API_KEY",
        "cost_per_1k": 0.0015,
        "model_arg": "command-r"
    }
]

# ----------------------------------------------------------------------

# Helper: call a model and measure latency

# ----------------------------------------------------------------------

def query_model(candidate: dict, prompt: str) -> Dict:
    api_key = os.getenv(candidate["key_env"])
    if not api_key:
        raise RuntimeError(f"Missing {candidate['key_env']} env var")
    headers = {"Authorization": f"Bearer {api_key}"}
    payload = {
        "model": candidate["model_arg"],
        "messages": [{"role": "user", "content": prompt}]
    }

    start = time.time()
    resp = requests.post(candidate["url"], json=payload, headers=headers, timeout=30)
    latency = (time.time() - start) * 1000  # ms

    resp.raise_for_status()
    text = resp.json()["choices"][0]["message"]["content"]
    return {"text": text, "latency_ms": latency}

# ----------------------------------------------------------------------

# 3️⃣  Simple factual consistency check (self‑check prompt)

# ----------------------------------------------------------------------

def factual_score(response: str) -> float:
    # Very naive check: ask the model to self‑evaluate truthfulness.

    # In production you'd use a dedicated LLM‑judge or human eval.

    check_prompt = (
        f"On a scale from 0 to 1, how confident are you that the following answer is factually correct?\n"
        f"Answer: {response}"
    )
    # Re‑use the same model (e.g., GPT‑4) as the judge

    judge = CANDIDATES[0]  # assume first candidate is a strong judge

    out = query_model(judge, check_prompt)
    try:
        return float(out["text"].strip().split()[0])
    except Exception:
        return 0.0

# ----------------------------------------------------------------------

# 4️⃣  Run private benchmark on a small test set

# ----------------------------------------------------------------------

TEST_QUESTIONS = [
    "What is the capital of France?",
    "Explain the core idea of the transformer architecture in two sentences."
]

def evaluate_candidate(candidate: dict) -> Dict:
    scores = {"latency_ms": 0, "cost_per_1k": candidate["cost_per_1k"], "factual_score": 0}
    total_latency = 0
    total_factual = 0
    for q in TEST_QUESTIONS:
        out = query_model(candidate, q)
        total_latency += out["latency_ms"]
        total_factual += factual_score(out["text"])
    scores["latency_ms"] = total_latency / len(TEST_QUESTIONS)
    scores["factual_score"] = total_factual / len(TEST_QUESTIONS)
    return scores

# ----------------------------------------------------------------------

# 5️⃣  Weighted ranking

# ----------------------------------------------------------------------

def weighted_rank(results: List[Dict]) -> List[Dict]:
    ranked = []
    for r in results:
        # Normalize latency (lower is better) and factual (higher is better)

        # Here we simply invert latency and scale to 0‑1 using a naive max value.

        # In a real system you would compute min/max across candidates.

        norm_latency = 1 / (r["latency_ms"] / 1000)  # crude normalization

        score = (
            norm_latency * AXES["latency_ms"]
            + (1 - r["cost_per_1k"] / max(c["cost_per_1k"] for c in CANDIDATES)) * AXES["cost_per_1k"]
            + r["factual_score"] * AXES["factual_score"]
        )
        ranked.append({**r, "overall_score": score})
    return sorted(ranked, key=lambda x: x["overall_score"], reverse=True)

# ----------------------------------------------------------------------

# Execute

# ----------------------------------------------------------------------

if __name__ == "__main__":
    all_results = []
    for c in CANDIDATES:
        print(f"Evaluating {c['name']} …")
        metrics = evaluate_candidate(c)
        all_results.append({**c, **metrics})
    final_ranking = weighted_rank(all_results)
    print("\n=== Model Ranking ===")
    for i, entry in enumerate(final_ranking, 1):
        print(f"{i}. {entry['name']}: overall={entry['overall_score']:.3f}, latency={entry['latency_ms']:.1f} ms, cost=${entry['cost_per_1k']:.4f}/k, factual={entry['factual_score']:.2f}")

This script demonstrates several key concepts from the repository:

• Axis definition: The AXES dictionary allows you to plug in additional dimensions (privacy, token limits, safety scores) and adjust weights according to stakeholder priorities.
• Candidate abstraction: Each model descriptor includes endpoint URLs, pricing, and environment variable requirements for API keys.
• Real-world latency measurement: The query_model function captures actual response times using time.time(), critical for production SLAs.
• Weighted ranking: The weighted_rank function combines heterogeneous metrics into a single decision score, implementing the private leaderboard concept described in chapter-summaries.md.

Iterative Refinement and Documentation

The first round of benchmarking usually reveals gaps—perhaps a model is cheap but fails factual consistency tests. Narrow the shortlist, add domain-specific data through RAG or PEFT (Parameter-Efficient Fine-Tuning), and re-benchmark. This loop forms the core of the AI Engineering workflow.

Document the weighting you applied to each axis and the final scores. This documentation proves valuable for future product revisions, compliance reviews, and communicating decisions to non-technical stakeholders. The case-studies.md and appendix.md files in the repository provide concrete examples and checklists for this documentation process.

Summary

• Define evaluation axes specific to your product requirements (latency, cost, privacy, accuracy) before comparing models.
• Gather candidates that meet deployment constraints (API vs. self-hosted, licensing, modalities).
• Build a private benchmark on held-out, domain-specific data to avoid contaminated public leaderboard scores.
• Apply a weighted ranking or Pareto optimization to transform multi-dimensional trade-offs into a clear decision.
• Iterate by refining your test set and re-benchmarking after integrating RAG or fine-tuning.

Frequently Asked Questions

How many models should I include in my initial candidate shortlist?

Start with 3-5 models that meet your hard constraints (API availability, licensing, context window size). Including too many candidates initially dilutes focus and increases evaluation costs. Use public benchmarks only as a rough filter to create this shortlist, then rely on your private benchmark for the final ranking.

What is the difference between public and private benchmarks in foundation model selection?

Public benchmarks like MMLU or HELM provide standardized comparisons across many models but suffer from data contamination—models may have been trained on the test questions. Private benchmarks use your own held-out, domain-specific data that models haven't seen, providing accurate signal for your specific use case. According to chapter-summaries.md (line 100), decisive rankings should come from private evaluations.

How do I weigh latency versus accuracy when selecting a foundation model?

Create a weighted scoring formula where latency (lower is better) and accuracy (higher is better) contribute to a final score based on your product requirements. For user-facing chat applications, latency might carry 40% weight while accuracy carries 60%. For batch processing pipelines, latency might drop to 10% while accuracy rises to 70%. The Python example above shows how to implement this normalization and weighting in code.

Should I choose an API or self-hosted deployment for my foundation model?

Choose API deployment when you need rapid scaling, don't want to manage infrastructure, and can accept vendor-dependent pricing and data handling policies. Choose self-hosted deployment when you require strict data privacy (keeping data on-premise), need to minimize per-token costs at scale, or must customize the model architecture. This decision should occur before model evaluation, as it changes which candidates are viable and how you weight axes like cost and latency (as noted in chapter-summaries.md line 98).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}