# How Does In-Context Learning Work in LLMs? A Technical Deep Dive

> Discover how in-context learning enables LLMs to tackle new tasks using prompt examples without weight updates. A technical deep dive into ICL.

- Repository: [Chip Huyen/aie-book](https://github.com/chiphuyen/aie-book)
- Tags: deep-dive
- Published: 2026-04-24

---

**In-context learning (ICL) allows large language models to perform new tasks by learning from examples embedded directly in the prompt, without updating any model weights.**

According to the `chiphuyen/aie-book` repository, this capability emerges from the transformer architecture's ability to treat the entire prompt as a unified context window. The model processes instructions and demonstrations as a single token sequence, using its pre-trained attention mechanisms to infer task patterns during the forward pass.

## Architectural Mechanisms of In-Context Learning

### Prompt Tokenization and Context Window

When you submit a prompt to an LLM, the entire text—including instructions, examples, and the query—is tokenized and fed into the model as a contiguous sequence. As noted in the repository's conceptual overview, the model does not distinguish between "training" and "test" data within this window. Instead, the stacked self-attention layers process the full context uniformly, allowing information to flow between demonstration examples and the target query.

### Self-Attention Pattern Recognition

The core mechanism enabling ICL is **self-attention**. Each token in the sequence attends to every other token, creating a fully connected graph of dependencies. When the model sees input-output pairs (e.g., "English: Hello → French: Bonjour"), the attention heads identify the mapping pattern in real time. These heads leverage knowledge acquired during pre-training to extrapolate the demonstrated behavior to new inputs without parameter changes.

### Implicit Gradient-Free Adaptation

Because the model's weights remain frozen during inference, adaptation occurs implicitly through **contextual bias**. The transformer's feed-forward networks and layer normalization operations shift hidden representations based on surrounding tokens. This effectively "programs" the model for the specific task by adjusting internal activations, creating a temporary, context-specific computation graph.

### Impact of Model Scale

Larger models exhibit qualitatively different in-context learning behaviors. The repository cites research from [`resources.md`](https://github.com/chiphuyen/aie-book/blob/main/resources.md) (line 188) indicating that bigger transformers possess higher-capacity attention patterns and richer world knowledge, making implicit adaptation more reliable and robust. Scale not only improves pattern recognition but also enables the model to handle more complex, multi-step reasoning within the prompt context.

### Task Alignment Through Prompting

As summarized in [`chapter-summaries.md`](https://github.com/chiphuyen/aie-book/blob/main/chapter-summaries.md) (line 118), the prompt functions as a *mini-instruction set* that aligns the model's internal representations with desired outputs. Simple structural cues—such as "think step-by-step" or clear instruction headers—help attention mechanisms latch onto task patterns more effectively by creating semantic anchors within the context window.

## Key Factors That Influence ICL Performance

Several variables determine how effectively a model leverages in-context examples:

- **Prompt length and quality**: Longer, well-structured prompts provide more contextual signal for the model to infer task boundaries and desired formats, as documented in the prompt engineering guides.

- **Example diversity**: Diverse demonstrations reduce over-fitting to spurious patterns, improving generalization to novel inputs. The [`prompt-examples.md`](https://github.com/chiphuyen/aie-book/blob/main/prompt-examples.md) file contains templates showing varied formatting approaches.

- **Instruction clarity**: Explicit task descriptors act as anchors that guide attention heads toward relevant feature dimensions, a concept emphasized in Chapter 5's summary.

- **Demonstration ordering**: The sequence of examples affects attention weight distribution, with certain orderings producing better contextual bias than others.

## Practical Implementation: Code Examples

The following implementations demonstrate in-context learning using the OpenAI API. The same principles apply to Anthropic, Cohere, and other transformer-based services.

### Few-Shot Translation with OpenAI API

This example shows how to construct a prompt with two translation pairs to teach the model a specific mapping:

```python
import openai

openai.api_key = "YOUR_OPENAI_API_KEY"   # ← keep this secret

def translate_few_shot(text: str) -> str:
    # Few-shot prompt: two example pairs + the new query

    prompt = """Translate English to French.

English: I love programming.
French: J'aime programmer.

English: The weather is nice today.
French: Le temps est agréable aujourd'hui.

English: {input}
French:""".format(input=text)

    response = openai.Completion.create(
        model="gpt-4o-mini",
        prompt=prompt,
        max_tokens=64,
        temperature=0.0,   # deterministic for demo

        stop=["\n"]
    )
    return response.choices[0].text.strip()

print(translate_few_shot("Artificial intelligence is fascinating."))

```

The model processes the three input-output pairs as in-context examples, allowing the attention mechanism to infer the "English → French" transformation pattern and apply it to the novel sentence.

### Step-by-Step Math Reasoning

This implementation uses chain-of-thought prompting, an advanced ICL technique that encourages the model to demonstrate its reasoning process:

```python
def solve_math(question: str) -> str:
    prompt = f"""Solve the problem step by step.

Q: 12 × 15
A: 12 × 15 = (10 + 2) × 15 = 150 + 30 = 180

Q: {question}
A:"""

    resp = openai.Completion.create(
        model="gpt-4o",
        prompt=prompt,
        temperature=0.0,
        max_tokens=200,
        stop=["\n\n"]
    )
    return resp.choices[0].text.strip()

print(solve_math("What is 23²?"))

```

The explicit "step-by-step" cue steers the model toward chain-of-thought reasoning, leveraging the demonstration to structure its internal computation trajectory.

## Source Files and References in chiphuyen/aie-book

The repository contains several files critical to understanding ICL implementation:

- **[`resources.md`](https://github.com/chiphuyen/aie-book/blob/main/resources.md)**: Contains the paper "Larger language models do in-context learning differently" and links to scaling studies (line 188).

- **[`chapter-summaries.md`](https://github.com/chiphuyen/aie-book/blob/main/chapter-summaries.md)**: Chapter 5 summary discusses why in-context learning works and provides prompt engineering guidance (line 118).

- **[`prompt-examples.md`](https://github.com/chiphuyen/aie-book/blob/main/prompt-examples.md)**: Concrete real-world templates adaptable for ICL experiments.

- **[`README.md`](https://github.com/chiphuyen/aie-book/blob/main/README.md)**: Overview of the book's scope with links to chapters covering ICL theory and practice.

## Summary

- **In-context learning** enables zero-shot and few-shot task performance by embedding examples directly in the prompt, requiring no weight updates.

- The mechanism relies on **self-attention** processing the entire prompt as a unified sequence, allowing the model to infer patterns during the forward pass.

- Adaptation occurs through **contextual bias** in frozen parameters, not gradient descent, making ICL computationally efficient compared to fine-tuning.

- **Model scale** significantly impacts ICL quality, with larger transformers demonstrating more robust pattern recognition and generalization.

- Effective ICL requires careful prompt engineering, including **diverse examples**, **clear instructions**, and **proper formatting** to guide attention mechanisms.

## Frequently Asked Questions

### What is the difference between in-context learning and fine-tuning?

**Fine-tuning** updates the model's weights through backpropagation on a training dataset, permanently altering its parameters. **In-context learning** leaves weights frozen and instead adapts behavior by conditioning the model on specific examples within the prompt. According to the `chiphuyen/aie-book` repository, ICL provides temporary task adaptation through attention mechanisms, making it faster to deploy but potentially less consistent than fine-tuning for specialized tasks.

### How many examples are needed for effective in-context learning?

The number of examples depends on task complexity and model size. Simple pattern-matching tasks may require only **1-3 examples**, while complex reasoning or multi-step transformations might benefit from **5-10 diverse demonstrations**. Research cited in [`resources.md`](https://github.com/chiphuyen/aie-book/blob/main/resources.md) suggests that larger models can extract meaningful patterns from fewer examples due to their increased attention capacity, though providing multiple diverse examples generally improves robustness against edge cases.

### Why does prompt formatting significantly impact ICL performance?

Formatting affects how the self-attention mechanism partitions the input sequence. When examples follow a consistent template (e.g., "Input: X Output: Y"), the model's attention heads learn to attend to specific position-based patterns and delimiter tokens. The [`chapter-summaries.md`](https://github.com/chiphuyen/aie-book/blob/main/chapter-summaries.md) file (line 118) notes that explicit structural cues like "think step-by-step" create **task descriptors** that align the model's computation toward specific reasoning paths, effectively programming its behavior without weight updates.

### Do smaller models support in-context learning effectively?

While smaller models can perform basic in-context learning, their capacity for complex pattern recognition is limited. The repository references research indicating that **model scale** is a critical factor for ICL capability. Smaller transformers have fewer attention heads and reduced representational capacity, making them more sensitive to prompt formatting and less reliable at extracting subtle patterns from limited examples. For production ICL applications, models above 6B parameters typically demonstrate substantially more robust performance.