# Immich Search Service Internals and CLIP Integration: How Smart Search Works

> Discover Immich smart search internals. Learn how CLIP integration with pgvector enables semantic asset search through a dual-layer architecture.

- Repository: [Immich/immich](https://github.com/immich-app/immich)
- Tags: internals
- Published: 2026-02-27

---

**Immich's smart search uses a dual-layer architecture where the SearchService orchestrates CLIP text and image embeddings via the MachineLearningRepository, stores them in PostgreSQL's pgvector extension, and performs nearest-neighbor queries through SearchRepository to return semantically similar assets.**

The immich-app/immich repository implements semantic search through a modular pipeline that separates embedding generation from vector retrieval. Understanding these internals reveals how the application converts natural language queries and visual content into searchable vector representations using OpenAI's CLIP models.

## Architecture Overview

Immich divides search responsibilities across four core layers:

- **API Service Layer** – [`server/src/services/search.service.ts`](https://github.com/immich-app/immich/blob/main/server/src/services/search.service.ts) parses requests, validates permissions, and manages an LRU cache for text embeddings.
- **Data Access Layer** – [`server/src/repositories/search.repository.ts`](https://github.com/immich-app/immich/blob/main/server/src/repositories/search.repository.ts) executes SQL queries against the `smart_search` table using pgvector operators.
- **ML Integration Layer** – [`server/src/services/smart-info.service.ts`](https://github.com/immich-app/immich/blob/main/server/src/services/smart-info.service.ts) and [`server/src/repositories/machine-learning.repository.ts`](https://github.com/immich-app/immich/blob/main/server/src/repositories/machine-learning.repository.ts) handle CLIP model configuration, dimension synchronization, and communication with the ML micro-service.
- **Configuration Layer** – [`server/src/constants.ts`](https://github.com/immich-app/immich/blob/main/server/src/constants.ts) defines `CLIP_MODEL_INFO` with model dimensions, while [`server/src/utils/misc.ts`](https://github.com/immich-app/immich/blob/main/server/src/utils/misc.ts) provides feature-flag helpers like `isSmartSearchEnabled`.

## Search Service Layer

The `SearchService` class in [`server/src/services/search.service.ts`](https://github.com/immich-app/immich/blob/main/server/src/services/search.service.ts) acts as the orchestration point for all smart search operations.

### Request Handling and Caching

Before processing, the service verifies that smart search is enabled via `isSmartSearchEnabled(machineLearning)` from [`misc.ts`](https://github.com/immich-app/immich/blob/main/misc.ts). To minimize latency, the service maintains an in-memory LRU cache with a capacity of 100 entries. The cache key combines `modelName + query + language`, ensuring that repeated identical queries bypass the ML micro-service.

### Smart Search Workflow

When `searchSmart` receives a request, it executes the following pipeline:

1. Resolves the current configuration and validates smart search availability.
2. Builds a list of accessible user IDs via `getUserIdsToSearch`.
3. **For text queries**: Calls `machineLearningRepository.encodeText` to generate a CLIP embedding, or retrieves the cached vector.
4. **For image queries**: Fetches the stored embedding via `searchRepository.getEmbedding` using a reference asset ID.
5. Passes the embedding and pagination parameters to `searchRepository.searchSmart`.
6. Converts database rows to DTOs using `mapResponse`.

## CLIP Integration and Embedding Generation

CLIP integration spans multiple services to handle model configuration, dimension management, and asynchronous job processing.

### Model Configuration and Dimension Management

The `SmartInfoService` in [`server/src/services/smart-info.service.ts`](https://github.com/immich-app/immich/blob/main/server/src/services/smart-info.service.ts) manages CLIP model lifecycle. Model metadata resides in `CLIP_MODEL_INFO` within [`server/src/constants.ts`](https://github.com/immich-app/immich/blob/main/server/src/constants.ts), mapping model names to their vector dimensions (e.g., 512 or 768 dimensions).

During initialization (`onConfigInit`) and configuration updates (`onConfigUpdate`), the service:

1. Checks the `isSmartSearchEnabled` flag.
2. Retrieves the target dimension size via `getCLIPModelInfo`.
3. Queries the current database dimension using `databaseRepository.getDimensionSize('smart_search')`.
4. If dimensions differ, acquires `DatabaseLock.CLIPDimSize` and either updates the schema via `setDimensionSize` or purges existing embeddings via `deleteAllSearchEmbeddings`.

### Image Embedding Pipeline

The `handleEncodeClip` method processes individual assets through a background job queue:

1. **Queue generation**: `handleQueueEncodeClip` streams all assets via `assetJobRepository.streamForEncodeClip` and enqueues `JobName.SmartSearch` entries.
2. **Single asset encoding**: When processing a job, `handleEncodeClip`:
   - Validates the asset is not hidden or truncated.
   - Invokes `machineLearningRepository.encodeImage`, sending the file to the ML micro-service at the `/predict` endpoint with payload `{ [ModelTask.SEARCH]: { [ModelType.VISUAL]: { modelName } } }`.
   - Waits for any concurrent dimension changes (`DatabaseLock.CLIPDimSize`).
   - Persists the embedding via `searchRepository.upsert(asset.id, embedding)`.

### Text Encoding for Queries

For natural language queries, `MachineLearningRepository.encodeText` constructs a payload using `ModelTask.SEARCH` and `ModelType.TEXTUAL`, including the language parameter for multilingual CLIP models. The repository posts to the ML micro-service's `/predict` endpoint and returns the base64-encoded embedding vector.

## Vector Search Implementation

The `SearchRepository` in [`server/src/repositories/search.repository.ts`](https://github.com/immich-app/immich/blob/main/server/src/repositories/search.repository.ts) executes the actual similarity search using PostgreSQL vector extensions.

When `searchSmart` receives an embedding vector, it executes:

```typescript
this.db.transaction().execute(async trx => {
  await sql`set local vchordrq.probes = ${sql.lit(probes[VectorIndex.Clip])}`.execute(trx);
  const items = await searchAssetBuilder(trx, options)
    .selectAll('asset')
    .innerJoin('smart_search', 'asset.id', 'smart_search.assetId')
    .orderBy(sql`smart_search.embedding <=> ${options.embedding}`)   // vector distance
    .limit(pagination.size + 1)
    .offset((pagination.page - 1) * pagination.size)
    .execute();
  return paginationHelper(items, pagination.size);
});

```

Key implementation details:

- The `smart_search` table stores CLIP embeddings in a vector column named `embedding`.
- The `<=>` operator computes Euclidean distance between vectors (provided by `pgvector` or `vchord` extensions).
- `vchordrq.probes` tunes the search scope for the `VectorIndex.Clip` index.
- Results are ordered by vector distance and paginated using limit/offset.

## Code Examples

### Direct HTTP API Call

Query the smart search endpoint directly using cURL:

```bash
curl -X POST https://my.immich.app/api/search/smart \
  -H "Authorization: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
        "query": "sunset over mountains",
        "language": "en",
        "page": 1,
        "size": 20,
        "visibility": "timeline"
      }'

```

### TypeScript SDK Usage

Using the official Immich SDK for typed access:

```typescript
import { ImmichClient } from '@immich/sdk';

const client = new ImmichClient({ baseUrl: 'https://my.immich.app', apiKey: 'your-key' });

const results = await client.searchSmart({
  smartSearchDto: {
    query: 'golden retriever playing fetch',
    language: 'en',
    page: 1,
    size: 15,
    visibility: 'timeline',
  },
});

```

### Manual CLIP Text Encoding

For custom pipelines that need direct access to CLIP embeddings:

```typescript
import { MachineLearningRepository } from './repositories/machine-learning.repository';
import { ConfigService } from './config/config.service';

const mlRepo = new MachineLearningRepository(configService);
const embedding = await mlRepo.encodeText('vintage car in black and white', {
  language: 'en',
  modelName: 'ViT-B-32__openai', // Must match CLIP_MODEL_INFO keys
});

// embedding is a base64-encoded vector string ready for searchRepository.searchSmart

```

### Triggering Full Re-indexing

Administrators can force regeneration of all CLIP embeddings after model changes:

```typescript
import { SmartInfoService } from './services/smart-info.service';

// Queue all assets for re-encoding
await smartInfoService.handleQueueEncodeClip({ force: true });

```

## Summary

- **Modular Architecture**: Immich separates search orchestration (`SearchService`), vector storage (`SearchRepository`), and ML inference (`SmartInfoService` + `MachineLearningRepository`) into distinct layers.

- **CLIP Pipeline**: The system uses OpenAI CLIP models to generate fixed-dimension embeddings (512/768D) for both images and text, stored in the `smart_search` table as PostgreSQL vectors.

- **Vector Search**: Similarity queries use the `<=>` Euclidean distance operator against the `smart_search.embedding` column, with `vchordrq.probes` tuning for the CLIP vector index.

- **Caching & Jobs**: An LRU cache prevents redundant text encoding calls, while background jobs (`JobName.SmartSearch`) handle asynchronous image embedding generation via the ML micro-service.

## Frequently Asked Questions

### How does Immich handle CLIP model updates or dimension changes?

When the configured CLIP model changes, `SmartInfoService` detects the dimension mismatch during `onConfigInit` or `onConfigUpdate`. It acquires `DatabaseLock.CLIPDimSize` and either updates the vector column dimension via `setDimensionSize` or purges existing embeddings using `deleteAllSearchEmbeddings`, forcing a complete re-index of all assets.

### What database extensions are required for smart search to function?

Immich requires PostgreSQL with the `pgvector` extension (or compatible alternatives like `vchord`) to store and query vector embeddings. The SQL queries in `SearchRepository` specifically use the `<=>` operator for Euclidean distance calculations and set `vchordrq.probes` to optimize index probing for the CLIP vector index.

### Can I use smart search without the machine learning micro-service?

No. The `isSmartSearchEnabled` helper in [`server/src/utils/misc.ts`](https://github.com/immich-app/immich/blob/main/server/src/utils/misc.ts) explicitly checks for a configured and healthy machine learning URL. If the ML service is unavailable or disabled, `SearchService.searchSmart` aborts with an error, as the system cannot generate or compare CLIP embeddings without the external inference endpoint.

### How does the embedding cache improve search performance?

`SearchService` maintains an in-memory LRU cache (`embeddingCache = new LRUMap<string, string>(100)`) keyed by `modelName + query + language`. When users submit identical text queries, the system returns the cached base64-encoded embedding instead of re-invoking `machineLearningRepository.encodeText`, reducing latency and ML service load for popular search terms.