Immich Search Service Internals and CLIP Integration: How Smart Search Works

Immich's smart search uses a dual-layer architecture where the SearchService orchestrates CLIP text and image embeddings via the MachineLearningRepository, stores them in PostgreSQL's pgvector extension, and performs nearest-neighbor queries through SearchRepository to return semantically similar assets.

The immich-app/immich repository implements semantic search through a modular pipeline that separates embedding generation from vector retrieval. Understanding these internals reveals how the application converts natural language queries and visual content into searchable vector representations using OpenAI's CLIP models.

Architecture Overview

Immich divides search responsibilities across four core layers:

• API Service Layer – server/src/services/search.service.ts parses requests, validates permissions, and manages an LRU cache for text embeddings.
• Data Access Layer – server/src/repositories/search.repository.ts executes SQL queries against the smart_search table using pgvector operators.
• ML Integration Layer – server/src/services/smart-info.service.ts and server/src/repositories/machine-learning.repository.ts handle CLIP model configuration, dimension synchronization, and communication with the ML micro-service.
• Configuration Layer – server/src/constants.ts defines CLIP_MODEL_INFO with model dimensions, while server/src/utils/misc.ts provides feature-flag helpers like isSmartSearchEnabled.

Search Service Layer

The SearchService class in server/src/services/search.service.ts acts as the orchestration point for all smart search operations.

Request Handling and Caching

Before processing, the service verifies that smart search is enabled via isSmartSearchEnabled(machineLearning) from misc.ts. To minimize latency, the service maintains an in-memory LRU cache with a capacity of 100 entries. The cache key combines modelName + query + language, ensuring that repeated identical queries bypass the ML micro-service.

Smart Search Workflow

When searchSmart receives a request, it executes the following pipeline:

1. Resolves the current configuration and validates smart search availability.
2. Builds a list of accessible user IDs via getUserIdsToSearch.
3. For text queries: Calls machineLearningRepository.encodeText to generate a CLIP embedding, or retrieves the cached vector.
4. For image queries: Fetches the stored embedding via searchRepository.getEmbedding using a reference asset ID.
5. Passes the embedding and pagination parameters to searchRepository.searchSmart.
6. Converts database rows to DTOs using mapResponse.

CLIP Integration and Embedding Generation

CLIP integration spans multiple services to handle model configuration, dimension management, and asynchronous job processing.

Model Configuration and Dimension Management

The SmartInfoService in server/src/services/smart-info.service.ts manages CLIP model lifecycle. Model metadata resides in CLIP_MODEL_INFO within server/src/constants.ts, mapping model names to their vector dimensions (e.g., 512 or 768 dimensions).

During initialization (onConfigInit) and configuration updates (onConfigUpdate), the service:

1. Checks the isSmartSearchEnabled flag.
2. Retrieves the target dimension size via getCLIPModelInfo.
3. Queries the current database dimension using databaseRepository.getDimensionSize('smart_search').
4. If dimensions differ, acquires DatabaseLock.CLIPDimSize and either updates the schema via setDimensionSize or purges existing embeddings via deleteAllSearchEmbeddings.

Image Embedding Pipeline

The handleEncodeClip method processes individual assets through a background job queue:

1. Queue generation: handleQueueEncodeClip streams all assets via assetJobRepository.streamForEncodeClip and enqueues JobName.SmartSearch entries.
2. Single asset encoding: When processing a job, handleEncodeClip:
   • Validates the asset is not hidden or truncated.
   • Invokes machineLearningRepository.encodeImage, sending the file to the ML micro-service at the /predict endpoint with payload { [ModelTask.SEARCH]: { [ModelType.VISUAL]: { modelName } } }.
   • Waits for any concurrent dimension changes (DatabaseLock.CLIPDimSize).
   • Persists the embedding via searchRepository.upsert(asset.id, embedding).

Text Encoding for Queries

For natural language queries, MachineLearningRepository.encodeText constructs a payload using ModelTask.SEARCH and ModelType.TEXTUAL, including the language parameter for multilingual CLIP models. The repository posts to the ML micro-service's /predict endpoint and returns the base64-encoded embedding vector.

Vector Search Implementation

The SearchRepository in server/src/repositories/search.repository.ts executes the actual similarity search using PostgreSQL vector extensions.

When searchSmart receives an embedding vector, it executes:

this.db.transaction().execute(async trx => {
  await sql`set local vchordrq.probes = ${sql.lit(probes[VectorIndex.Clip])}`.execute(trx);
  const items = await searchAssetBuilder(trx, options)
    .selectAll('asset')
    .innerJoin('smart_search', 'asset.id', 'smart_search.assetId')
    .orderBy(sql`smart_search.embedding <=> ${options.embedding}`)   // vector distance
    .limit(pagination.size + 1)
    .offset((pagination.page - 1) * pagination.size)
    .execute();
  return paginationHelper(items, pagination.size);
});

Key implementation details:

• The smart_search table stores CLIP embeddings in a vector column named embedding.
• The <=> operator computes Euclidean distance between vectors (provided by pgvector or vchord extensions).
• vchordrq.probes tunes the search scope for the VectorIndex.Clip index.
• Results are ordered by vector distance and paginated using limit/offset.

Code Examples

Direct HTTP API Call

Query the smart search endpoint directly using cURL:

curl -X POST https://my.immich.app/api/search/smart \
  -H "Authorization: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
        "query": "sunset over mountains",
        "language": "en",
        "page": 1,
        "size": 20,
        "visibility": "timeline"
      }'

TypeScript SDK Usage

Using the official Immich SDK for typed access:

import { ImmichClient } from '@immich/sdk';

const client = new ImmichClient({ baseUrl: 'https://my.immich.app', apiKey: 'your-key' });

const results = await client.searchSmart({
  smartSearchDto: {
    query: 'golden retriever playing fetch',
    language: 'en',
    page: 1,
    size: 15,
    visibility: 'timeline',
  },
});

Manual CLIP Text Encoding

For custom pipelines that need direct access to CLIP embeddings:

import { MachineLearningRepository } from './repositories/machine-learning.repository';
import { ConfigService } from './config/config.service';

const mlRepo = new MachineLearningRepository(configService);
const embedding = await mlRepo.encodeText('vintage car in black and white', {
  language: 'en',
  modelName: 'ViT-B-32__openai', // Must match CLIP_MODEL_INFO keys
});

// embedding is a base64-encoded vector string ready for searchRepository.searchSmart

Triggering Full Re-indexing

Administrators can force regeneration of all CLIP embeddings after model changes:

import { SmartInfoService } from './services/smart-info.service';

// Queue all assets for re-encoding
await smartInfoService.handleQueueEncodeClip({ force: true });

Summary

• Modular Architecture: Immich separates search orchestration (SearchService), vector storage (SearchRepository), and ML inference (SmartInfoService + MachineLearningRepository) into distinct layers.

• CLIP Pipeline: The system uses OpenAI CLIP models to generate fixed-dimension embeddings (512/768D) for both images and text, stored in the smart_search table as PostgreSQL vectors.

• Vector Search: Similarity queries use the <=> Euclidean distance operator against the smart_search.embedding column, with vchordrq.probes tuning for the CLIP vector index.

• Caching & Jobs: An LRU cache prevents redundant text encoding calls, while background jobs (JobName.SmartSearch) handle asynchronous image embedding generation via the ML micro-service.

Frequently Asked Questions

How does Immich handle CLIP model updates or dimension changes?

When the configured CLIP model changes, SmartInfoService detects the dimension mismatch during onConfigInit or onConfigUpdate. It acquires DatabaseLock.CLIPDimSize and either updates the vector column dimension via setDimensionSize or purges existing embeddings using deleteAllSearchEmbeddings, forcing a complete re-index of all assets.

What database extensions are required for smart search to function?

Immich requires PostgreSQL with the pgvector extension (or compatible alternatives like vchord) to store and query vector embeddings. The SQL queries in SearchRepository specifically use the <=> operator for Euclidean distance calculations and set vchordrq.probes to optimize index probing for the CLIP vector index.

Can I use smart search without the machine learning micro-service?

No. The isSmartSearchEnabled helper in server/src/utils/misc.ts explicitly checks for a configured and healthy machine learning URL. If the ML service is unavailable or disabled, SearchService.searchSmart aborts with an error, as the system cannot generate or compare CLIP embeddings without the external inference endpoint.

How does the embedding cache improve search performance?

SearchService maintains an in-memory LRU cache (embeddingCache = new LRUMap<string, string>(100)) keyed by modelName + query + language. When users submit identical text queries, the system returns the cached base64-encoded embedding instead of re-invoking machineLearningRepository.encodeText, reducing latency and ML service load for popular search terms.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}