How the `/search/similar` Endpoint Finds Visually Similar Images

The /search/similar endpoint locates visually similar images by performing a vector-based nearest-neighbor search that retrieves the stored embedding of the requested image and queries Qdrant for the closest matches using cosine similarity.

The hv0905/nekoimagegallery repository implements an AI-powered image gallery that stores vector embeddings in Qdrant to enable semantic search capabilities. When a client requests to find visually similar images, the system leverages these precomputed embeddings to identify neighbors in the vector space without requiring additional inference or image processing.

Handling Similarity Requests in the Controller

When a GET request arrives at /search/similar/{image_id}, the controller in app/Controllers/search.py (lines 105-121) constructs a database query object that encapsulates the search criteria. The system creates a DbQuery containing a single positive criterion of type DbQueryCriteriaId, which holds the UUID of the reference image.

# app/Controllers/search.py (lines 105-121)

query = DbQuery(criteria={
    basis.basis: DbQueryBasis(positive=[DbQueryCriteriaId(id=str(image_id))]),
})

By default, the search uses the vision basis, which instructs the system to compare image vectors rather than text embeddings or other modalities. The basis parameter determines which vector field Qdrant will query during the search operation.

Converting Criteria to Qdrant Queries

Inside app/Services/vector_db_context.py (lines 103-127), the VectorDbContext._convert_basis_to_qdrant_query method processes the DbQueryCriteriaId. The helper function map_criteria extracts the raw ID string from the criterion. When the system detects a single positive entry containing an ID, it instantiates a NearestQuery object.

# app/Services/vector_db_context.py (lines 103-127)

if isinstance(criteria, DbQueryCriteriaId):
    return criteria.id  # ← returns the point ID for Qdrant

# ... 

else:
    return NearestQuery(
        nearest=map_criteria(basis.positive[0]),
    )

Qdrant interprets this NearestQuery containing a point ID as an instruction to use the stored vector of that specific point as the query vector for the similarity search. This avoids the need to resubmit the original image for feature extraction.

Executing the Nearest Neighbor Search

The VectorDbContext.query_search method receives the NearestQuery and executes the search against the Qdrant collection. In app/Services/vector_db_context.py (lines 161-171), the system calls query_points with the image_vector field (mapped via vector_name_for_basis at lines 86-94) to retrieve the top-K closest points.

# app/Services/vector_db_context.py (lines 161-171)

result = await self._client.query_points(
    collection_name=self.collection_name,
    query=self._convert_basis_to_qdrant_query(criteria),
    using=self.vector_name_for_basis(basis),   # maps vision → "image_vector"

    query_filter=filters,
    limit=top_k,
    offset=skip,
    with_payload=True
)

Qdrant calculates cosine similarity between the reference image's embedding and all other stored vectors, returning the closest matches according to angular distance in the high-dimensional vector space.

Response Processing and Delivery

After retrieving the raw SearchResult objects from Qdrant, the controller's query_and_postprocess method wraps them into a SearchApiResponse. If the deployment uses remote storage, the system generates presigned URLs for both full images and thumbnails before serializing the response to JSON.

Practical API Usage

To find visually similar images via the API, send an authenticated GET request with the target image UUID:

import requests

# Example: find images similar to a known UUID

response = requests.get(
    "https://my-neko-gallery.com/search/similar/3fa85f64-5717-4562-b3fc-2c963f66afa6",
    params={"count": 10, "skip": 0},
    headers={"Authorization": "Bearer <access-token>"}
)
print(response.json())

Summary

• The /search/similar endpoint uses stored vector embeddings to find visually similar images without recomputing features or processing the original image bytes.
• The search pipeline converts image UUIDs into NearestQuery objects that instruct Qdrant to use the reference vector as the query point.
• Qdrant executes cosine similarity calculations against the image_vector field to retrieve nearest neighbors based on angular distance.
• Key implementation files include app/Controllers/search.py for request handling and app/Services/vector_db_context.py for query translation and execution.

Frequently Asked Questions

What vector database does NekoImageGallery use to find visually similar images?

NekoImageGallery uses Qdrant as its vector database to store and query image embeddings. The system calls Qdrant's query_points method with NearestQuery objects to perform approximate nearest neighbor searches using cosine similarity metrics against the stored vector collection.

How does the system handle the search when only an image ID is provided?

When only an image ID is provided, the system retrieves the precomputed vector embedding associated with that UUID from Qdrant's storage. In app/Services/vector_db_context.py, the DbQueryCriteriaId is converted to a NearestQuery containing the point ID, which tells Qdrant to use that stored vector as the query vector for the similarity search rather than accepting a raw vector array.

What similarity metric is used to determine visual similarity?

The system uses cosine similarity to measure the angular distance between vector embeddings in high-dimensional space. When query_points is executed with the image_vector field, Qdrant returns points that are closest to the query vector according to cosine distance, ensuring that images with similar visual features appear in the results.

Can the /search/similar endpoint search using modalities other than vision?

While the vision basis is the default for finding visually similar images, the architecture supports multiple search bases through the SearchBasisEnum. The vector_name_for_basis method in app/Services/vector_db_context.py (lines 86-94) maps different bases to their respective vector fields in Qdrant, though the /search/similar endpoint specifically targets the image_vector field for visual similarity queries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}