What Is the File Indexing Manager in OpenCTI and How Does It Work?

The file indexing manager in OpenCTI is a background module that maintains a searchable Elasticsearch or OpenSearch index of every platform attachment, synchronizing file metadata and content with entity-level access controls in real time.

This specialized manager ensures that uploaded documents, reports, and artifacts remain discoverable through the platform’s search interface while respecting dynamic security markings. According to the OpenCTI-Platform/opencti source code, the feature is available exclusively in the Enterprise Edition and operates through a combination of periodic batch jobs and event-driven stream processing.

Core Architecture and Responsibilities

The file indexing manager serves as the bridge between OpenCTI’s file storage layer and its search infrastructure. It extracts binary content, generates internal identifiers, and maintains strict consistency between file accessibility and parent entity restrictions.

Periodic File Ingestion

At its core, the manager runs a locked-resource job every minute (configurable via file_index_manager:interval) to index newly imported files. The fileIndexHandler function in src/manager/fileIndexManager.ts acquires a distributed lock using FILE_INDEX_MANAGER_KEY to prevent duplicate execution across clustered API nodes.

Once locked, the handler verifies Enterprise Edition status through isEnterpriseEditionFromSettings, retrieves the last successful run timestamp via getIndexFromDate, and delegates to indexImportedFiles. This function processes files in batches of 20, streaming content with BluePromise.map at a concurrency of 5 to balance throughput with resource utilization.

Real-Time Stream Processing

Beyond periodic scans, the manager listens to the platform’s event stream to handle immediate updates. The handleStreamEvents function monitors UPDATE events for STIX objects that own files. When the patch contains granted_refs or object_marking_refs changes, the manager triggers elUpdateFilesWithEntityRestrictions to propagate access control modifications to the search index without waiting for the next batch cycle.

Enterprise Edition Requirements

The manager is strictly an Enterprise-only feature. The bootstrap logic in src/managers.js checks ENABLED_FILE_INDEX_MANAGER and isAttachmentProcessorEnabled() before invoking fileIndexManager.start(). Additionally, the runtime handlers verify isEnterpriseEditionFromSettings before executing indexing operations, ensuring community editions skip the overhead entirely.

Configuration and Initialization

System administrators control the manager through settings defined in src/config/conf.ts. Key configuration parameters include:

• file_index_manager:interval – Polling frequency in milliseconds (default: 60000 for 1 minute)
• file_index_manager:lock_key – Distributed lock identifier for the periodic job
• file_index_manager:stream_lock_key – Lock key for the stream processor
• ENABLED_FILE_INDEX_MANAGER – Boolean flag enabling the module

The initialization sequence in src/managers.js demonstrates the conditional startup:

import fileIndexManager from './manager/fileIndexManager';

if (ENABLED_FILE_INDEX_MANAGER && isAttachmentProcessorEnabled()) {
  await fileIndexManager.start();
}

The start method creates two asynchronous intervals: one for the periodic indexing job and one for the stream handler, both managed through the initFileIndexManager lifecycle.

Indexing Workflow and Code Implementation

The fileIndexHandler Loop

The primary indexing logic resides in src/manager/fileIndexManager.ts. The handler implements a defensive locking pattern to ensure singleton execution across distributed deployments:

const fileIndexHandler = async () => {
  const settings = await getEntityFromCache(context, SYSTEM_USER, ENTITY_TYPE_SETTINGS);
  if (isEnterpriseEditionFromSettings(settings)) {
    const lock = await lockResources([FILE_INDEX_MANAGER_KEY], { retryCount: 0 });
    try {
      const managerConfiguration = await getManagerConfigurationFromCache(context, SYSTEM_USER, 'FILE_INDEX_MANAGER');
      if (managerConfiguration?.manager_running) {
        const indexFromDate = await getIndexFromDate(context);
        await indexImportedFiles(context, indexFromDate);
      }
    } finally {
      if (lock) await lock.unlock();
    }
  }
};

This pattern ensures that only one node processes files at any given time, while the retryCount: 0 parameter causes immediate exit if another node holds the lock.

Batch Processing and Content Extraction

The indexImportedFiles function (lines 69-78 in the manager file) queries for files created after the last indexed date, validates enabled MIME-type prefixes, and constructs documents containing:

• File ID and generated internal OpenCTI identifier (via generateFileIndexId from src/schema/identifier.ts)
• Owner entity reference (if applicable)
• File name and upload date
• Base64-encoded binary content

These documents are sent to elIndexFiles in src/database/file-search.ts for bulk insertion into the search cluster.

Handling Entity Restriction Updates

When security markings change, the stream processor ensures index consistency:

if (event.data.type === EVENT_TYPE_UPDATE) {
  const updateEvent = event.data;
  const isDataRestrictionsUpdate = updateEvent.context?.patch?.some(
    op => op.path?.includes('granted_refs') || op.path?.includes('object_marking_refs')
  );
  if (stixFiles?.length && isDataRestrictionsUpdate) {
    const entity = await internalLoadById(context, SYSTEM_USER, entityId, { type: entityType });
    await elUpdateFilesWithEntityRestrictions(entity);
  }
}

This code path prevents search results from returning files the user no longer has permission to access following entity updates.

GraphQL API and Domain Functions

The platform exposes file indexing operations through GraphQL resolvers defined in src/resolvers/indexedFile.js, with domain logic implemented in src/domain/indexedFile.ts.

Search and Metrics Queries

• indexedFilesMetrics – Returns statistics about the file index state
• indexedFiles – Paginated search results with filtering capabilities
• indexedFilesCount – Aggregate count matching specified criteria

Resetting the Index

Administrators can completely rebuild the index using the resetFileIndexing mutation:

mutation {
  resetFileIndexing
}

The resolver invokes resetFileIndexing(context, user) from the domain layer, which performs three critical operations: clearing the manager configuration state, deleting all indexed documents via elDeleteAllFiles, and publishing a user-action audit event. This function is located at lines 42-62 of src/domain/indexedFile.ts.

Graceful Shutdown

The shutdown method in initFileIndexManager clears active intervals and terminates the stream processor, ensuring no partial indexing operations corrupt the state during platform restarts.

Summary

• The file indexing manager in OpenCTI maintains a real-time Elasticsearch/OpenSearch index of all platform attachments, enabling full-text search across uploaded content.
• It operates through dual mechanisms: a periodic batch job (default 1-minute intervals) processing 20 files per batch, and a stream processor reacting immediately to entity permission changes.
• The feature requires Enterprise Edition licensing and explicit configuration via ENABLED_FILE_INDEX_MANAGER in src/config/conf.ts.
• Core implementation files include src/manager/fileIndexManager.ts for orchestration, src/domain/indexedFile.ts for GraphQL domain logic, and src/database/file-search.ts for low-level Elasticsearch operations.
• Distributed locking via FILE_INDEX_MANAGER_KEY prevents race conditions in clustered deployments, while BluePromise.map concurrency controls resource utilization during bulk indexing.

Frequently Asked Questions

Does the file indexing manager work in OpenCTI Community Edition?

No. The file indexing manager is an Enterprise-only feature that requires a valid Enterprise Edition license. The platform explicitly checks isEnterpriseEditionFromSettings in fileIndexHandler, fileIndexStreamHandler, and the status endpoint before executing any indexing operations. Community Edition deployments will skip initialization entirely based on the ENABLED_FILE_INDEX_MANAGER flag check in src/managers.js.

How can administrators reset or rebuild the file index?

Administrators can trigger a complete index reset through the GraphQL API using the resetFileIndexing mutation. This operation, implemented in src/domain/indexedFile.ts, clears the manager’s configuration cache, deletes all existing indexed documents via elDeleteAllFiles, and publishes an audit event. Alternatively, modifying the file_index_manager:interval configuration and restarting the platform allows for gradual re-indexing without data deletion.

What happens to file indexing when entity permissions change?

When a STIX object owning files receives an update to its granted_refs (sharing) or object_marking_refs (classification) properties, the stream processor immediately detects the change through the handleStreamEvents function. It then invokes elUpdateFilesWithEntityRestrictions to update the indexed documents’ access control metadata in Elasticsearch, ensuring search results respect the latest security posture without requiring a full re-index.

Which Elasticsearch operations does the manager perform?

The manager relies on three primary operations from src/database/file-search.ts: elIndexFiles for bulk insertion of new file documents, elUpdateFilesWithEntityRestrictions for patching access control metadata on existing entries, and elDeleteAllFiles for administrative index resets. These functions handle the translation between OpenCTI’s internal file identifiers (generated via generateFileIndexId) and the search cluster’s document structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}