How to Configure Watched Folders and Document Synchronization in AnythingLLM

Watched folders and document synchronization in AnythingLLM rely on the experimental "live file sync" feature, which uses a background worker to periodically refresh watched documents from external sources like YouTube, links, or Confluence, automatically updating vector embeddings when content changes.

The live file sync subsystem enables automatic re-indexing of documents when their external sources change. According to the Mintplex-Labs/anything-llm source code, this feature operates through a queue-based architecture that monitors documents per workspace and triggers background synchronization jobs.

Enable the Live File Sync Feature Flag

Before watching documents, you must activate the experimental feature globally. The system stores this state in the experimental_live_file_sync setting within server/models/documentSyncQueue.js.

// server/models/documentSyncQueue.js
enabled: async function () {
  return (
    (await SystemSettings.get({ label: this.featureKey }))?.value === "enabled"
  );
},

You can enable live file sync via the UI at Settings → Experimental, or programmatically through the API endpoint defined in server/endpoints/experimental/liveSync.js:

curl -X POST http://localhost:3000/experimental/toggle-live-sync \
  -H "Content-Type: application/json" \
  -d '{"updatedStatus": true}'

Setting updatedStatus to true enables the feature and boots the background workers via DocumentSyncQueue.bootWorkers(). Disabling it stops all synchronization activity.

Mark Documents as Watched

Once live sync is enabled, you designate specific documents for monitoring. AnythingLLM tracks watched status per workspace using the watched column in the workspace_documents table.

Toggle Watch Status via API

The primary endpoint for managing watched documents is:

POST /workspace/:slug/update-watch-status
{
  "docPath": "folder/my-doc.json",
  "watchStatus": true
}

This endpoint invokes DocumentSyncQueue.toggleWatchStatus() in server/models/documentSyncQueue.js, which either creates a queue entry via watch() or removes it via unwatch():

// server/models/documentSyncQueue.js
watch: async function (document) { 
  // Creates queue entry and sets watched = true 
},
unwatch: async function (document) { 
  // Removes queue entry and sets watched = false 
}

Frontend Implementation

The frontend consumes this API through the LiveDocumentSync model located at frontend/src/models/experimental/liveSync.js:

import LiveDocumentSync from '@/models/experimental/liveSync';

async function toggleWatch(workspaceSlug, docPath, shouldWatch) {
  await LiveDocumentSync.setWatchStatusForDocument(
    workspaceSlug,
    docPath,
    shouldWatch
  );
}

Background Synchronization Process

The sync worker runs as a BackgroundService and executes server/jobs/sync-watched-documents.js periodically. This process handles the actual document synchronization logic.

How the Worker Processes Documents

1. Select stale queues: Queries DocumentSyncQueue.staleDocumentQueues() to find documents due for refresh
2. Resolve source type: Calls Document.parseDocumentTypeAndSource() to identify whether the source is a link, youtube, confluence, or other supported type
3. Fetch fresh content: Uses the Collector API with source-specific payloads to retrieve updated content
4. Compare and update: If content changed, the worker deletes old vectors, re-embeds the new content via the vector DB driver, updates the source JSON file using updateSourceDocument(), and propagates changes to other workspaces sharing the same filename

The worker also implements failure protection through maxRepeatFailures, automatically un-watching documents that exceed the retry threshold.

Verify Watch Status in the UI

When browsing folders in the file manager, the server annotates each item with its watch status. The viewLocalFiles() function in server/utils/files/index.js enriches directory listings:

// server/utils/files/index.js
const watchedDocumentsFilenames = await getWatchedDocumentFilenames(filenames);
for (const item of subdocs.items) {
  item.watched = watchedDocumentsFilenames.hasOwnProperty(item.name) || false;
}

This allows the frontend to render the "eye" icon for watched documents. A sample API response from GET /files includes:

{
  "name": "custom-documents",
  "type": "folder",
  "items": [
    {
      "name": "my-doc.json",
      "type": "file",
      "watched": true,
      "pinnedWorkspaces": []
    }
  ]
}

Implementation Code Examples

Programmatically Watch Documents via Node.js

For server-side scripts or custom integrations, interact directly with the queue model:

const { DocumentSyncQueue } = require('./server/models/documentSyncQueue');
const { Document } = require('./server/models/documents');

async function watchDocument(workspaceId, docPath) {
  const doc = await Document.get({ workspaceId, docpath: docPath });
  if (!doc) throw new Error('Document not found');
  await DocumentSyncQueue.watch(doc);
}

async function unwatchDocument(workspaceId, docPath) {
  const doc = await Document.get({ workspaceId, docpath: docPath });
  if (!doc) throw new Error('Document not found');
  await DocumentSyncQueue.unwatch(doc);
}

Check Live Sync Status

To verify whether the feature is enabled before performing watch operations:

const { DocumentSyncQueue } = require('./server/models/documentSyncQueue');

async function checkSyncStatus() {
  const isEnabled = await DocumentSyncQueue.enabled();
  console.log(`Live file sync is ${isEnabled ? 'enabled' : 'disabled'}`);
  return isEnabled;
}

Summary

• Enable globally: Set the experimental_live_file_sync system setting to "enabled" via the UI or POST /experimental/toggle-live-sync endpoint to activate the background worker.
• Watch per document: Use POST /workspace/:slug/update-watch-status or the DocumentSyncQueue.toggleWatchStatus() method to mark specific documents for monitoring.
• Automatic synchronization: The sync-watched-documents.js job periodically refreshes content from external sources and updates vector embeddings automatically.
• UI integration: The viewLocalFiles() utility in server/utils/files/index.js adds a watched boolean to file listings, enabling visual indicators in the frontend.

Frequently Asked Questions

What document sources support live synchronization?

The synchronization worker in server/jobs/sync-watched-documents.js supports any source type that implements the Collector API interface, including web links, YouTube videos, Confluence pages, and GitHub repositories. The system uses Document.parseDocumentTypeAndSource() to determine the appropriate fetch strategy for each document type.

How often does the background worker check for updates?

The synchronization frequency depends on the BackgroundService configuration in server/utils/BackgroundWorkers/index.js. The worker runs continuously while live sync is enabled, processing stale queues based on internal scheduling logic. Documents are selected for refresh when DocumentSyncQueue.staleDocumentQueues() determines they are due for an update based on their last synchronization timestamp.

Can I watch the same document across multiple workspaces?

Yes. When the background worker updates a watched document, it propagates changes to every workspace that contains the same filename. This occurs in the synchronization logic where updateSourceDocument() updates the source file, and the system ensures all workspace embeddings reflect the new content.

What happens if a watched source becomes unavailable?

The system implements failure tolerance through the maxRepeatFailures counter in the synchronization worker. If a document fails to sync repeatedly, the system automatically un-watches the document via DocumentSyncQueue.unwatch() to prevent infinite retry loops and conserve resources.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}