How Plausible Handles Bot Filtering at the Database Query Level

Plausible Analytics filters bot traffic during the ingestion pipeline by dropping detected bot events before they reach ClickHouse, eliminating the need for database-level filtering in analytics queries.

The plausible/analytics repository implements a zero-overhead approach to bot management. Instead of storing bot traffic and filtering it out during query execution, Plausible identifies and discards bot-generated events at the edge of its ingestion pipeline. This architectural decision ensures that the ClickHouse database contains only legitimate visitor data, making all downstream analytics queries inherently bot-free without additional WHERE clauses or performance penalties.

Why Bot Filtering Happens Before the Database

Plausible’s ingestion architecture prioritizes data cleanliness over storage completeness. Bot detection occurs immediately after request parsing but before any database write operations begin. This early-exit pattern guarantees that malicious or automated traffic never consumes storage resources or complicates analytical queries.

The Ingestion Pipeline Architecture

When a tracking request arrives, it flows through Plausible.Ingestion.Event.build_and_buffer/2, which executes a processing pipeline defined in pipeline/0. Each step transforms the event struct until it reaches the persistor. The critical observation is that this pipeline short-circuits upon encountering any validation failure, including bot detection.

Early Exit Pattern

The pipeline uses Enum.reduce_while/3 to halt processing immediately when an event is marked as dropped. In lib/plausible/ingestion/event.ex, the put_user_agent/2 function inspects the User-Agent string and calls drop(event, :bot) if it detects automated traffic. Once dropped, the event never reaches the register_session/2 function that handles ClickHouse persistence, effectively sanitizing the database at the ingestion boundary.

Bot Detection Implementation in the Ingestion Pipeline

Bot identification relies on UAInspector, a library that parses User-Agent strings against known bot signatures. The detection logic resides in the ingestion event handler and distinguishes between legitimate browsers, headless clients, and automated crawlers.

User-Agent Parsing with UAInspector

In lib/plausible/ingestion/event.ex (lines 56-64), the put_user_agent/2 function implements the detection logic:

defp put_user_agent(%__MODULE__{} = event, _context) do
  case parse_user_agent(event.request) do
    {:ok, %UAInspector.Result{client: %UAInspector.Result.Client{name: "Headless Chrome"}}} ->
      drop(event, :bot)

    {:ok, %UAInspector.Result.Bot{}} ->
      drop(event, :bot)

    {:ok, %UAInspector.Result{} = user_agent} ->
      update_session_attrs(event, %{user_agent: user_agent})

    _any ->
      event
  end
end

This implementation catches two specific bot categories: explicit bot results (%UAInspector.Result.Bot{}) and headless Chrome instances often used for automated scraping. Normal visitors proceed to session attribute updates, while detected bots trigger the drop mechanism.

The Drop Mechanism: Preventing Bots from Reaching ClickHouse

When bot detection fires, Plausible converts the event into a dropped struct rather than persisting it. This transformation happens in the drop/2 helper function and prevents any subsequent database operations.

How drop/2 Marks Events as Invalid

The drop/2 function (lines 78-86 in lib/plausible/ingestion/event.ex) sets dropped?: true and assigns a specific :bot reason:

defp drop(event, reason) do
  %{
    event
    | dropped?: true,
      drop_reason: reason,
      telemetry: [{:drop, reason} | event.telemetry]
  }
end

This struct modification signals to downstream pipeline stages that the event should not proceed to persistence. The function also emits telemetry data, allowing operators to monitor bot traffic volumes through metrics without storing the actual events.

The Persistor Logic in register_session/2

The final gatekeeper resides in register_session/2 (lines 145-152), which coordinates event persistence. This function checks the dropped? flag before calling the persistor:

defp register_session(event, context) do
  if event.dropped? do
    {:error, event.drop_reason}
  else
    Persistor.persist_event(event, context)
  end
end

Because event.dropped? returns true for bot traffic, the persistor never receives the event, and ClickHouse remains free of bot rows. All analytics queries executed against the events table therefore return only genuine visitor data by default.

Database Query Implications

By preventing bot persistence rather than filtering post-hoc, Plausible eliminates the computational overhead typically associated with bot exclusion in analytical databases.

Zero-Overhead Analytics Queries

Since bots never enter ClickHouse, analytics queries require no additional WHERE predicates to exclude automated traffic. Queries like SELECT count() FROM events WHERE site_id = ? automatically return accurate visitor counts without filtering clauses such as AND user_agent NOT LIKE '%bot%'. This design maintains query simplicity and execution performance, particularly important for high-traffic installations processing millions of events.

Summary

• Early detection: Plausible identifies bots using UAInspector during the ingestion pipeline in lib/plausible/ingestion/event.ex, not during query execution.
• Drop mechanism: The drop/2 function marks events with dropped?: true and drop_reason: :bot, preventing persistence.
• Pipeline short-circuit: Enum.reduce_while/3 halts processing immediately when bots are detected, before register_session/2 executes.
• Clean storage: Because Persistor.persist_event/3 receives only non-dropped events, ClickHouse contains no bot traffic, eliminating the need for database-level filtering.
• Telemetry visibility: Despite dropping events, Plausible emits :plausible, :ingest, :event, :dropped telemetry for monitoring bot volume without storing the data.

Frequently Asked Questions

Does Plausible store bot traffic in a separate table?

No. Plausible does not store bot traffic in any table. When the ingestion pipeline detects a bot via put_user_agent/2, it immediately marks the event as dropped using drop/2. The register_session/2 function then skips persistence entirely, returning {:error, :bot} instead of writing to ClickHouse. This zero-storage approach ensures the database remains free of bot data.

How does Plausible detect headless browsers?

Plausible detects headless browsers through pattern matching in lib/plausible/ingestion/event.ex. Specifically, the code checks for UAInspector.Result.Client{name: "Headless Chrome"} in addition to standard %UAInspector.Result.Bot{} structs. This catches automation tools that use headless Chromium but may not identify explicitly as crawlers in standard User-Agent databases.

Is there any way to include bot traffic in Plausible analytics?

No, there is no configuration option to include bot traffic in the analytics interface. The bot filtering is hardcoded in the ingestion pipeline and operates as a binary gate before storage. Because bot events never reach ClickHouse, they cannot be retrieved through any query interface or API endpoint. Operators wishing to analyze bot traffic must monitor the telemetry events emitted when drop/2 is invoked.

What happens to dropped bot events?

Dropped bot events are discarded in memory without ever touching the database. The drop/2 function sets the dropped? flag and records the reason (:bot), then the pipeline returns an error tuple that bypasses the persistor. While the events are not stored, Plausible does emit telemetry metrics (:plausible, :ingest, :event, :dropped) allowing infrastructure monitoring of bot detection rates without data retention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}