How Sherlock Measures and Tracks Response Time for Each Request

Sherlock measures response time by injecting a custom hook into SherlockFuturesSession that timestamps requests with time.monotonic() and attaches the elapsed duration to each HTTP Response object.

The sherlock-project/sherlock tool performs asynchronous username enumeration across hundreds of platforms. Understanding how Sherlock measures and tracks response time for each request reveals the precision behind its network performance monitoring and exported reporting data.

The Core Mechanism: SherlockFuturesSession

Sherlock implements timing through a specialized session class that intercepts every HTTP request before it enters the thread pool. This approach ensures consistent measurement across all asynchronous operations without modifying individual site queries.

Overriding the request() Method

In sherlock_project/sherlock.py (lines 70‑78), the SherlockFuturesSession class extends requests_futures.FuturesSession and overrides the request() method to inject timing instrumentation:

• Timestamp capture: The method records the current monotonic clock value using start = monotonic() immediately upon invocation.
• Hook definition: An inner function response_time(resp, *args, **kwargs) computes the delta by calculating resp.elapsed = monotonic() - start and attaches this value directly to the Response object.
• Priority insertion: The hook is inserted as the first element in the response hooks list, ensuring it executes before any user‑provided or site‑specific hooks modify the response.

The modified request is then dispatched to a background thread via the parent class's asynchronous machinery, preserving the concurrent.futures interface while adding millisecond‑precision timing.

Capturing and Storing Response Times

Once futures complete, Sherlock extracts the measured duration and propagates it through the result pipeline for display and export.

Retrieving Elapsed Time from Completed Futures

After each future finishes execution, Sherlock calls get_response() to obtain the Response object (lines 61‑66 in sherlock_project/sherlock.py). The code accesses the attached timing via r.elapsed. If the attribute is missing—typically when a request fails or times out—the implementation gracefully sets response_time to None rather than raising an exception.

Persisting Timing Data in QueryResult

The extracted duration is passed to the QueryResult constructor as the query_time parameter (lines 81‑88). This value persists throughout the result lifecycle and ultimately appears in exported reports as the response_time_s column, enabling users to analyze site‑specific latency patterns across their investigations.

Code Implementation Examples

The following example demonstrates creating a timed session and accessing the response duration directly:

from sherlock_project.sherlock import SherlockFuturesSession

# Create a futures session with a limited worker pool

session = SherlockFuturesSession(max_workers=5)

# Issue a request (the hook automatically adds resp.elapsed)

future = session.get('https://example.com/username', timeout=30)

# Wait for completion and retrieve the response

response = future.result()

# The elapsed time in seconds is stored on the response object

print(f"Request took {response.elapsed:.3f} seconds")

To extract timing data from a completed Sherlock run after querying multiple sites:

from sherlock_project.sherlock import sherlock
from sherlock_project.sites import SitesInformation
from sherlock_project.notify import QueryNotifyPrint

# Load site definitions and notification handler

sites = SitesInformation(data_file_path="data.json")
notifier = QueryNotifyPrint()

# Execute queries for a target username

results = sherlock(
    username="targetuser",
    site_data=sites,
    query_notify=notifier,
)

# Access query_time from each QueryResult object

for site_name, data in results.items():
    resp_time = data["status"].query_time
    if resp_time is not None:
        print(f"{site_name}: {resp_time:.3f}s")
    else:
        print(f"{site_name}: No response time available")

Summary

• Sherlock uses Python's time.monotonic() to capture high‑precision start timestamps before each request enters the thread pool.
• A custom response_time hook injected into SherlockFuturesSession.request() computes the delta and stores it as resp.elapsed on the Response object.
• The timing value propagates through QueryResult as query_time and appears in CSV/Excel exports under the response_time_s column header.
• Failed requests gracefully return None for timing data, ensuring robust operation during network enumeration.

Frequently Asked Questions

What timing function does Sherlock use to measure response time?

Sherlock uses time.monotonic() to capture start timestamps. This function is immune to system clock adjustments and provides a steady, high‑resolution counter suitable for measuring short‑duration network operations across different platforms.

How does Sherlock handle failed requests when tracking response times?

When a request fails or times out, the Response object may lack the elapsed attribute. Sherlock's get_response() implementation checks for this condition and assigns None to response_time, allowing the enumeration to continue without crashing while maintaining a consistent data structure in the results.

Where does Sherlock store the response time data for export?

The measured duration is stored in the query_time attribute of the QueryResult object defined in sherlock_project/result.py. During CSV or Excel export generation, this value is serialized into a column named response_time_s, making it available for post‑processing and latency analysis.

Why does Sherlock insert the timing hook as the first response hook?

Sherlock inserts the response_time hook at index 0 of the response hooks list to ensure it executes immediately upon receipt of the HTTP response. This positioning prevents subsequent hooks—which might parse content or handle redirects—from consuming additional time that would skew the raw network latency measurement.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}