# How to Troubleshoot Connection Issues with the Hindsight Server

> Troubleshoot Hindsight server connection issues by checking server logs, testing the health endpoint for database connectivity, and validating your API database URL and port configuration. Resolve problems fast.

- Repository: [vectorize-io/hindsight](https://github.com/vectorize-io/hindsight)
- Tags: how-to-guide
- Published: 2026-03-13

---

**To troubleshoot Hindsight server connection issues, verify the server process started successfully by checking for the "Hindsight server started" log message, test the `/health` endpoint for database connectivity, and validate your `HINDSIGHT_API_DATABASE_URL` and port configuration.**

Connection problems with the Hindsight server from vectorize-io/hindsight typically originate from startup failures, port binding conflicts, or PostgreSQL connectivity issues. The server runs as a FastAPI-based HTTP API in a background thread managed by the **`Server`** class in [`hindsight-all/hindsight/server.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-all/hindsight/server.py). Understanding how to interpret health checks and environment configuration will help you rapidly pinpoint whether the issue lies in server initialization, database access, or reverse-proxy routing.

## Verifying Server Startup and Port Binding

The first step to troubleshoot connection issues is confirming that the `Server` class successfully initialized its background thread and bound to a network interface.

### Connection Refused or Timeout Errors

If you receive a connection refused error, the server process likely never started or crashed during initialization. In [`hindsight-all/hindsight/server.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-all/hindsight/server.py), the **`Server.start()`** method blocks until the server is reachable, then logs a success message:

```python
logger.info(f"Hindsight server started at {self.url}")

```

This line appears at lines 80-83 in the source code. If you do not see this message in your logs, `Server.start()` may have raised a `RuntimeError` indicating the server failed to start within the timeout period.

### Port Availability and Auto-Selection

Port conflicts occur when another process already occupies the configured port. The server handles port selection in [`hindsight-all/hindsight/server.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-all/hindsight/server.py) at lines 62-64:

```python
self.port = port or _find_free_port()

```

When you initialize `Server()` without specifying a port (or pass `port=None`), the `_find_free_port()` helper automatically selects an available ephemeral port. To verify which port was selected, inspect the `server.port` attribute immediately after calling `server.start()`:

```python
from hindsight import Server

server = Server()  # Auto-selects port

server.start()
print(f"Server bound to port: {server.port}")
print(f"Full URL: {server.url}")

```

If you need to use a specific port, explicitly pass it to avoid conflicts:

```python
server = Server(port=8890)  # Use a known free port

server.start()

```

## Diagnosing Database and Health Check Failures

Once the server responds to HTTP requests, the next layer to troubleshoot is database connectivity, which is exposed through the health endpoint.

### Interpreting 503 Errors from the Health Endpoint

The `/health` endpoint is defined in [`hindsight-api-slim/hindsight_api/api/http.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/hindsight_api/api/http.py) (lines 55-66). It returns HTTP 200 when healthy or HTTP 503 when the internal **`MemoryEngine`** cannot communicate with PostgreSQL:

```bash
curl http://127.0.0.1:<port>/health

```

- **HTTP 200**: Returns `{ "status": "healthy", "database": "connected" }`
- **HTTP 503**: Returns JSON with `"status": "unhealthy"` and an `"error"` field describing the failure

The endpoint delegates its check to `app.state.memory.health_check()`, which validates that the database connection pool is initialized and responsive.

### Database Connection Verification

The health check implementation resides in [`hindsight-api-slim/hindsight_api/engine/memory_engine.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/hindsight_api/engine/memory_engine.py) at lines 92-101. The **`MemoryEngine.health_check()`** method executes a `SELECT 1` query against the connection pool:

```python

# Conceptual flow from memory_engine.py lines 92-101

async def health_check(self):
    try:
        await self.pool.execute("SELECT 1")
        return {"status": "healthy", "database": "connected"}
    except Exception as e:
        return {"status": "unhealthy", "database": "error", "error": str(e)}

```

If you see a 503 response with a database error, verify your `HINDSIGHT_API_DATABASE_URL` environment variable. If you are using the embedded **pg0** PostgreSQL instance (the default), ensure the binary is present and the process has permission to create temporary data directories, typically under `/tmp/pg0`.

## Configuration and Environment Verification

Misconfigured environment variables are a common source of connectivity issues. The server reads configuration from [`hindsight-api-slim/hindsight_api/config.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/hindsight_api/config.py) via the **`get_config()`** function.

### Critical Environment Variables

Check these variables against your deployment environment:

- **`HINDSIGHT_API_HOST`**: Interface to bind (default: `127.0.0.1`)
- **`HINDSIGHT_API_PORT`**: Explicit port number (overrides auto-selection)
- **`HINDSIGHT_API_BASE_PATH`**: URL prefix for all routes (e.g., `/hindsight`)
- **`HINDSIGHT_API_DATABASE_URL`**: PostgreSQL connection string
- **`HINDSIGHT_API_LOG_LEVEL`**: Set to `debug` for verbose output

You can inspect the active configuration programmatically:

```python
from hindsight_api.config import get_config
print(get_config())

```

The `.env.example` file in the repository root enumerates all available variables and their default values, serving as a reference for proper configuration.

## Reverse Proxy and Base Path Issues

When deploying behind Nginx, Traefik, or similar proxies, base path mismatches cause 404 errors that appear as connection failures.

The FastAPI application respects `app.root_path`, which is set from `config.base_path` in [`hindsight-api-slim/hindsight_api/api/http.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/hindsight_api/api/http.py). When you configure a custom base path, the health endpoint moves accordingly. As noted in the test suite at [`hindsight-api-slim/tests/test_base_path.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/tests/test_base_path.py) (lines 55-59), a base path of `/hindsight` means the health check is accessible at `/hindsight/health`, not `/health`.

Verify your proxy configuration matches `HINDSIGHT_API_BASE_PATH`:

```nginx

# Nginx example

location /hindsight/ {
    proxy_pass http://127.0.0.1:8000/hindsight/;
}

```

## Step-by-Step Troubleshooting Workflow

Follow this systematic approach to isolate connection issues:

1. **Start the server and monitor logs**
   
   ```python
   from hindsight import Server
   import logging
   
   logging.basicConfig(level=logging.DEBUG)
   server = Server()
   server.start()  # Blocks until ready

   ```

   Look for the "Hindsight server started" confirmation message.

2. **Test connectivity with curl**
   
   ```bash
   curl -v http://127.0.0.1:<port>/health
   ```

   
   Replace `<port>` with the value from `server.port`.

3. **Analyze health check response**
   
   If you receive HTTP 503, examine the JSON error field to determine if the issue is database authentication, network unreachable, or pool exhaustion.

4. **Verify port availability**
   
   ```bash
   ss -ltnp | grep <port>
   ```

   
   If another process holds the port, restart Hindsight with a different port number.

5. **Enable debug logging**
   
   ```bash
   export HINDSIGHT_API_LOG_LEVEL=debug
   python your_server_script.py
   ```

   This reveals detailed startup sequences, including PostgreSQL initialization and LLM client creation.

6. **Run integration tests**
   
   ```bash
   pytest hindsight-api-slim/tests/test_http_api_integration.py
   ```

   
   These tests spin up an in-process server and validate the health endpoint, providing a clean-room verification of your environment.

## Summary

- **Server startup failures** are indicated by the absence of the "Hindsight server started" log message in [`hindsight-all/hindsight/server.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-all/hindsight/server.py); check for exceptions during `Server.start()`.
- **Port conflicts** can be avoided by using the auto-selection feature (`port=None`) or explicitly specifying a free port; verify with `server.port` after initialization.
- **Database issues** manifest as HTTP 503 responses from `/health` with detailed error messages from `MemoryEngine.health_check()` in [`hindsight-api-slim/hindsight_api/engine/memory_engine.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/hindsight_api/engine/memory_engine.py).
- **Environment configuration** is centralized in [`hindsight-api-slim/hindsight_api/config.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/hindsight_api/config.py); validate variables like `HINDSIGHT_API_DATABASE_URL` and `HINDSIGHT_API_BASE_PATH` against your deployment topology.
- **Reverse proxy setups** require matching `HINDSIGHT_API_BASE_PATH` with the proxy's forwarding rules to ensure the `/health` endpoint is accessible at the correct path.

## Frequently Asked Questions

### Why does my Hindsight server return a 503 error on the health endpoint?

A 503 status indicates the `MemoryEngine` cannot communicate with the PostgreSQL database. According to the implementation in [`hindsight-api-slim/hindsight_api/engine/memory_engine.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/hindsight_api/engine/memory_engine.py), the health check runs `SELECT 1` against the connection pool; if this query fails due to missing `HINDSIGHT_API_DATABASE_URL`, network issues, or authentication errors, the endpoint returns `{"status": "unhealthy", "database": "error", "error": "<message>"}`. Check your database URL and ensure PostgreSQL is reachable.

### How do I check if the Hindsight server is using the correct port?

After calling `server.start()`, inspect the `server.port` attribute. In [`hindsight-all/hindsight/server.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-all/hindsight/server.py), the `Server` class sets `self.port = port or _find_free_port()` during initialization. If you passed `port=None`, the system selected an available ephemeral port automatically. The complete URL is available via `server.url`, which is also logged as "Hindsight server started at {self.url}".

### What should I do if the embedded PostgreSQL (pg0) fails to start?

First, ensure the pg0 binary is present in your environment and executable. The embedded database requires permission to create temporary data directories, typically under `/tmp/pg0`. Set `HINDSIGHT_API_LOG_LEVEL=debug` to see detailed startup logs from the pg0 initialization sequence. If problems persist, switch to an external PostgreSQL instance by setting `HINDSIGHT_API_DATABASE_URL` to a valid connection string instead of relying on the embedded default.

### How do I configure Hindsight to work behind a reverse proxy like Nginx?

Set the `HINDSIGHT_API_BASE_PATH` environment variable to match the path prefix configured in your proxy. For example, if Nginx forwards requests from `/hindsight/` to the Hindsight server, set `HINDSIGHT_API_BASE_PATH=/hindsight`. The FastAPI app assigns this to `app.root_path`, ensuring that internal routing and the `/health` endpoint respond correctly at `/hindsight/health` rather than the root path. Verify this configuration using the test patterns shown in [`hindsight-api-slim/tests/test_base_path.py`](https://github.com/vectorize-io/hindsight/blob/main/hindsight-api-slim/tests/test_base_path.py).