How to Debug the OpenMetadata Backend: A Complete Developer Guide

Set LOG_LEVEL=DEBUG, check /api/v1/system/health, and use the Dropwizard admin health check UI to systematically isolate backend issues in OpenMetadata.

The OpenMetadata backend is a Dropwizard-based Java service that powers the platform's metadata ingestion, search, and governance capabilities. Learning how to debug the OpenMetadata backend effectively saves hours when troubleshooting startup failures, API errors, or performance bottlenecks. This guide walks through the exact source locations, configuration files, and diagnostic techniques used by the core maintainers.

---

Understanding the Backend Architecture

Before diving into debugging techniques, you need to understand where the service starts and how components are wired together.

Entry Point: OpenMetadataApplication

The bootstrap sequence begins in OpenMetadataApplication.java, located at openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplication.java. This class extends Dropwizard's Application and performs three critical functions:

1. Registers bundles — including the cache bundle, search index bundle, and authentication filters
2. Wires health checks — OpenMetadataServerHealthCheck and cache health checks
3. Configures JDBI — database connection pooling and SQL object mapping

Any failure during this startup sequence appears in the logs before the HTTP server begins accepting requests.

Configuration: OpenMetadataApplicationConfig

Runtime behavior is controlled by OpenMetadataApplicationConfig.java at openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplicationConfig.java. This POJO maps directly to your openmetadata.yaml file and includes nested configuration for:

• Database (dataSourceFactory)
• Search engine (elasticSearchConfiguration or openSearchConfiguration)
• Authentication (authenticationConfiguration)
• Cache (cacheConfiguration)

When debugging, verify that the loaded configuration matches your expectations by inspecting these values in a debugger or adding temporary log statements.

---

Debugging Startup Issues

Startup failures are the most common backend problems. Follow this sequence to isolate the root cause.

Step 1: Enable Debug Logging Before Launch

The logging system is configured in logback.xml at openmetadata-service/src/main/resources/logback.xml. The root logger uses variable substitution:

<root level="${LOG_LEVEL:-INFO}">
    <appender-ref ref="FILE"/>
    <appender-ref ref="CONSOLE"/>
</root>

Set the environment variable before starting the service:

export LOG_LEVEL=DEBUG

# or for maximum verbosity

export LOG_LEVEL=TRACE

The logs write to logs/openmetadata-operation.log and the console. Look for lines containing DEBUG org.openmetadata.service.OpenMetadataApplication to confirm the setting took effect.

Step 2: Verify the Service Started Successfully

After the Dropwizard banner, you should see this log line:

INFO  org.openmetadata.service.OpenMetadataApplication – Started OpenMetadataApplication

If this line is missing, the service failed before completing initialization. Common failure points include:

• Database connectivity — Check dataSourceFactory configuration and network access
• Search engine connection — Elasticsearch or OpenSearch must be reachable
• Authentication setup — JWT key configuration errors fail silently in some configurations

Step 3: Confirm Basic Health with the REST Endpoint

The SystemResource class at openmetadata-service/src/main/java/org/openmetadata/service/resources/system/SystemResource.java exposes a lightweight health check:

curl -s http://localhost:8585/api/v1/system/health

Expected response:

OK

HTTP 200 confirms the JVM is alive and the HTTP connector is accepting requests. If this fails but the process is running, examine the Dropwizard admin interface next.

---

Using the Dropwizard Admin Interface

Dropwizard provides a built-in administrative interface on a separate port (default 8585, same as application port in OpenMetadata). This interface exposes critical diagnostic information.

Health Check Endpoint

Visit http://localhost:8585/admin/healthcheck to see all registered health checks:

Check	Source File	Purpose
ServerHealthCheck	OpenMetadataServerHealthCheck.java	Basic JVM liveness
CacheHealthCheck	CacheBundle.java	Redis or in-memory cache connectivity

The OpenMetadataServerHealthCheck at openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataServerHealthCheck.java is intentionally simple — it returns healthy unless the JVM has crashed. More sophisticated checks like CacheHealthCheck in CacheBundle.java (at openmetadata-service/src/main/java/org/openmetadata/service/cache/CacheBundle.java) validate actual connectivity to external services.

Red entries indicate component failures. Use these to isolate whether problems are in the database, cache, or search layer.

Metrics Endpoint

The /admin/metrics endpoint exposes JVM memory, thread states, and custom metrics. Look for:

• JVM heap usage — Memory pressure causes GC pauses and slowdowns
• Database connection pool metrics — openmetadata.db.pool gauges show active and idle connections
• Cache hit/miss rates — cache.* metrics from the cache bundle

Threads and Diagnostics

Use /admin/threads to download a thread dump. This helps identify:

• Deadlocks in database connection pools
• Blocked threads in authentication filters
• Runaway threads in search indexing

---

Debugging Authentication and Security

Authentication failures are common when configuring SSO or API access. The relevant code lives in the security package.

JWT Filter Debugging

The JwtFilter at openmetadata-service/src/main/java/org/openmetadata/service/security/JwtFilter.java validates tokens on every request. Enable debug logging for this package:

export LOGGING_LEVEL_ORG_OPENMETADATA_SERVICE_SECURITY=DEBUG

Or modify logback.xml to add:

<logger name="org.openmetadata.service.security" level="DEBUG"/>

Debug output shows:

• Token extraction from the Authorization header
• Signature validation results
• Claims parsing (user name, email, roles)

Request Filter Chain

The ContainerRequestFilterManager in OpenMetadataApplication registers multiple filters. Check the registration order in OpenMetadataApplication.java — filters execute sequentially, and early failures prevent later filters from running.

---

Debugging Database and JDBI Queries

Database issues manifest as slow queries, connection timeouts, or data inconsistencies.

Enable SQL Logging

For detailed query visibility, add the JVM flag:

java -Dorg.jdbi.v3.Logger=DEBUG \
     -jar openmetadata-service/target/openmetadata-service-*.jar \
     server openmetadata-server.yaml

JDBI utilities are in util/jdbi/ (e.g., JdbiUtils). The debug output includes:

• SQL statement text
• Bound parameter values
• Execution timing

Connection Pool Monitoring

Check openmetadata.yaml for the dataSourceFactory section. The maxSize and minSize parameters control pool sizing. Monitor the /admin/metrics endpoint for openmetadata.db.pool.active and openmetadata.db.pool.idle — sustained high active counts indicate pool exhaustion.

---

Debugging Search Indexing

Search indexing issues cause missing entities or stale search results.

Search Bundle Logs

The search index bundle logs under org.openmetadata.service.apps.bundles.searchIndex. Enable dedicated logging:

export LOGGING_LEVEL_ORG_OPENMETADATA_SERVICE_APPS_BUNDLES_SEARCHINDEX=DEBUG

This logger is defined in logback.xml — search for the logger name to confirm the exact package path.

Reindexing Operations

Forced reindexing can be triggered via API when debugging search inconsistencies. Check SystemResource for reindex endpoints — these delegate to the search bundle for execution.

---

Local Development Debugging

For deepest inspection, run from source with a debugger attached.

Build and Run from Source

# Build the service module

mvn clean install -DskipTests

# Run with remote debugging enabled

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 \
     -jar openmetadata-service/target/openmetadata-service-*.jar \
     server openmetadata-server.yaml

The pom.xml at openmetadata-service/pom.xml defines the build artifact and dependencies.

IDE Configuration

In IntelliJ IDEA or Eclipse:

1. Create a "Remote JVM Debug" configuration
2. Set host to localhost and port to 5005
3. Set breakpoints in key classes:
   • OpenMetadataApplication.java — startup sequence
   • JwtFilter.java — authentication
   • SystemResource.java — health endpoints

Breakpoints in OpenMetadataApplication let you step through bundle registration and catch configuration errors before the server starts.

---

Summary

• Enable debug logging with LOG_LEVEL=DEBUG to inspect startup and runtime behavior in openmetadata-operation.log
• Verify basic health using the /api/v1/system/health endpoint and the Dropwizard admin interface at /admin/healthcheck
• Inspect component health checks including OpenMetadataServerHealthCheck and CacheHealthCheck to isolate database, cache, or search failures
• Enable SQL debugging with -Dorg.jdbi.v3.Logger=DEBUG to trace database queries and parameter binding
• Debug authentication flows by enabling org.openmetadata.service.security logging and inspecting JwtFilter behavior
• Run from source with a remote debugger attached to set breakpoints in OpenMetadataApplication, resource classes, and security filters

---

Frequently Asked Questions

How do I check if the OpenMetadata backend is running correctly?

Send a request to http://localhost:8585/api/v1/system/health using curl or a browser. A 200 OK response with body OK confirms the JVM is alive and the HTTP connector is accepting requests. For deeper verification, visit /admin/healthcheck to see individual component statuses.

Where are the OpenMetadata backend logs stored?

The service writes operational logs to logs/openmetadata-operation.log by default, with console output also enabled. The logging configuration in logback.xml at openmetadata-service/src/main/resources/logback.xml controls appenders, log rotation, and the root level via the LOG_LEVEL environment variable.

How do I enable debug logging for database queries in OpenMetadata?

Pass the JVM system property -Dorg.jdbi.v3.Logger=DEBUG when starting the service, or set the Logback logger org.jdbi to DEBUG in logback.xml. This outputs every SQL statement with bound parameters and execution timing, making it easy to identify slow queries or parameter mismatches.

What is the best way to debug authentication issues in OpenMetadata?

Enable debug logging for the org.openmetadata.service.security package by setting LOGGING_LEVEL_ORG_OPENMETADATA_SERVICE_SECURITY=DEBUG. This activates detailed output from JwtFilter, showing token extraction, signature validation, and claims parsing. For interactive debugging, attach a remote debugger to JwtFilter.doFilter() and step through the validation logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}