# Log Analysis and SIEM Integration Tools for Security Monitoring > Discover recommended log analysis and SIEM integration tools for security monitoring. Explore lightweight command-line utilities for efficient log parsing and filtering. [trimstray/the-book-of-secret-knowledge] - Repository: [Michał Ży/the-book-of-secret-knowledge](https://github.com/trimstray/the-book-of-secret-knowledge) - Tags: tutorial - Published: 2026-02-24 --- **The *trimstray/the-book-of-secret-knowledge* repository recommends four lightweight command-line utilities—angle-grinder, lnav, GoAccess, and ngxtop—that parse and filter log files for immediate ingestion into any SIEM via JSON, CSV, or syslog streams.** The `trimstray/the-book-of-secret-knowledge` repository curates essential **log analysis and SIEM integration tools** for security professionals and system administrators. According to the **Log Analyzers** section in [[`README.md`](https://github.com/trimstray/the-book-of-secret-knowledge/blob/main/README.md)](https://github.com/trimstray/the-book-of-secret-knowledge/blob/master/README.md), these open-source utilities are selected specifically for their ability to process high-volume logs and feed structured security events into monitoring pipelines. ## Recommended Log Analysis Tools The repository lists four primary utilities in its Log Analyzers catalog. Each serves a distinct forensic or operational purpose, from ad-hoc slicing of authentication logs to real-time web-server metrics. ### angle-grinder: Structured Log Slicing **angle-grinder** is a command-line log processor that uses a tiny domain-specific language (DSL) to extract, transform, and format log lines. As documented in the repository, it excels at slicing large auth logs or application traces into structured JSON for SIEM consumption. The tool supports pattern matching with regular expressions and can output aggregated results or per-line JSON objects. Security teams use it to isolate failed authentication attempts or suspicious error bursts before forwarding them to a central collector. ```bash # Extract failed sudo attempts and POST as JSON to SIEM endpoint angle-grinder -i /var/log/auth.log -e ' if $msg =~ /sudo: pam_unix/ && $msg =~ /authentication failure/ { print json({"time": $time, "user": $user, "msg": $msg}) } ' | curl -XPOST -H "Content-Type: application/json" \ -d @- https://YOUR_SIEM_ENDPOINT/api/logs ``` ### lnav: Interactive Log Navigation **lnav** provides a terminal-based user interface that auto-detects log formats and offers live refresh, search, and timestamp jumping. The repository notes its utility for real-time incident response, allowing analysts to filter massive syslog or nginx files without preprocessing. Because lnav can write filtered views to JSON and follow log rotations, it serves as an interactive front-end that can feed sanitized event streams into downstream SIEM agents like Filebeat or Fluent Bit. ```bash # Live tail kernel errors and pipe matching lines to a forwarder lnav -f -i /var/log/syslog -c ':filter-in "kernel:.*error"' \ -c ':write-json - /tmp/lnav.json' && \ cat /tmp/lnav.json | nc YOUR_SIEM_ENDPOINT 514 ``` ### GoAccess: Real-Time Web Analytics **GoAccess** generates live HTML dashboards from web server logs and can run as a CGI, Docker container, or background daemon. According to [`README.md`](https://github.com/trimstray/the-book-of-secret-knowledge/blob/main/README.md), it is ideal for security teams monitoring web attack patterns, bot traffic, or brute-force login attempts against applications. While it produces visual reports, GoAccess also exports parsed data in JSON and CSV formats, enabling scheduled batch uploads to SIEM platforms that accept file-based ingestion. ```bash # Generate nightly HTML report and transfer to SIEM host goaccess -f /var/log/nginx/access.log -a -o /tmp/goaccess.html \ && scp /tmp/goaccess.html user@YOUR_SIEM_HOST:/var/www/reports/ ``` ### ngxtop: Nginx Metrics Streaming **ngxtop** streams nginx access-log statistics directly to the console, reporting request rates, status-code distributions, and top client IPs in real time. The repository highlights it as a lightweight alternative to heavy analytics stacks for nginx-specific monitoring. By outputting metrics as JSON lines, ngxtop integrates easily with time-series databases or direct SIEM HTTP endpoints, allowing security teams to detect DDoS patterns or scanning activity within seconds of occurrence. ```bash # Stream nginx stats as JSON to InfluxDB (commonly used by SIEM dashboards) ngxtop -f -l /var/log/nginx/access.log --json | \ curl -i -XPOST http://INFLUX_HOST/write?db=nginx \ --data-binary @- ``` ## SIEM Integration Patterns The repository emphasizes that these tools are **front-end processors** rather than full SIEM platforms. Effective integration typically follows three patterns: - **Batch Export** – Run tools on a cron schedule, dump JSON or CSV files, and forward them via the SIEM's REST API or file-drop ingestion. - **Streaming Pipeline** – Launch tools in follow mode (`lnav -f`, `ngxtop -f`), pipe stdout to a local log forwarder such as **Filebeat**, **Fluent Bit**, or **syslog-ng**, which already ships to the SIEM. - **Embedded Alerting** – Use `angle-grinder` or `lnav` filters to flag suspicious events (e.g., failed logins, SQL injection patterns) and trigger immediate webhooks or email alerts before the data reaches the central store. ## Key Source Files - **[`README.md`](https://github.com/trimstray/the-book-of-secret-knowledge/blob/main/README.md)** – Contains the **Log Analyzers** section cataloging `angle-grinder`, `lnav`, `GoAccess`, and `ngxtop` as recommended security monitoring tools. This is the primary source file referenced in the repository root. ## Summary - **angle-grinder** provides DSL-based log slicing for structured JSON output ideal for SIEM APIs. - **lnav** offers an interactive terminal UI with live filtering and JSON export capabilities. - **GoAccess** delivers real-time web log visualization with scheduled batch export options. - **ngxtop** streams nginx-specific metrics as JSON for direct ingestion into time-series SIEM backends. - All four tools listed in `trimstray/the-book-of-secret-knowledge` are designed to integrate with commercial or open-source SIEMs via standard syslog, HTTP POST, or file-based ingestion. ## Frequently Asked Questions ### Which log analysis tool is best for real-time nginx monitoring? **ngxtop** is specifically recommended for nginx environments. It parses access logs in real time and outputs request rates, status codes, and client IP statistics as JSON, which can be piped directly to InfluxDB or HTTP endpoints used by SIEM dashboards. ### How do I stream angle-grinder output to a SIEM? Use angle-grinder's JSON output mode combined with a pipe to `curl`. The tool can filter logs with regular expressions and print structured JSON objects, which you then POST to your SIEM's ingestion API using standard shell redirection. ### Can these tools replace a commercial SIEM? No. According to the repository, these are **log analysis front-ends** that preprocess and filter data. They lack long-term storage, correlation engines, and alerting workflows found in commercial SIEMs, but they serve as lightweight collection agents that feed into those platforms. ### What log formats do these tools support? **lnav** auto-detects common formats including syslog, nginx, and Apache logs. **GoAccess** specializes in web logs (nginx, Apache, Amazon S3). **ngxtop** handles nginx access logs specifically. **angle-grinder** is format-agnostic and works with any line-oriented text file using user-defined patterns.