How to Use `--pre` and `--preproc` Flags for Custom Input Filtering in ripgrep

Use the --pre <COMMAND> flag to execute an external preprocessor on files before searching, passing the file path as the first argument and raw content via stdin, while the legacy --preproc alias remains functional but undocumented.

The BurntSushi/ripgrep repository enables powerful text extraction from binary or complex formats through its custom input filtering interface. By intercepting file content before pattern matching, you can search compressed archives, PDFs, and proprietary formats as if they were plain text.

Internal Architecture of the Preprocessor System

The preprocessor configuration resides in the SearchConfig struct defined in crates/core/search.rs (lines 20-22), which stores the command path in the preprocessor field and glob patterns in preprocessor_globs. When parsing arguments, --pre and --pre-glob are defined in crates/core/flags/defs.rs (lines 5488-5500), while crates/core/flags/hiargs.rs wires these values into the search configuration.

Before opening each file, ripgrep invokes the should_preprocess function (lines 284-291) to validate that a preprocessor exists and that the target file matches any specified --pre-glob constraints. Upon validation, search_preprocessor (lines 294-317) manages the external process lifecycle.

Command Execution and Data Flow

The subprocess protocol follows a strict stdin/stdout contract. RiFgrep spawns the preprocessor with the target file path as the first positional argument and pipes the raw file bytes to stdin. The external command must write transformed text to stdout, which ripgrep then searches as if it were the original file content.

When preprocessors fail, search_preprocessor surfaces explicit errors using this format:

preprocessor command failed: '"./preprocess" "file.pdf"': <error>

This immediate feedback simplifies debugging of custom filtering scripts.

Practical Examples

Extracting Text from PDF Files

Convert binary documents to searchable text using standard Unix tools. Create an executable preprocessor script:

#!/bin/sh
exec pdftotext - -

Execute a search against PDF content:

rg --pre ./preprocess 'The Commentz-Walter algorithm' 1995-watson.pdf

Building Conditional Preprocessor Scripts

Handle mixed repositories by filtering specific extensions while passing others through unchanged:

cat > preprocessor <<'EOF'
#!/bin/sh
case "$1" in
  *.pdf)
    if [ -s "$1" ]; then
      exec pdftotext - -
    else
      exec cat
    fi
    ;;
  *)
    exec cat
    ;;
esac
EOF
chmod +x preprocessor

Run the conditional filter across your codebase:

rg --pre ./preprocessor 'fn is_empty' -c

Restricting Preprocessing with --pre-glob

Minimize performance overhead by limiting preprocessor invocation to specific patterns. The should_preprocess function evaluates these globs before spawning external processes:

rg --pre ./preprocessor --pre-glob '*.pdf' 'search term' .

Only files matching *.pdf trigger the external script; all other files proceed directly to the search phase.

Using the Legacy --preproc Syntax

Older releases accepted --preproc as the primary flag. While current documentation favors --pre, the alias persists in crates/core/flags/defs.rs for backward compatibility:

rg --preproc ./preprocess 'pattern' file.pdf

Functionally equivalent to --pre, this syntax may emit deprecation warnings in future releases.

Chaining with Other Flags

Combine preprocessing with standard search modifiers like case-insensitive matching:

rg -i --pre ./preprocess --pre-glob '*.pdf' 'error' .

The preprocessor transforms the PDF content before ripgrep applies the -i flag to the resulting text stream.

Summary

• The --pre flag stores the external command path in SearchConfig.preprocessor within crates/core/search.rs, enabling dynamic content transformation.
• --pre-glob leverages the preprocessor_globs field to restrict filtering to specific file patterns, evaluated by the should_preprocess function.
• The preprocessor receives the file path as the first argument and raw bytes via stdin, and must output searchable text to stdout.
• --preproc serves as a deprecated alias for --pre, maintained in crates/core/flags/defs.rs for legacy script compatibility.
• Execution errors trigger explicit messages from search_preprocessor, providing clear diagnostics for debugging custom input filtering pipelines.

Frequently Asked Questions

What is the difference between --pre and --preproc?

There is no functional difference; --preproc is the deprecated predecessor to --pre. According to the flag definitions in crates/core/flags/defs.rs, both populate the identical internal configuration field. Modern releases document only --pre, though --preproc remains parseable for backward compatibility.

How does ripgrep pass data to the preprocessor?

As implemented in the search_preprocessor function in crates/core/search.rs (lines 294-317), ripgrep spawns the external command with the target file path as the first positional argument and pipes the raw file contents directly to the subprocess's stdin. The preprocessor must write the transformed text to stdout for ripgrep to consume.

Can I use --pre with binary files like PDFs or DOCX?

Yes, converting binary formats to text represents the primary use case for custom input filtering. Your preprocessor script can invoke utilities like pdftotext, antiword, or custom parsers to extract searchable content before ripgrep executes pattern matching against the transformed stream.

How do I limit which files trigger the preprocessing overhead?

Use the --pre-glob flag to define inclusion patterns such as *.pdf or *.docx. The should_preprocess function (lines 284-291) checks these globs before invoking your command, ensuring that only matching files incur the performance cost of external process execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}