# How to Use Custom Encoders with Stemmers and Filters for Language-Specific Search in FlexSearch

> Learn to implement language specific search in FlexSearch using custom encoders with stemmers and filters. Customize your text processing for optimal results.

- Repository: [Nextapps GmbH/flexsearch](https://github.com/nextapps-de/flexsearch)
- Tags: how-to-guide
- Published: 2026-02-23

---

**FlexSearch processes text through an `Encoder` class that applies a configurable pipeline of stemmers, filters, mappers, and replacers, enabling language-specific search by passing custom `EncoderOptions` to `new Encoder()` or extending built-in presets from `flexsearch/lang/*`.**

FlexSearch is a high-performance full-text search library that handles tokenization, normalization, and fuzzy matching through its `Encoder` class. To implement language-specific search with custom text processing, you configure the encoder pipeline with **stemmers** for word reduction, **filters** for stop-word removal, and **character mappers** for normalization. This guide demonstrates how to build custom encoders using the FlexSearch source code architecture found in `nextapps-de/flexsearch`.

## Understanding the FlexSearch Encoder Pipeline

The `Encoder` class defined in [`src/encoder.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/encoder.js) merges every supplied option into an internal processing pipeline. When an `Index` is instantiated, FlexSearch builds an encoder instance from the `options.encoder` (or `options.encode`) field in [`src/index.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/index.js) (lines 76-89):

```javascript
const encoder = SUPPORT_CHARSET && is_string(options.encoder)
    ? Charset[options.encoder]                     // built‑in charset preset
    : options.encode || options.encoder || (
        SUPPORT_ENCODER ? {} : fallback_encoder
      );
this.encoder = encoder.encode
    ? encoder
    : typeof encoder === "object"
        ? new Encoder(/** @type {EncoderOptions} */ (encoder))
        : { encode: encoder };

```

The encoder pipeline processes text through sequential steps: `normalize` → `prepare` → `split` → `mapper` → `matcher` → `stemmer` → `filter` → `finalize`. Each step corresponds to an option in `EncoderOptions` defined in [`src/type.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/type.js) (lines 14-38):

| Step | Option | Purpose |
|------|--------|---------|
| Stemming | `stemmer` | `Map<string,string>` that replaces term endings (e.g., `ly → ''`). |
| Filtering | `filter` | `Set<string>` or `function` that removes stop-words or applies custom logic. |
| Character Mapping | `mapper` | `Map<string,string>` that maps single characters (e.g., `é → e`). |
| String Matching | `matcher` | `Map<string,string>` that replaces whole strings before stemming. |
| Regex Replacement | `replacer` | `Array<[RegExp, string]>` for pattern-based replacements after mapping. |
| Preparation | `prepare` | `function(str)` for custom pre-split processing (e.g., HTML entity replacement). |
| Finalization | `finalize` | `function(array)` for post-split processing (e.g., dropping short terms). |

## Creating a Basic Custom Encoder with Stemmers and Filters

To implement language-specific processing, instantiate `Encoder` with a configuration object containing your custom stemmers and filters. The `Encoder.prototype.assign` method in [`src/encoder.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/encoder.js) (lines 93-124) merges these options into the pipeline.

```javascript
import FlexSearch from "flexsearch";
import { Encoder } from "flexsearch";

// Create a custom encoder with stemming and stop-word filtering
const myEncoder = new Encoder({
  // Replace English adverb suffixes
  stemmer: new Map([["ly", ""]]),

  // Filter out common stop-words
  filter: new Set(["and", "the", "of"]),

  // Optional pre-process step (e.g., replace ampersand)
  prepare: str => str.replace(/&/g, " and "),

  // Optional post-process step (drop very short terms)
  finalize: terms => terms.filter(t => t.length > 2)
});

// Attach encoder to an index
const idx = new FlexSearch.Index({
  encoder: myEncoder,
  tokenize: "strict",   // Required for context search
  resolution: 9
});

```

The `addStemmer` and `addFilter` helper methods (defined in [`src/encoder.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/encoder.js), lines 74-88 and 83-92) allow you to extend an existing encoder instance without reconstructing the configuration object.

## Extending Built-in Language Presets

FlexSearch provides language-specific presets in [`src/lang/en.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/lang/en.js) (and similar files for German, French, etc.) that export pre-filled `EncoderOptions` objects. You can extend these presets by passing them to the `Encoder` constructor along with override options. The constructor merges multiple option objects sequentially (see [`src/encoder.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/encoder.js), lines 80-84).

```javascript
import FlexSearch from "flexsearch";
import { Encoder } from "flexsearch";
import EnglishPreset from "flexsearch/lang/en";

// Start from the English preset and modify specific options
const encoder = new Encoder(
  EnglishPreset,               // Built-in stemmer, filter, mapper, etc.
  { filter: false }            // Disable the preset's default stop-word filter
);

// Add domain-specific terms to the filter
encoder.addFilter("example");

// Create the index with the customized encoder
const idx = new FlexSearch.Index({
  encoder,
  tokenize: "strict"
});

```

This approach ensures you inherit the optimized stemming rules and character mappings from the language preset while customizing the filtering logic for your specific domain.

## Building CJK-Aware Encoders with Character Mapping

For languages that require character-level processing, such as Chinese, Japanese, and Korean (CJK), FlexSearch provides charset presets in [`src/charset.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/charset.js). You can combine these with custom mappers and replacers to handle full-width punctuation and ideographic spaces.

```javascript
import FlexSearch from "flexsearch";
import { Encoder, Charset } from "flexsearch";

// Start from the CJK charset (character-by-character split)
const encoder = new Encoder(Charset.CJK)
  .addMapper("，", ",")            // Map full-width comma to ASCII comma
  .addReplacer(/[\u3000]/g, " ")  // Replace IDEOGRAPHIC SPACE with ordinary space
  .addFilter(term => term.length > 1); // Keep only multi-character terms

const idx = new FlexSearch.Index({
  encoder,
  tokenize: "strict"
});

```

The `addMapper` and `addReplacer` methods (implemented in [`src/encoder.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/encoder.js), lines 83-106 and 115-124) allow you to inject character-level transformations into the pipeline after the initial split but before stemming and filtering.

## How the Encoder Integrates with Index Operations

The encoder is attached to the index instance during construction in [`src/index.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/index.js) (lines 76-89). All subsequent search and add operations internally call `this.encoder.encode(...)` to ensure consistent text processing. This synchronization is essential for language-aware search, as it guarantees that query terms undergo the same stemming, filtering, and normalization as indexed content.

According to the source code in [`src/index/search.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/index/search.js) (line 85) and [`src/index/add.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/index/add.js) (line 40), the encoder's `encode` method is invoked immediately before indexing or matching, ensuring the pipeline is applied uniformly across both operations.

## Summary

- **FlexSearch encoders** process text through a configurable pipeline defined in [`src/encoder.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/encoder.js), handling normalization, stemming, filtering, and finalization.
- **Custom encoders** are created by instantiating `new Encoder()` with an `EncoderOptions` object containing `stemmer` (Map), `filter` (Set/function), `mapper` (Map), and other pipeline steps.
- **Language presets** in [`src/lang/en.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/lang/en.js) (and similar) provide pre-configured encoders for specific languages that you can extend or override.
- **Charset presets** like `Charset.CJK` in [`src/charset.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/charset.js) enable character-level processing for CJK languages, combinable with custom mappers and replacers.
- **Integration** occurs in [`src/index.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/index.js), where the encoder is attached to the index and automatically invoked during `add()` and `search()` operations to ensure consistent language processing.

## Frequently Asked Questions

### How do I disable the default stop-word filter in a language preset?

Pass `{ filter: false }` as a second argument to the `Encoder` constructor when extending a preset. According to [`src/encoder.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/encoder.js) (lines 80-84), the constructor merges multiple option objects sequentially, allowing you to override specific properties while inheriting the preset's stemmer and mapper configurations.

### Can I use a function instead of a Set for the filter option?

Yes, the `filter` option accepts either a `Set<string>` for exact string matching or a `function(term: string): boolean` for custom logic. When a function is provided, the encoder invokes it for each term during the filter stage of the pipeline defined in [`src/encoder.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/encoder.js).

### What is the difference between mapper and matcher in FlexSearch encoders?

The `mapper` option processes single characters (e.g., mapping `é` to `e`), while the `matcher` replaces whole strings before stemming occurs (e.g., mapping "New York" to "newyork"). According to [`src/type.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/type.js), both accept `Map<string,string>` but operate at different pipeline stages.

### How do I ensure my custom encoder uses the same processing for indexing and searching?

Attach your encoder instance to the `Index` constructor via the `encoder` option. As implemented in [`src/index.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/index.js) (lines 76-89), the index stores the encoder reference and automatically invokes `this.encoder.encode()` during both `add()` operations ([`src/index/add.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/index/add.js), line 40) and `search()` operations ([`src/index/search.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/index/search.js), line 85), ensuring identical text processing for both stages.