# How to Use a Custom Tokenizer to Split Comma-Separated Property Values in Owner

> Learn how to use a custom tokenizer in OWNER to split comma separated property values. Override the default tokenizer with the TokenizerClass annotation for flexible parsing.

- Repository: [Matteo Baccan/owner](https://github.com/matteobaccan/owner)
- Tags: how-to-guide
- Published: 2026-03-07

---

**Use the `@TokenizerClass` annotation to specify a custom `Tokenizer` implementation at either the method or interface level, overriding the default `SplitAndTrimTokenizer` that splits on commas.**

The Owner library simplifies Java configuration management by mapping properties to type-safe interfaces. When converting a single property value into an array or collection, Owner must split the string. While the default tokenizer handles comma-separated values, you can inject custom logic by implementing the `Tokenizer` interface and declaring it via annotations.

## Understanding the Tokenizer Interface and Default Behavior

Owner delegates string splitting to implementations of the `Tokenizer` interface, which defines a single method:

```java
public interface Tokenizer {
    String[] tokens(String values);
}

```

By default, Owner uses `SplitAndTrimTokenizer` (located in [`org.aeonbits.owner.SplitAndTrimTokenizer.java`](https://github.com/matteobaccan/owner/blob/main/org.aeonbits.owner.SplitAndTrimTokenizer.java)), which splits input on commas and trims whitespace from each token. This satisfies most use cases where properties follow the pattern `value1, value2, value3`.

## Creating a Custom Tokenizer

To implement custom splitting logic—such as preserving empty tokens, splitting on multiple delimiters, or handling escaped characters—create a class that implements `Tokenizer`.

The following example from the Owner test suite demonstrates a tokenizer that splits on commas while preserving trailing empty strings:

```java
package org.aeonbits.owner.typeconversion.arrays;

import org.aeonbits.owner.Tokenizer;

public class CustomCommaTokenizer implements Tokenizer {
    @Override
    public String[] tokens(String values) {
        // The -1 limit preserves trailing empty tokens (e.g., "a,b," becomes ["a", "b", ""])
        return values.split(",", -1);
    }
}

```

*Source:* [`owner/src/test/java/org/aeonbits/owner/typeconversion/arrays/CustomCommaTokenizer.java`](https://github.com/matteobaccan/owner/blob/main/owner/src/test/java/org/aeonbits/owner/typeconversion/arrays/CustomCommaTokenizer.java)

## Applying Custom Tokenizers with @TokenizerClass

Owner provides the `@TokenizerClass` annotation to specify which tokenizer to use. You can apply this annotation at two scopes, with method-level declarations taking precedence over interface-level declarations.

### Method-Level Tokenizer Configuration

Annotate individual methods to use a specific tokenizer for that property only. This overrides any class-level tokenizer or `@Separator` annotation.

```java
import org.aeonbits.owner.Config;
import org.aeonbits.owner.Config.TokenizerClass;
import org.aeonbits.owner.ConfigFactory;

public interface ServerConfig extends Config {

    @TokenizerClass(CustomCommaTokenizer.class)
    @DefaultValue("8080,8081,8082")
    int[] ports();
}

// Usage
ServerConfig cfg = ConfigFactory.create(ServerConfig.class);
int[] ports = cfg.ports();  // Returns [8080, 8081, 8082]

```

### Class-Level Tokenizer Configuration

Apply `@TokenizerClass` to the interface declaration to set a default tokenizer for all methods that do not specify their own.

```java
@TokenizerClass(CustomCommaTokenizer.class)
public interface AppConfig extends Config {

    @DefaultValue("one,two,three")
    String[] names();  // Uses CustomCommaTokenizer

    @Separator(";")   // Overrides class-level tokenizer for this method
    @DefaultValue("a; b; c")
    String[] semicolons();  // Uses built-in SplitAndTrimTokenizer with semicolon
}

```

### Handling Conflicts with @Separator

Owner enforces mutual exclusivity between `@TokenizerClass` and `@Separator`. If both annotations appear on the same method or class, the library throws an `UnsupportedOperationException` during configuration creation.

This validation occurs in [`Config.java`](https://github.com/matteobaccan/owner/blob/main/Config.java) at lines 320-343, where the annotation processor checks for the presence of both annotations and raises the exception if they conflict.

## How Tokenizer Resolution Works

Owner resolves the appropriate tokenizer at runtime through `TokenizerResolver.resolveTokenizer(Method)` (located in [`org.aeonbits.owner.TokenizerResolver.java`](https://github.com/matteobaccan/owner/blob/main/org.aeonbits.owner.TokenizerResolver.java)). The resolution follows this precedence:

1. **Method-level `@TokenizerClass`** — Highest priority
2. **Method-level `@Separator`** — Uses built-in `SplitAndTrimTokenizer` with custom delimiter
3. **Class-level `@TokenizerClass`** — Default for all methods in the interface
4. **Default `SplitAndTrimTokenizer`** — Fallback when no annotations are present

## Complete Working Example

The following example demonstrates class-level tokenizer configuration with method-level overrides, based on the test case [`TokenizerAnnotationOnClassLevelTest.java`](https://github.com/matteobaccan/owner/blob/main/TokenizerAnnotationOnClassLevelTest.java):

```java
// Custom tokenizer that splits on dashes
public class CustomDashTokenizer implements Tokenizer {
    @Override
    public String[] tokens(String values) {
        return values.split("-");
    }
}

// Interface configuration
@TokenizerClass(CustomDashTokenizer.class)
public interface MixedConfig extends Config {

    @TokenizerClass(CustomCommaTokenizer.class)  // Override: use commas
    @DefaultValue("10,20,30")
    int[] commaSeparated();

    @Separator(";")  // Override: use semicolons with default tokenizer
    @DefaultValue("1;2;3")
    int[] semicolonSeparated();

    @DefaultValue("7-8-9")  // Uses class-level dash tokenizer
    int[] dashSeparated();
}

// Usage
MixedConfig cfg = ConfigFactory.create(MixedConfig.class);
System.out.println(Arrays.toString(cfg.commaSeparated()));      // [10, 20, 30]
System.out.println(Arrays.toString(cfg.semicolonSeparated())); // [1, 2, 3]
System.out.println(Arrays.toString(cfg.dashSeparated()));      // [7, 8, 9]

```

## Summary

- Implement the `Tokenizer` interface to define custom splitting logic for property values.
- Apply custom tokenizers using the `@TokenizerClass` annotation at either the method or interface level.
- Method-level `@TokenizerClass` declarations override class-level settings.
- Do not combine `@TokenizerClass` with `@Separator` on the same element; this triggers an `UnsupportedOperationException` in [`Config.java`](https://github.com/matteobaccan/owner/blob/main/Config.java).
- The default `SplitAndTrimTokenizer` handles comma-separated values when no custom tokenizer is specified.

## Frequently Asked Questions

### How do I split property values on a delimiter other than commas?

Create a class implementing the `Tokenizer` interface and override the `tokens(String values)` method to use your preferred delimiter. For example, split on semicolons using `values.split(";")`. Then annotate your config method or interface with `@TokenizerClass(YourTokenizer.class)`.

### Can I use @TokenizerClass and @Separator together?

No. Owner explicitly forbids combining these annotations on the same method or interface. If both are present, the library throws an `UnsupportedOperationException` during configuration initialization, as validated in [`Config.java`](https://github.com/matteobaccan/owner/blob/main/Config.java) lines 320-343.

### What happens if I don't specify a custom tokenizer?

Owner defaults to `SplitAndTrimTokenizer`, which splits input strings on commas and trims whitespace from each resulting token. This behavior applies to all array or collection return types when no `@TokenizerClass` or `@Separator` annotation is present.

### How does Owner choose which tokenizer to use?

The resolution order implemented in `TokenizerResolver.resolveTokenizer(Method)` is: method-level `@TokenizerClass` → method-level `@Separator` → class-level `@TokenizerClass` → default `SplitAndTrimTokenizer`. Method-level annotations always take precedence over interface-level settings.