How to Adjust Max Concurrent Requests in oMLX's Scheduler: 3 Configuration Methods

oMLX controls simultaneous request processing through the max_concurrent_requests setting stored in SchedulerSettings at omlx/settings.py, which you can override via CLI flag, environment variable, or settings file to directly configure scheduler capacity in omlx/scheduler.py.

The request scheduler in the jundot/omlx repository manages inference workloads by limiting how many sequences can be processed simultaneously. Adjusting the max concurrent requests in oMLX's scheduler is essential for optimizing throughput on different hardware configurations. This setting defaults to 8 and determines both max_num_seqs and completion_batch_size in the underlying scheduler configuration.

Understanding the Scheduler Architecture

SchedulerSettings Definition

In omlx/settings.py, the SchedulerSettings dataclass defines the configuration boundary for the scheduler. The max_concurrent_requests field (line 219) defaults to 8 and serves as the upstream source for all concurrency limits according to the source code.

SchedulerConfig Application

When the engine initializes, settings.to_scheduler_config() converts the value into a SchedulerConfig instance defined in omlx/scheduler.py (lines 362-370). Here, the value populates both max_num_seqs and completion_batch_size (line 1199), making the scheduler's capacity directly proportional to your configured limit.

Configuration Methods

You can adjust max concurrent requests in oMLX's scheduler through three hierarchical overrides following the standard precedence: CLI arguments > environment variables > settings file > defaults.

Method 1: Command-Line Flag

The --max-concurrent-requests flag provides immediate, session-specific overrides without persisting changes to disk.

omlx serve --max-concurrent-requests 64

Method 2: Environment Variable

Set OMLX_MAX_CONCURRENT_REQUESTS to persist the setting across command invocations without modifying configuration files.

export OMLX_MAX_CONCURRENT_REQUESTS=128
omlx serve

Method 3: Settings File

For persistent configuration across restarts, edit the scheduler.max_concurrent_requests key in your JSON or YAML settings file:

{
  "scheduler": {
    "max_concurrent_requests": 256
  }
}

Configuration Propagation and Validation

The init_settings() function processes these overrides in sequence: first parsing CLI arguments, then checking for the environment variable, and finally falling back to the settings file. If you provide an invalid value (≤ 0), validation raises an error and aborts startup before the scheduler initializes.

Internally, the conversion happens in to_scheduler_config():

SchedulerConfig(
    max_num_seqs=settings.scheduler.max_concurrent_requests,
    completion_batch_size=settings.scheduler.max_concurrent_requests,
    # ...

)

Programmatic Access

For testing or custom scripts, you can initialize settings programmatically using argparse.Namespace as implemented in the source:

from argparse import Namespace
from omlx.settings import init_settings, get_settings

# Override via fake CLI namespace

init_settings(cli_args=Namespace(max_concurrent_requests=64))

settings = get_settings()
print("Concurrent requests:", settings.scheduler.max_concurrent_requests)  # → 64

This approach is validated in tests/test_settings.py and tests/test_cli.py, which verify default values, CLI parsing, environment overrides, and input validation.

Summary

• The max concurrent requests setting resides in SchedulerSettings (omlx/settings.py) and defaults to 8
• It propagates to SchedulerConfig in omlx/scheduler.py to control max_num_seqs and completion_batch_size
• Override via --max-concurrent-requests, OMLX_MAX_CONCURRENT_REQUESTS, or the scheduler.max_concurrent_requests key in your settings file
• Values ≤ 0 trigger validation errors during init_settings() and prevent startup
• Programmatic configuration is available via init_settings() with argparse.Namespace

Frequently Asked Questions

What is the default max concurrent requests value in oMLX?

The default value is 8 concurrent requests, defined in the SchedulerSettings dataclass at line 219 of omlx/settings.py. This conservative default works for most consumer GPUs but may need increasing for server-class hardware with larger VRAM capacity.

Where does the max concurrent requests setting get applied in the scheduler?

The setting transfers from SchedulerSettings to SchedulerConfig in omlx/scheduler.py via the to_scheduler_config() method. It populates both max_num_seqs and completion_batch_size at line 1199, directly determining how many sequences the scheduler can process simultaneously during inference.

Can I set max concurrent requests through multiple methods simultaneously?

Yes, oMLX follows a strict configuration hierarchy. CLI flags take precedence over environment variables, which override settings files, which finally fall back to the default value of 8. You can define a base value in your settings JSON and temporarily override it with environment variables or command-line flags for specific sessions.

What happens if I set max concurrent requests to zero or a negative number?

The init_settings() function validates the input and raises a configuration error before the engine starts, as validated in tests/test_settings.py. The scheduler requires at least one concurrent request to function, so invalid values (≤ 0) abort startup immediately with a validation error.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}