CEM Solver Configuration Parameters for World Models: Understanding num_samples, horizon, and receding_horizon

The num_samples, horizon, and receding_horizon configuration parameters in the stable-worldmodel CEM solver respectively control the breadth of action sequence sampling, the number of future steps to predict, and how many steps are executed before triggering a replan.

In model-based reinforcement learning, the Cross-Entropy Method (CEM) generates optimal action sequences by sampling candidates and refining them against a learned world model. In the galilai-group/stable-worldmodel repository, these three parameters defined in stable_worldmodel/solver/cem.py and stable_worldmodel/policy.py govern the trade-off between planning accuracy, computational cost, and control reactivity.

Where CEM Solver Configuration Parameters Are Defined

Understanding the source location of each parameter clarifies how they flow from configuration to execution.

num_samples: Controlling Search Breadth

The num_samples parameter is defined in CEMSolver.__init__ at line 35 of stable_worldmodel/solver/cem.py. It specifies the number of candidate action sequences drawn from the Gaussian distribution during each CEM iteration. Inside the optimization loop (for step in range(self.n_steps)), this value determines the allocation of the candidates tensor with shape (batch, num_samples, horizon, action_dim) (lines 90-99).

horizon: Setting the Planning Depth

The horizon parameter lives in the PlanConfig dataclass at lines 29-30 of stable_worldmodel/policy.py. It represents the total number of future model steps the solver predicts when evaluating action sequences. The policy exposes this to the solver via self.solver.configure(..., config=self.cfg), and the CEM solver accesses it through its horizon property (lines 84-86 of cem.py). This value directly determines the temporal dimension of the mean and variance tensors maintained by the optimizer.

receding_horizon: Managing Replanning Frequency

The receding_horizon parameter is also defined in PlanConfig at lines 30-31 of policy.py. Unlike horizon, this parameter governs execution rather than planning. The WorldModelPolicy converts it to environment steps using flatten_receding_horizon (lines 26-28), accounting for any action repeat (action_block). This calculation determines the size of the internal action buffer (self._action_buffer) initialized in set_env (lines 42-44), which stores planned actions awaiting execution.

How num_samples, horizon, and receding_horizon Interact

These three parameters form a pipeline that balances exploration, foresight, and responsiveness.

Planning Horizon (horizon) dictates how far into the future the world model predicts. The CEM optimizer maintains a mean-variance distribution over an array of shape (batch, horizon, action_dim). When solving, the sampler generates num_samples sequences spanning this full horizon, and the world model evaluates the cumulative cost across all future steps.

Number of Samples (num_samples) determines the granularity of the search. Higher values provide a more accurate estimate of the elite action region but increase memory consumption proportional to batch × num_samples × horizon × action_dim. This is the primary knob for trading computation against optimization quality.

Receding Horizon (receding_horizon) controls replanning frequency. After the policy extracts the first flatten_receding_horizon actions from the solver’s output mean, it deposits them into a deque buffer. The policy consumes these buffered actions for subsequent environment steps, only triggering a fresh call to solver.solve when the buffer empties. If warm_start=True, the previous plan initializes the new optimization.

Action Blocking (action_block) scales both horizon and receding_horizon from model steps to environment steps. As implemented in PlanConfig.plan_len and WorldModelPolicy.flatten_receding_horizon, an action_block of 2 means each model step represents two environment frames, effectively doubling the physical time horizon without increasing the model rollout length.

Practical Configuration Examples

Complete Workflow Setup

import torch
from stable_worldmodel.solver.cem import CEMSolver
from stable_worldmodel.policy import WorldModelPolicy, PlanConfig
from stable_worldmodel.world.world import WorldModel

# Initialize trained world model

wm = WorldModel.load("path/to/checkpoint.pt")

# Configure planning parameters

cfg = PlanConfig(
    horizon=15,            # Look-ahead 15 model steps

    receding_horizon=5,    # Replan every 5 environment steps

    action_block=2,        # Model step = 2 env frames

    warm_start=True,
)

# Instantiate solver with search breadth

solver = CEMSolver(
    model=wm,
    batch_size=4,
    num_samples=400,       # Evaluate 400 candidates per iteration

    var_scale=1.0,
    n_steps=30,            # 30 CEM refinement iterations

    topk=30,
    device="cuda",
)

# Create policy wrapper

policy = WorldModelPolicy(solver=solver, config=cfg)

# Attach to environment

import gymnasium as gym
env = gym.make("CartPole-v1")
policy.set_env(env)

# Execution loop

obs, _ = env.reset()
for _ in range(200):
    action = policy.get_action(obs)
    obs, reward, done, truncated, info = env.step(action)
    if done or truncated:
        obs, _ = env.reset()

Inspecting Solver State

After calling solve(), you can verify the parameter effects on tensor shapes:

out = solver.solve(info_dict)  # Returns dict with 'mean', 'var', 'actions'

print("Planned horizon:", out['actions'].shape[1])  # Matches cfg.horizon

print("Samples used:", solver.num_samples)

Runtime Parameter Adjustment

Modify planning depth without reconstructing the solver:

# Reduce horizon mid-execution for faster computation

policy.cfg = PlanConfig(
    horizon=8, 
    receding_horizon=4, 
    action_block=1
)

# Next call to policy.get_action() reconfigures automatically

Deterministic Debugging Configuration

Disable sampling stochasticity for reproducible debugging:

solver = CEMSolver(
    model=wm,
    num_samples=1,    # Only evaluate current mean

    var_scale=0.0,      # Zero variance

    n_steps=1,          # Single iteration

)

Summary

• num_samples is defined in CEMSolver.__init__ (line 35) and controls candidate action sequence generation, directly impacting memory usage via tensors of shape (batch, num_samples, horizon, action_dim).
• horizon is defined in PlanConfig (lines 29-30) and determines the temporal dimension of the optimization problem, accessed by the solver through its horizon property.
• receding_horizon is defined in PlanConfig (lines 30-31) and governs how many steps are buffered before replanning, converted to environment steps via flatten_receding_horizon.
• Interaction: horizon sets prediction depth, num_samples sets search breadth, and receding_horizon sets execution latency, with action_block scaling the latter two to environment timesteps.

Frequently Asked Questions

What does num_samples control in the CEM solver?

The num_samples parameter controls how many candidate action sequences are drawn from the Gaussian distribution during each iteration of the CEM optimization loop. According to stable_worldmodel/solver/cem.py lines 90-99, increasing this value improves the estimate of the elite action region but raises computational cost proportional to the product of batch size, horizon length, and action dimension.

How does receding_horizon differ from horizon?

While horizon defines how many future steps the world model predicts during planning, receding_horizon determines how many of those planned steps are actually executed before the policy triggers a new solve call. As implemented in stable_worldmodel/policy.py, the WorldModelPolicy converts receding_horizon to environment steps using flatten_receding_horizon and buffers exactly that many actions in self._action_buffer (lines 42-44).

Can I adjust horizon at runtime without recreating the solver?

Yes. You can modify policy.cfg with a new PlanConfig containing a different horizon value. The WorldModelPolicy passes this configuration to the solver via self.solver.configure(), which updates the internal planning tensors. The next call to policy.get_action() will automatically use the new horizon without requiring you to reinstantiate the CEMSolver object.

How does action_block affect these parameters?

The action_block parameter (defined in PlanConfig) scales both horizon and receding_horizon from model steps to environment steps. For example, with action_block=2, a horizon of 15 model steps represents 30 environment frames. This allows you to plan at a lower temporal resolution than the environment requires while maintaining the physical time horizon, reducing computational load while preserving long-horizon reasoning.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}