How the fell_over Termination is Handled in the VelStand Task Curriculum

The VelStand task curriculum manages the fell_over termination by activating it with a 70° tilt limit during the first 500 training iterations, then progressively disabling it by raising the limit to π radians, and finally removing it entirely during evaluation (play mode).

The VelStand environment in the pollen-robotics/microduck_rl repository extends the base walking configuration to teach bipedal robots stable locomotion followed by fall recovery. Understanding how the fell_over termination is handled across training phases is critical for implementing similar curriculum learning strategies in robotic reinforcement learning.

Phase 1: Active Termination for Clean Walking (0–500 Iterations)

During the initial training phase, the environment enforces strict stability criteria. The fell_over termination remains active with a tilt limit of 70 degrees (approximately 1.22 radians), forcing episodes to end immediately when the robot's body tilt exceeds this threshold.

This design choice prioritizes learning stable walking patterns before exposing the policy to complex recovery scenarios. By terminating episodes that result in falls, the agent receives clear negative signals early in training, preventing the accumulation of noisy experiences from uncontrolled ground contact.

The configuration for this phase is defined in the module docstring of src/mjlab_microduck/tasks/microduck_velstand_env_cfg.py, specifically noting that Phase 1 focuses on "clean walking first" before transitioning to recovery behaviors.

Disabling the Termination via Curriculum at Iteration 500

Once the policy demonstrates competent walking at iteration 500, the curriculum disables the fell_over termination by manipulating its angle parameter. Rather than removing the termination check entirely, the system raises the limit_angle from 70° to π radians (180°), effectively preventing it from ever triggering during normal operation.

This gradual disablement is implemented using a CurriculumTermCfg that interfaces with the termination_param_curriculum function from the MDP utilities:

cfg.curriculum["fell_over_disable"] = CurriculumTermCfg(
    func=microduck_mdp.termination_param_curriculum,
    params={
        "term_name": "fell_over",
        "param_stages": [
            {"step": 0,
             "params": {"limit_angle": math.radians(70.0)}},
            {"step": FELL_OVER_DISABLE_ITER * NUM_STEPS_PER_ENV,
             "params": {"limit_angle": math.pi}},
        ],
    },
)

The FELL_OVER_DISABLE_ITER constant (set to 500) multiplied by NUM_STEPS_PER_ENV determines the exact environment step at which the parameter transition occurs. This approach allows the agent to learn fall recovery behaviors without episode resets, treating falls as recoverable states rather than terminal failures.

Play Mode: Complete Termination Removal

During evaluation or deployment, the curriculum system does not execute, requiring a separate handling mechanism for the termination condition. When initializing the environment with play=True, the configuration explicitly removes the fell_over termination from the termination dictionary:

if play:
    cfg.terminations.pop("fell_over", None)

This code appears in src/mjlab_microduck/tasks/microduck_velstand_env_cfg.py and ensures that evaluation episodes continue regardless of body orientation. Removing the termination entirely in play mode prevents unintended episode truncation when testing the final policy's robustness.

Implementation Architecture

The VelStand task builds upon the proven walking recipe defined in make_microduck_velocity_env_cfg from src/mjlab_microduck/tasks/microduck_velocity_env_cfg.py. The make_microduck_velstand_env_cfg function adds the recovery layer by configuring the curriculum parameters and conditional termination logic.

Key implementation details include:

• Source File: src/mjlab_microduck/tasks/microduck_velstand_env_cfg.py (lines 62–70 and 221–231)
• Curriculum Function: termination_param_curriculum located in src/mjlab_microduck/tasks/mdp.py
• Configuration Class: CurriculumTermCfg manages staged parameter updates
• Base Environment: make_microduck_velocity_env_cfg provides the underlying locomotion task structure

Summary

• The fell_over termination starts with a strict 70° tilt limit during the first 500 iterations to enforce clean walking.
• A curriculum parameter stage at iteration 500 raises the limit to π radians, functionally disabling the termination without removing the check.
• In play mode, the termination is completely removed from the configuration to prevent evaluation interruptions.
• The implementation resides in microduck_velstand_env_cfg.py and utilizes termination_param_curriculum for smooth parameter transitions.

Frequently Asked Questions

What is the initial tilt angle limit for the fell_over termination?

The initial tilt limit is 70 degrees (approximately 1.22 radians). This value is set at environment initialization and remains active until the curriculum intervenes at iteration 500.

Why does the curriculum raise the angle to π instead of removing the termination?

Raising the limit to π radians (180 degrees) preserves the termination infrastructure while ensuring it never triggers during normal operation. This approach maintains code consistency with the base velocity environment and allows for potential reactivation or debugging without structural changes to the termination dictionary.

How does play mode affect the fell_over termination differently from training?

In play mode (play=True), the termination is completely popped from the configuration using cfg.terminations.pop("fell_over", None), whereas during training it remains present but with a disabled angle threshold. This distinction exists because the curriculum system does not execute during evaluation, requiring hard removal to prevent unintended episode terminations.

Where is the curriculum stage timing defined for the fell_over disablement?

The timing is controlled by the FELL_OVER_DISABLE_ITER constant (set to 500) multiplied by NUM_STEPS_PER_ENV, calculated in the CurriculumTermCfg parameters within src/mjlab_microduck/tasks/microduck_velstand_env_cfg.py at lines 221–231.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}