Performance Cliffing for G1 Locomotion

[Buzz-Beater]

Hi,

Thanks for the great work. I was trying to reproduce the results from the paper on the G1 locomotion task. However, I observe a significant performance cliff at around 4K steps (figures attached) when I extend the training steps to 5K. Any ideas on why this is happening?

To fully reproduce my issue, I'm training with 4096 environments on a single NVIDIA 3090 GPU using the isaaclab_experiments codebase. All other hyperparameters and package versions match the default description.

Looking forward to thoughts from you guys :-)

[stergiosba]

Hi, did you ever figure this out?

[Abhishek21g]

Paste on fpo-control#4

Hi — I dug into this on a branch without local CUDA (no 3090 handy), so sharing analysis + a proposed fix for maintainer/community GPU verification rather than claiming reproduced curves.

Root cause (code review)

G1 locomotion configs are tuned for ≤2000 iterations. Extending to 5000 with default hyperparams continues applying:

• ema_decay=0.95 (documented in rl_cfg.py for ~1500–2000 step runs)
• num_learning_epochs=32
• normalize_advantage=True with fixed LR

Once reward plateaus (~37 by iter 2000), 32 extra epochs × 3000 more iterations drives unstable CFM ratio updates — classic overtraining past the tuned budget, not an env-reset bug.

Proposed fix

When --max_iterations > 2000 on G1/H1 locomotion (and no explicit agent.algorithm.* overrides), auto-apply:

Param	Default	Long-run
ema_decay	0.95	0.99
num_learning_epochs	32	16
normalize_advantage	true	false

Manual workaround:

python isaaclab_fpo/scripts/train.py \
    --task Isaac-Velocity-Flat-G1-v0 --headless \
    --num_envs 4096 --max_iterations 5000 \
    agent.algorithm.ema_decay=0.99 \
    agent.algorithm.num_learning_epochs=16 \
    agent.algorithm.normalize_advantage=false

For paper-matching curves: stop at 2000 iters.

Ask

Could someone with a CUDA box run isaaclab_experiments/scripts/reproduce_g1_cliff.sh and paste before/after Train/mean_reward curves? I'll open a PR with the fix + a small log-analysis tool (fpo_training_receipts doctor) once there's one confirming run.

Branch: fix/g1-reward-cliff (on my fork — link when pushed)