SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning

Luo, Lirui; Zhang, Guoxi; Xu, Hongming; Fang, Cong; Li, Qing

TL;DR

SPHERE studies why Mixture-of-Experts (MoE) policies lose their ability to adapt in continual RL. It shows that learning updates collapse into too few directions, then keeps expert features diverse so later tasks remain learnable.

Phenomenon → diagnosis → regularization

Problem A policy trained across many tasks can stop adapting to later tasks.

Method A regularizer that keeps expert features diverse instead of collapsed.

Result Better continual-control performance and healthier feature geometry across tasks.

Mechanism: Why Policies Stop Learning in Continual RL

The mechanism story starts with the observed learning slowdown, connects it to collapsed update directions, and then shows how SPHERE keeps those directions more diverse.

Phenomenon → spectral collapse → SPHERE

Phenomenon In continual RL, policies keep receiving new experience but can stop changing effectively.

Diagnosis The update geometry loses rank, meaning learning concentrates into too few functional directions.

SPHERE SPHERE keeps the weighted expert-feature Gram more isotropic, which helps preserve diverse update directions.

Overview showing continual-RL plasticity loss, spectral collapse, and SPHERE regularization

Update Geometry: Collapse vs. SPHERE

Use the slider · or press Play

The visualization makes the diagnosis concrete. A unit sphere of possible gradient directions becomes an ellipsoid after multiplication by the empirical neural tangent kernel (eNTK) matrix $K$ . When $K$ becomes low-rank, one axis shrinks toward zero and the ellipsoid degenerates toward a near-plane or line; SPHERE keeps the spectrum more isotropic.

Top row ( $\nablafL$ ) shows the input sphere of directions; bottom row ( $K\nablafL$ ) shows the stretching process.

Real-data interactive panel

This interactive animation is generated from the underlying HumanoidBench experiment data used for Fig. 1. For each task, the top-3 eigenvalues of

K

shape the ellipsoid, and gradient-direction samples are projected into the same 3D eigenspace so the slider shows how the measured update geometry evolves.

Baseline (Top‑K MoE)

collapsed spectrum → near low-rank

SPHERE

isotropic spectrum → diverse updates

Experiments & Analysis

These panels show where continual training fails, how the update geometry collapses, how SPHERE improves performance on two control benchmarks, and which design choices matter.

Phenomenon · Diagnosis · Performance · Design · Feature Evidence

Phenomenon

Continual RL Degrades Across Architectures

Before introducing SPHERE, the same HumanoidBench architectures succeed less when trained continually than when trained task by task.

Diagnosis

Spectral Plasticity Collapses During CRL

The performance drop comes with lower eNTK effective rank: baseline updates collapse, while SPHERE keeps the update geometry better conditioned.

Performance

SPHERE Improves Continual Training Performance

MetaWorld

On MetaWorld, SPHERE narrows the gap between task-by-task training and continual training, giving the strongest average success among the compared methods.

HumanoidBench

On HumanoidBench, the same pattern holds: SPHERE improves average success over the unregularized MoE and continual-learning baselines.

Design

What Matters in SPHERE?

HumanoidBench continual-RL design ablation

Variant	Average success
Without SPHERE	0.36 ± 0.08
With SPHERE	0.54 ± 0.12
All hidden expert layers	0.42 ± 0.07
Per-expert loss sum	0.40 ± 0.08
Gradient-factor regularization	0.43 ± 0.09

The ablation asks which SPHERE design choices matter. The default last-layer, routing-weighted expert-feature Gram is the strongest tested design; alternatives help, but none matches the full setup.

Qualitative

Feature Collapse Across Tasks

Using the same held-out Stair states after each task, we visualize expert features. Without SPHERE, points quickly concentrate along one dominant direction; with SPHERE, multiple directions remain active across the sequence.

Feature Evidence

Weighted Expert Features Track the eNTK Rank

Scatter: effective rank of weighted expert feature Gram vs effective rank of eNTK

The weighted expert-feature Gram follows the same trend as the eNTK effective rank, supporting it as a practical proxy for spectral plasticity.

BibTeX

@inproceedings{luo2026sphere,
  title   = {SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning},
  author    = {Luo, Lirui and Zhang, Guoxi and Xu, Hongming and Fang, Cong and Li, Qing},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  year      = {2026}
}