Jacob Haight
All work

Advancing Multi-Agent Robotics Simulations Through Heterogeneous Reinforcement Learning in IsaacLab

M.S. thesis, Utah State University, 2025

MAPPO: dragging feet, frequent stumbles
HAPPO: higher leg lift, legs in sync
The same quadruped with each leg controlled by a separate agent, trained with two different multi-agent algorithms.

The question

The thesis builds on my IROS 2025 work, which brought heterogeneous-agent RL into NVIDIA Isaac Lab. Here I asked whether that extension pays off in practice: do the theoretical advantages of heterogeneous algorithms show up as better policies inside Isaac Lab, and what do they cost in training speed?

Isaac Lab's built-in multi-agent algorithms, MAPPO and IPPO, update every agent at the same time. Each agent improves against an outdated picture of what its teammates will do, so individually good updates can combine into a worse team. HAPPO updates agents one after another, with each update accounting for the ones before it, which gives a guarantee that the team's joint policy keeps improving.

One robot, many agents

To test this, I made a single robot the team: several agents jointly control one body, so they have to coordinate every step to walk at all. A Spot quadruped's twelve joints were split between cooperating agents in two ways. Each agent can also see its teammates' observations, but with much more noise added.

Quadruped diagram with four agents, one controlling each leg
Spot-Factorized-4x3. Four agents, one per leg, each with the same action and observation space. The more homogeneous task.
Quadruped diagram with two agents splitting the joints into two groups
Spot-Factorized-4+8. Two agents splitting the joints 4 and 8 along hip and knee lines, with different action and observation spaces. The more heterogeneous task.

Results

I compared HAPPO with MAPPO in two forms: with shared parameters (MAPPO), and with a separate network per agent (MAPPO*).

Average reward per step over 10 million steps for MAPPO, MAPPO*, and HAPPO on both tasks
First 10 million steps. MAPPO with shared parameters falls behind on the four-leg task, while HAPPO and MAPPO* stay close on both.

The first result did not match the theory: HAPPO didn't beat MAPPO*. I considered four explanations. HAPPO's extra machinery might only help in niche cases, the tasks might be too simple to tell the algorithms apart, its advantage might only appear with longer training, or MAPPO* might be overfitting the reward. A 150-million-step run on the four-leg task answered all four.

Four learning curves over 150 million steps: total reward, gait reward, linear velocity error, and angular velocity error for MAPPO* and HAPPO
150 million steps on Spot-Factorized-4x3. HAPPO pulls ahead on total and gait reward and keeps a lower angular velocity tracking error.

With enough training, HAPPO's sequential updates find a better joint policy, and the difference is visible in the gait: the HAPPO-trained robot lifts its legs higher and moves them in sync, while the MAPPO-trained robot drags its feet and stumbles.

Cost in training speed

Sequential updates add almost no overhead. Throughput stayed within 1% of MAPPO*, and the number of agents affected speed more than the choice of algorithm.

AlgorithmTaskSteps per secondTime (s)
HAPPOSpot-Factorized-4+824,148410
MAPPO*Spot-Factorized-4+824,131410
HAPPOSpot-Factorized-4x321,607458
MAPPO*Spot-Factorized-4x321,817453
MAPPOSpot-Factorized-4x324,226408

Limitations and next steps

  • Isaac Lab released updates during the project, so full integration with the latest version needs some rework.
  • The new environments need more reward tuning before their policies could carry over to real robots.
  • Natural next steps are graph neural network policies in Isaac Lab and more advanced algorithms built on sequential updates.