The project
Search in the real world happens in places like rock fields and forests, at night as often as during the day. This project coordinates a team of different robots searching those environments. Sensor data from the team is turned into a structured knowledge graph using foundation models, and a centralized multi-agent policy decides what each robot does next.
The work was a collaboration between Utah State University's DIRECT Lab, NVIDIA, the Army Research Laboratory, and the University of Washington. Environments were built in NVIDIA Isaac Sim and GRUtopia: destructible rock fields, forests, and day and night lighting.
My contribution
I implemented the Multi-Agent Transformer (MAT) in Isaac Sim as the team's centralized decision-maker. MAT treats a robot team as a sequence. A transformer encodes every agent's observations together, then decodes actions one agent at a time, with each agent's action conditioned on the actions already chosen for the agents before it.
That sequential structure is the same idea behind HAPPO, which became the focus of my IROS 2025 paper and master's thesis: agents coordinate better when each one accounts for what its teammates are doing.