The US Military Is Building Its Own Metaverse

Defense tech companies have latched on to the metaverse hype—but what they’re building will be a far cry from Meta’s virtual world.
Red6 Berkut flight aircraft flying over seaside ridge landscape at sunset
Mike Killian Photography/Red 6

On May 10, two fighter pilots performed a high-altitude proto-metaverse experiment. A few thousand feet above the desert of California, in a pair of Berkut 540 jets, they donned custom AR headsets to connect to a system that overlaid a ghostly, glowing image of a refueling aircraft flying alongside them in the sky. One of the pilots then performed a refueling maneuver with the virtual tanker while the other looked on. Welcome to the fledgling military metaverse.

It isn’t only Silicon Valley that’s gripped by metaverse mania these days. Just as tech companies and corporations are scrambling to develop strategies for virtual worlds, many defense startups, contractors, and funders are increasingly talking up the metaverse, even if its definition and utility aren’t always clear.

The key technologies needed for the metaverse—augmented and virtual reality, headmounted displays, 3D simulations and virtual environments built by artificial intelligence—are already found in the defense world. The result is a lot less polished, cutesy, and spacious than Mark Zuckerberg’s virtual world vision, but that’s partly the point. And there’s a good chance that the underlying tech could take off, even if it stutters in the civilian realm.

Courtesy of Red 6

A mix of augmented reality, artificial intelligence, and video game graphics, for instance, have enabled fighter pilots to practice dogfighting against virtual opponents, including Chinese and Russian warplanes, while pulling several Gs. Red 6, the company that’s developing the technology, says this delivers a far more realistic test of a pilot’s abilities than a conventional flight simulator. “We can fly against whatever threat we want,” says Daniel Robinson, founder and CEO of Red 6. “And that threat could be controlled either by an individual remotely or by artificial intelligence.”

Red6’s AR technology has to work in more extreme conditions, with lower latency and higher reliability than consumer AR or VR headsets. Robinson adds that the company is now working on a platform that will allow many different scenarios to be represented in augmented or virtual reality. “What we’re building is really a military metaverse,” he says. “It’s like a multiplayer video game in the sky.”

Metaverse-related ideas are already part of some of the latest military systems. The high-tech helmet for the new F-35 fighter jet, for instance, includes an augmented reality display that shows telemetry data and target information on top of video footage from around the aircraft. In 2018, the US Army announced that it would pay Microsoft up to $22 billion to develop a version of its HoloLens augmented reality system for warfighters, known as the Integrated Visual Augmentation System (IVAS).

Courtesy of Red 6

VR and AR have become routine aspects of military training in recent years. In 2014, the Office of Naval Research and the Institute for Creative Technologies at the University of Southern California developed Project BlueShark, a system that allowed sailors to drive vessels and collaborate in a virtual environment. Another effort, called Project Avenger, is now used to help train US Navy pilots. The US Air Force is using VR to teach pilots how to manage aircraft and missions. VR is also used to help treat veterans for chronic pain and post-traumatic stress. And Boeing has created an AR environment that lets mechanics practice working on planes before stepping aboard a real one.

Recently, the US military has begun exploring more complex virtual worlds. There is also growing interest in connecting and combining virtual worlds in a way that resembles metaverse thinking. In December 2021, the US Air Force held a high-level conference involving over 250 people in locations stretching from the US to Japan, via a virtual environment. “The promise is integrating these technologies,” says Caitlin Dohrman, general manager of the defense division of Improbable, a company that develops virtual world technologies, has created sprawling virtual battlefields featuring over 10,000 individually controlled characters for the UK’s military wargames, and also works with the US Department of Defense (DOD). “It is an extremely complex type of simulation, especially given the fidelity that the military demands,” Dohrman says. “You can either have live players who are participating in the simulation or [characters] can be AI-enabled, which is often what the military does.”

Palmer Luckey, the founder of Oculus, a VR company Facebook acquired in 2014, says Zuckerberg’s decision to go all-in on VR and the metaverse created a massive amount of expectation in the commercial world. “Everyone on their quarterly corporate calls, like a week or two later, they’re being asked by investors, ‘What’s your metaverse play?’,” he says.

In 2017, Luckey cofounded the defense company Anduril. He says that despite all the recent metaverse hype, there is big defense potential, partly because military training is so important and costly. But he says the technology does not have to be hyper-realistic to be useful, and he wants Anduril to focus on only using the technology where necessary. “Everything we’re doing with VR is something where it is uniquely better than any other option,” he says. This includes using VR to train people to operate Anduril’s drones, he says, or to display information about an area using data from sensors on the ground.

As with Zuckerberg’s planned metaverse, newer military systems rely heavily on AI to be effective. In October 2020, the AR technology developed by Red6 was used to pit a real fighter pilot against an aircraft controlled by an AI algorithm developed as part of a Defense Advanced Research Projects Agency (DARPA) AI dogfighting project. The AI top gun, created by another startup called EpiSci, learned how to outmaneuver and outgun an opponent through a process of trial and error. The AI pilot eventually developed superhuman skills and was able to beat its human opponent every time.

Another DARPA project, called Perceptually-enabled Task Guidance, aims to create an AI assistant that watches what a soldier is doing and offers advice through speech, sound, or graphics. In contrast to the augmented reality system developed by Boeing, which only works in a specific setting, such a system would need to make sense of the real world. Bruce Draper, the DARPA program manager responsible, says the real value of technologies being explored by the military lies in merging the real and the virtual. “The metaverse is mostly virtual, and virtual worlds are useful for training, but we live in the physical world,” he says. “The military domain is inherently physical, it’s not about an abstract metaverse.”

But efforts to merge the virtual and the real world have encountered problems. In March 2022, a leaked Microsoft memo reportedly showed that those working on IVAS, the US Army version of the HoloLens AR headset, expected it to be received badly by users. And an audit released by the DOD in April 2022 concluded that the US Army could waste its money as a result. Jason Kuruvilla, a senior communications manager at Microsoft, shared several statements from high-ranking army figures proclaiming the potential of the IVAS. He also pointed to a 2021 DOD report that discusses the importance of developing IVAS rapidly, allowing problems to be ironed out along the way.

Such high-profile and expensive endeavors have only boosted the confidence of those pushing the military metaverse. “I know that this is the future of military training,” says Doug Philippone, global defense lead at Palantir, a defense company that has invested in Anduril. Philippone is also a cofounder of Snowpoint Ventures, which has invested in Red6. “But I also see it as the future of the way that the military fights and makes decisions. So it's not just about fighting, it's about making decisions.”

Luckey says Anduril is already working on technology that could do this in training missions and combat. “The next big step for us, which I am really excited about, is taking from our core product and piping that data to heads-up displays that troops on the front line are going to be able to wear,” he says.

But how much of this cutting-edge tech makes it to the front line—or even into training exercises—remains unclear. Sorin Adam Matei, a professor at Purdue University in West Lafayette, Indiana, who has developed virtual battlefield training platforms for the US military, says the tech deployed will often be considerably simpler than metaverse boosters imagine. He suggests that a simpler version of the IVAS headset may eventually be integrated into an AR rifle scope. “When you are out there shooting and being shot at, the last thing you want to worry about is another piece of equipment,” he says. And technology does not need to be as expansive as a metaverse to be useful. “We need to think a bit more about this metaverse metaphor—which is powerful but also has its limitations.”

Updated 19/05/2022, 11:00 am ET: The article previously stated incorrectly that Palantir has invested in Red6.