FEBRUARY 11, 2021 – What does a computer have that humans don’t? The answer depends on who you ask. Those who build surveillance systems would say it’s near-perfect, objective recall. A robotics engineer might say it’s the ability to handle tedium.
A team at Naval Information Warfare Center (NIWC) Pacific conducting research in wargaming says it’s the ability to find winning moves in a game with more possible game combinations than there are particles in the observable universe.
Now they’re trying to discover how an artificial intelligence (AI) agent that uses machine learning (ML) to find those winning moves could help commanders make better tactical decisions.
“Gamifying” command and control
Dr. Doug Lange, senior science and technology manager for ML and AI at NIWC Pacific, likens it to a high-powered game of chess.
“Chess is a relatively simple game compared to warfare,” said Lange. “But tweak the game environment enough and it starts to look like a command and control problem.” NIWC Pacific wants to solve that problem by experimenting with games that mimic the complications of warfare.
One way to use AI for wargaming is to code decision processes into an AI agent, but those processes are products of the human mind, and so, prone to human error. “If I program an agent to play a game, its strategies for how to play are going to be very predictable because I’ve programmed them into it,” said Lange.
A better way to use AI for wargaming is to program into it the bare minimum: give it basic parameters about what types of moves are legal to do, then step back and let it teach itself. Then the agent can learn the value of different moves in the context of the game as it plays, a technique called reinforcement learning.
The benefit of using ML to develop wargaming AI is that the early moves don’t have to be good ones. They just have to be made repeatedly so the agent can accumulate knowledge about which moves lead to desirable outcomes. Eventually the agent, impervious to tedium and subjectivity, begins to paint a picture for optimal game strategy. And because an AI agent can play the game more times than a human could over their lifetime, that picture is better than one any human mind could paint.
This concept isn’t new. A London-based AI company and research laboratory has also demonstrated methods which result in superhuman ability for games such as chess and Go, an abstract strategy game believed to be the oldest board game still played today. Lange and his team at NIWC Pacific are attempting to build on this technology by creating reinforcement learning-based algorithms for military simulations.
In existing wargames created by industry, players must manage resources, synchronize forces, and engage in battles. The interface isn’t realistic because they’re often science fiction-based computer games, but they’re similar to the types of decisions military leaders make. The NIWC Pacific team began using these games to experiment with algorithms, then learned their allies, the U.K. and Australia, were doing the same.
Sharing solutions
The NIWC Pacific team began collaborating with the U.K. and Australia on ML in war games through The Technical Cooperation Program (TTCP), a “Five Eyes” partnership among Australia, Canada, New Zealand, the U.K. and the U.S. Their relationship with Australian and British researchers began during the Autonomy Strategic Challenge, which concluded in 2018, and continues through the AI Strategic Challenge, which begins in 2021 and concludes in 2023.
The U.K. and Australia have been developing and sharing computer games such as the U.K.’s “Hunting of the Plark,” an antisubmarine warfare game, and an Australian surface warfare game. NIWC Pacific developers were using ML to program AI agents to play these abstract war games, but the teams had been developing different AI and ML strategies for wargaming. Now they could share knowledge and, together, learn to win more challenging games.
Along the way, the team at NIWC Pacific also learned the U.S. Army Futures Command was interested in ML for wargaming. NIWC Pacific and their Army partners worked with their British and Australian allies to propose a Coalition Warfare Program (CWP) on wargaming, led by Lange and Dr. Keith Brawner, a senior engineer with the U.S. Army Combat Capabilities Development Command Soldier Center, as co-principle investigators. The CWP, an Office of the Executive Director for International Cooperation program, fosters international partnerships to advance research in technology gaps and helps strengthen strategic alliances.
The British and Australian warfare games’ distribution status made open communication possible but, although they add complexity missing in chess or Go, they only vaguely resemble military decision-making processes. The CWP will provide a secure structure and resources, starting in January 2021, to partner on more complex games built specifically for pursuit of real-world, high-fidelity decision support to warfighters.
When asked what that deliverable would look like, Lange described a war game with all-domain integration in well-defined, realistic military scenarios, with which an AI agent could help decision makers discover winning outcomes.
What it might look like is an AI agent, breaking the silence in a mission control center racked with indecision, by speaking up: “I have an insight.”
‘Making Kennedy real’
This is the plot of NIWC Pacific’s short film “Conflict 2037,” which outlines the Center’s technical vision for using ML capabilities to enhance naval operations. In a futuristic combat information center aboard a fictionalized ship, the ship’s AI agent, named Kennedy, uses its seamless access to virtual oceans of information to present recommendations to an admiral and her staff. They ask Kennedy follow-up questions before concurring with Kennedy’s recommendation.
The film’s star, AI agent Kennedy, is a reminder of what’s possible — and what will soon become crucial for maintaining a tactical edge in the information warfare domain. For the NIWC Pacific team and their CWP partners, it’s a reminder of their roles in building the technology needed to “make Kennedy real.” But NIWC Pacific’s research on wargaming is just one piece of the CWP, and just one part of the Center’s mission to amplify lethality in all domains.
“We want to foster significantly faster learning about war-winning, all-domain integration,” said Capt. Andrew Gainer, commanding officer of NIWC Pacific. “‘Kennedy’ is just an example, and technology is only part of the equation. Along with the infrastructure, policies, and budget needed for rapid capability development and experimentation, we require partners with a similar sense of urgency and understanding of the high-end maritime fight.”
Winning harder games
When it comes to “making Kennedy real,” the partnership with the U.K. and Australia could be a crucial piece in helping NIWC Pacific realize the Center’s vision depicted in “Conflict 2037.” As Lange explains, superiority in information warfare is a more complicated game than any one human mind can win.
“In chess, computers exhibit superhuman capabilities by using machine learning algorithms to learn to play the game. The best players in chess aren’t people, they’re programs,” said Lange. “When programs designed for wargaming are used in tandem with human intelligence, we’ll make the best warfighting decisions.”
But it’s the best naval, joint, and foreign players who are needed to build those programs. Through the wargaming CWP, NIWC Pacific has helped assemble the tools required for rapid innovation: the infrastructure, policies, and budget. In the coming years they’ll get to use these tools to further their research on war-winning AI. “Now the question becomes,” said Lange, “can we learn to play more complicated games?”
Story by Maison Piedfort
Naval Information Warfare Center Pacific