OCTOBER 20, 2021 – The 76th Software Engineering Group has a few highly specialized and advanced teams working on cutting edge technologies and advancements for the Air Force. One of those teams is Xanatos Gambit, a team of eight individuals who are making big splashes in the field of embedded development and hardware emulation.
The main strategy the XG team uses as they think about the necessary digital transformation of embedded development is: “Think Big, Start Small, Scale Fast.”
“This thought process allows them to think outside the box, step outside the current limitations within embedded development, and come up with solutions that are innovative and impactful to the warfighter,” said Director Kevin Higgs, with the 559th Software Engineering Squadron.
In August 2020, Air Force Chief of Staff Gen. Charles Q. Brown Jr. released a Strategic Approach document titled, “Accelerate Change or Lose” that implored the Air Force to make some big changes, fast. In his introductory note, Brown said the Air Force “must accelerate change to control and exploit the air domain to the standard the nation expects and requires from us. If we don’t change – if we fail to adapt – we risk losing the certainty with which we have defended our national interests for decades.”
Initially, the XG team began their journey by reaching out to industry leaders to determine some of the biggest constraints facing embedded development as it relates to real-time, safety-critical avionics software. During this endeavor, XG inspired the creation of an Embedded Consortium, which was named “Team 8.” This consortium was commissioned by the Air Force chief software officer.
Following the ACOL strategy of “collaborating within and without,” XG is leading the way by chairing and administering Team 8. Through Team 8, XG has been in contact with high-level leadership of more than 20 companies, multiple Department of Defense agencies, major contractors and various research institutions. Firsthand insight has allowed XG to be in a unique position to understand some of the biggest constraints impacting embedded development.
“The main impediment that XG noted is that traditional avionics embedded development requires writing software applications that are specific to its hardware and operating system,” said Felix Cortez, Xanatos Gambit product owner with the 559th SWES. “Unlike desktop/web applications, which can be installed on a variety of general-purpose computer systems, embedded software has fixed hardware requirements.”
Embedded software is written to run on a particular device with processing and memory restrictions tied directly to that device’s specifications.
“This proves to be cumbersome early in the lifecycle process as significant architecture needs to be defined before writing code,” Cortez said. Furthermore, he added that this constraint limits reuse and refactoring during hardware end-of-life that increases overall lifetime costs. This process hinders the fundamentals of Agile by requiring physical access to the hardware for testing, pushing the lifecycle further to the right. Lastly, avionics software has unique certification and safety standards that must be met as a foundation for any Development Security Operations implementation.
Understanding the challenges of “Good enough today will fail tomorrow,” which is another strong point that Brown made in his note, the XG team focused on addressing various constraints within embedded development. Starting from the beginning of the development phase, XG explored possibilities of separating the software architecture from the specific hardware using Model-Based Systems Engineering.
XG then used their experience with emulating hardware, and they found Free and Open-Source Software they could use to reduce dependencies on licensing and access to flight hardware. Implementing these technologies with automation and cloud integration was determined to be foundational in supporting continuous integration principles within a DevSecOps workflow.
Cortez said the prospects of these big ideas started small with incremental capabilities.
“XG set out to create a simple application, starting from the requirements phase,” he said. “Models were created to represent high-level and detailed functional requirements. These models became the authoritative source of truth from which qualified source code was auto generated that is hardware and operating system agnostic.”
Throughout this phase, XG continued to remove the physical constraints from hardware with the use of hardware emulation. Eventually, code generated from models need to be run on selected avionics hardware. However, hardware emulation reduces that constraint, providing early high availability development testing prior to physical System Integration Lab testing.
In order to “Scale Fast” this approach, Cortez said XG exploited cloud native technologies via the use of pipelines, containers and Kubernetes.
“We immediately discovered that containerized embedded toolchains are limited, both commercially and in the DoD Iron Bank,” he said.
XG used a FOSS Real Time Operating System and containerized the associated build tools and unit testing tools.
“There are no products that can containerize a running RTOS, a requirement for integration testing in a cloud pipeline,” Cortez said. “So, XG created a multilayer abstraction technique called a ‘nesting doll’ which allows embedded software to run on emulated hardware in a container for automated testing. The XG team created these containers for distribution via the Iron Bank by balancing the need for flexibility of different hardware configurations and the benefits and reusability of standardization.”
XG’s demonstration standing up non-x86 embedded software pipelines providing automated scanning, building and functional testing in a cloud environment allows software development driving the primary Air Force mission to fully take advantage of Agile and DevSecOps benefits.
This demonstration caught the attention of an Air Force Vanguard project called Golden Horde. Vanguards are high-profile Air Force research projects to rapidly advance emerging weapon systems and warfighting concepts through prototyping and experimentation. The Golden Horde’s goal is to develop a swarm of networked munitions that can autonomously attack multiple targets simultaneously. XG is working to scale these demonstrated capabilities within Golden Horde inspiring other Air Force projects to evolve current embedded development practices.
By Heather Fairbanks and James Brewer
76th Software Engineering Group