ESA Puts Augmented Reality Through the Paces

There’s a lot of attention currently focused on how NASA is planning to send Microsoft HoloLens hardware to space to help astronauts perform tasks. According to a post published on the Trove blog in June 2015, the first use case being tested will permit NASA professionals on Earth to see what the astronauts see on the International Space Station (ISS). In Remote Expert Mode, HoloLens will be valuable when the astronaut encounters undocumented situations. It will also be possible for HoloLens to provide procedural guidance, for example, to retrieve objects or to put objects away in their correct place after use.

Tests of HoloLens, both on the ground and in underwater laboratories simulating space, will certainly validate the latest technology components Microsoft provides but will not be the first tests of Augmented Reality in space.

A First Use Case for Augmented Reality in Space

According to David Martinez, a simulation and visualization engineer and member of the European Space Agency (ESA) Software Systems Division, Augmented Reality was first evaluated by ESA for space use in a project beginning in 2006. Using the ESA-designed Wearable Augmented Reality (WEAR) system, Augmented Reality was tested on Earth and, eventually, on the ISS in 2009. The use case was for an astronaut to inspect and, if needed, to service ISS air quality system components. Before examining and changing filters on the air quality system, an astronaut had to remove a panel on the floor. Then cables and hoses needed to be repositioned. Once the filter was accessible, the color of an indicator had to be examined. 

“We learned a lot about what was and wasn’t possible with the technology at that point in time,” recalls Martinez.

Exploring Guidance and Remote Expert Assistance

ESA works with payloads designed for a wide variety of different purposes. Some of the payloads end up on the International Space Station. As astronauts on the ISS cannot be trained on all the possible payloads in advance, they would like to have clear and compact Augmented Reality-assisted systems that make sure the astronauts conduct experiments consistently and correctly, even when they are not trained on them before going into space.

In 2014 the ESA team collaborated with the Technical University of Delft to explore the use of Augmented Reality using hands-free and head-mounted displays to provide remote expert assistance for performing experiments. The study used a payload representative of what’s on the Columbus module, a science laboratory that is part of the ISS and one of the most important contributions to the ISS made by the European Space Agency.

“We demonstrated that the remote expert was able to support the hands-on use of the various dials, buttons and knobs,” explains Mikael Wolff, a senior software manager who manages several projects in the domain of crew informatics. 

“The remote expert could speak to the user and also annotate the object in the astronaut’s field of view with arrows and text messages that would remain in place with respect to the payload,” clarifies ESA engineer Sérgio Agostinho.

Technologies are continually advancing and ESA is testing systems for their ability to track targets in 3D with far higher flexibility than earlier generations. “We’re not using fiducial markers on any of our current projects,” assures Agostinho. He feels that if a system it is to be deployed on the ISS, it can’t rely on markers.  “We’re aiming for the Iron Man quality of experience,” he says enthusiastically.

Long List of Use Cases

“We know that there are many ways Augmented Reality may bring value to projects and people on the ground and in space,” reports Wolff. “We’re always coming up with new ideas.”

In collaboration with partners in industry and academia, ESA is currently focused on several use cases it considers to be relatively low hanging fruit. One of these is support for complex product assembly, integration and testing on Earth. ESA and European aerospace industry engineers are routinely involved in, support or perform the final assembly and integration of parts procured from aerospace industry suppliers. Components include everything from printed circuit boards to large payload systems and harnesses that eventually go into space.

Augmented Reality could assist technicians during the assembly of telecommunication satellites. Currently the manual procedures take days or weeks to complete. By highlighting for users the steps directly on the parts of the satellite with Augmented Reality, the assembly, integration and testing processes could be performed with fewer errors and more quickly.

Barriers Remain

The ESA team has segmented its current and potential future Augmented Reality projects into those that could provide value when engineers perform tasks on Earth and others that could lead to AR being deployed in space for use by astronauts. This is due to the fact that systems or components that meet requirements on Earth are not immediately ready to go to the ISS. Not only is hardware certification for custom built and commercial off-the-shelf devices required, but software conflicts or bugs simply aren’t tolerated in space.

Before anything is sent to the ISS, it must undergo extremely rigorous testing and validation. “This means that almost everything on ISS is at least one generation behind what’s available on Earth, in terms of technology maturity,” explains Martinez.

“We also have real challenges with lack of interoperability,” says Wolff. “As an industry and as a public agency we can’t rely on a single supplier for any technology component. The Augmented Reality ecosystem needs to expand and different vendors need to provide components that are comparable or else we could put the agency or a mission at risk.”

Despite delays and the complex testing environments, ESA engineers continue to study AR use cases and to evaluate the latest technologies. As commercial solutions mature and pass required reliability and accuracy thresholds, having them in use on the ISS and on complex space assembly and integration projects on Earth could become commonplace.