Augmented Reality and the Internet of Things boost human performance

Smart connected objects allow extensive optimizations and accurate predictions in the production line. However, this is not the only benefit that IoT can generate in industrial settings.

The purpose of this post is to explain how Augmented Reality (AR) can provide additional value to IoT data serving as a visualization tool on the shop floor. Operators can achieve better results in less time in a number of use cases by using AR devices to consume up-to-date contextually relevant information about IoT-enabled machines.

Industry 4.0 and the Internet of Things

The extensive use of Information and Communication Technologies (ICT) in industry is gradually leading the sector to what is called the “fourth industrial revolution,” also known as Industry 4.0. In the Industry 4.0 production line, sensors, machines, workers and IT systems will be more deeply integrated than ever before in the enterprise and in the value chain. The complete integration will ultimately optimize the industrial process, fostering its growth and driving greater competition within markets. A report from the Boston Consulting Group, summarizes the nine technology advancements that are driving this revolution and will eventually define its success:

• Big Data and Analytics
• Autonomous Robots
• Simulation
• Horizontal and Vertical Integration
• The Internet of Things
• Cybersecurity
• Cloud Computing
• Additive Manufacturing
• Augmented Reality

The Internet of Things (IoT) leads the advancements in the field as an enabling technology. The IoT concept is based on building intelligence into objects, equipment and machinery and enabling data about their status to be transmitted over the Internet for human or software use. Through connectivity and unique addressing schemes things are able to cooperate in order to reach a common goal. Research has identified the three basic characteristics of smart objects:

• to be identifiable through unique addresses or naming systems,
• to be able to connect to a network,
• to be able to interact with each other, end users or other automatic components.

Industrial settings are paving the way for the introduction of IoT into modern society. In the Industrial IoT (IIoT) vision, any single segment of the production line can be constantly monitored through the introduction of sensors, and intelligent machine and pervasive networking capabilities. Central data gathering systems can collect and analyze data about the status of the entire supply chain and dynamically react in case of failures, resource shortages and demand variations. The value brought to industry by IoT is cumulative, as more devices are brought online and their interactions captured and analyzed. In fact, data gathering and aggregation of supply chain variables can help to optimize production in terms of reduced waste of resources, reduced downtime, improved safety, sustainability and greater throughput.

Big Data Analytics and Machine Learning are the core technologies through which the enterprise can make sense of this enormous flow of data coming from industrial facilities. These enable the creation of mathematical models that constantly improve the precision with which they represent the real-world settings as more data feeds into them. Called “digital twins”, these models are then used not only to analyze and optimize the behavior of the equipment and the production line, but also to forecast potential failures (preventive maintenance is a byproduct of Big Data analysis).

IoT as a tool for human effectiveness

The abovementioned benefits that come from the integration of IoT into advanced process automatization (using technology to allow processes to take place without human input) are not the only advantages. The introduction of smart objects into industrial contexts provides the possibility of greater effectiveness among the people working on the shop floor.

Data gathered from sensors is essential for on-site decision-making and correct completion of tasks as workers operate with smart equipment. Smart objects, also called cyber-physical systems, can support workers, improving proficiency and safety, on different levels.

Design, maintenance, repair and fault diagnosis are complex tasks that require a human operator to interact with sophisticated machinery in the new industrial paradigm. The information needed to successfully carry out these tasks is proportional to the complexity of the tasks and the equipment involved. Real-time and historical data about the functional activities of the equipment are therefore critical for the decision-making process as the complexity of the systems increases. Access to this information on the site where the operator is performing these tasks becomes essential to correctly and efficiently perform them.

To give an example, the recovery procedure of a complex machine experiencing a failure needs to be informed by the current status of the components of the machine itself. Similarly, the proper configuration of complex mechanical systems is conditional on the values of certain internal variables measured by equipped sensors. The operator in charge of these procedures needs to be able to diagnose the problem and pinpoint the exact location of the failure when in front of the equipment in order to immediately revert it to an optimal state. Generally this is done by analyzing real-time sensor data, computer-generated analyses or historically aggregated data.

Current issues with human consumption of IIoT data

In the current state of integration, in cases where IoT technologies are deployed, the data is sent to central repositories where operators in control rooms are in charge of monitoring and analyzing it. However, in most situations, these central control rooms are distant from the location where the data is actually needed. The engineer in front of the machine in need of assistance is required to cooperate remotely with the central control room in order to diagnose a fault. The interaction in this scenario can be very slow as the on-site engineer needs to verbally interpret the information provided by the remote operator, while the operators in the control room do not have the on-site engineer’s spatial reference information to guide them, thereby slowing down the cooperation and increasing the time required to solve the problem.

Some organizations have attempted to address this problem by deploying laptops on the shop floor that can access remote data. Despite being somewhat effective, laptops are only a partial solution to the problem, as the devices are usually not aware of the physical surroundings and the intention of the operator, thus dividing his attention between the object of interest and the interaction with the mobile device. In general, mobile devices currently used to interact with IoT data on the shop floor lack the ability to interpret what the operator is looking at and the intent of the operation unless the operator manually interacts with the software interface, filtering out the unneeded data.

Other companies are deploying advanced touch interfaces directly on the smart equipment. While this partially solves the issue, it also multiplies the number of screens on the shop floor and does not provide a solution for equipment that cannot be fitted with a screen (e.g., outdoor heavy machinery, oil and gas pipes, etc.).

Another crucial piece of information missing from current Human-Machine Interfaces (HMIs) is the spatial reference of the data stream. In certain situations, it is very important to visualize how the data sources are physically located in the three-dimensional space in order to diagnose a fault. This information gets lost if the data streams are visualized exclusively using 2D interfaces or schemes that do not take into account the physical structure of the equipment. For example, the figure below references two different visualizations of an oil pipeline with IoT-connected valves that stream data about their functional status. The representation on the left is not aware of the spatial disposition of the valves, while the visualization on the right makes it much easier to diagnose that the problems with the valves are caused by an external interference around the southern portion of the pipeline.

AR and IoT: a match made in heaven

Augmented Reality provides an effective answer to all the aforementioned issues with IoT data consumption on the shop floor. Modern AR-enabled devices (both handheld and head-worn) provide a media-rich ubiquitous interface to any type of network data via wireless connection. Using sensing technologies, these devices are capable of understanding what the operator is looking at and therefore only display the data that is actually needed for the operation at hand. Using AR devices, the operator is empowered with the ability to visualize processed or unprocessed IoT data in an incredibly intuitive way.

The worker starts the interaction by pointing the AR-enabled device towards the piece of equipment in need of assistance. The device scans the equipment using cameras, identifies the object and reconstructs a spatial model of it. The application automatically gathers the list of available sensors connected to the machine interrogating the central repository and displays the gathered information on the equipment itself, in the exact location where the sensors are currently measuring the data. Interacting via the interface, the operator can also search for historical data needed to diagnose the fault. The data thus visualized not only contains the same informative power as it does on other mobile devices, but also provides the operator with the spatial relationship of the data with the machine itself.

AR provides a display for anything. As all the objects/screens AR devices can render are completely digital, there are no restrictions as to how and where IoT data can be visualized. Even the dirtiest and most remote oil pipe, the hottest jet engine or the loudest metal printing machine can be overlaid with a number of virtual data visualizations for the operator to analyze during the process. All in all, if an object generates IoT data, AR can visualize it.

In addition, AR allows the same information to be displayed in different, more intuitive ways. Traditionally, sensor data is visualized using a mix of numbers, graphs and gauges. However, using AR, new forms of visualization, customized for the purpose, can be designed. These visualizations can speed up the interpretation of data and better highlight faults. For example, the pressure and temperature measurements along a pump output pipe can be displayed using a color mapped three-dimensional flow visualization overlaid directly on the pipe itself, allowing the operator to virtually “visualize” the behavior of fluids inside the pipe, speeding up parameters for tuning or fault detection processes.

Use cases

AR and IoT can be combined to address a number of use cases that benefit both private and public sectors. There are some common factors shared by most of these use cases, such as mobile access to data in remote locations, the inaccessibility to certain parts of the equipment, the difficulty to fit a screen on the object of interest or the need for extreme operative precision.

1. Complex machinery service efficiency:  for organizations that operate and maintain large fleets of complex machinery, from aircraft to locomotives, service and repairs can be slow and costly. Without specific data on particular components in need of repair or the ability to predict when service is needed, assets may be taken out of service unexpectedly and service technicians may need to spend valuable time testing and isolating issues. Organizations can accelerate the process and improve efficiency by combining IoT and AR technologies. Arming assets with sensors enables them to stream data directly from the assets. Using this data to create digital twins of the assets, organizations can self-analyze and self-predict when and how components need to be maintained. Using AR, that data can be translated into visual information — for example, highlighting which fuel injectors in an engine are causing oil pressure problems and need to be replaced. By guiding the repair technician immediately to the source of the issue, the AR/IoT combination limits the scope of the work to only what is needed. Step-by-step instructions delivered via AR ensure that the repair work is performed correctly and efficiently. GE Transportation is applying PTC’s ThingWorx and Predix software to realize efficiency gains in the 1,300 locomotive engines it repairs every year.

2. Mechanical equipment monitoring and diagnosis:  many mechanical parts, such as engines, pumps, pipelines and industrial machines, are fitted with a large number of sensors to control physical variables, such as temperature, pressure, speed, torque or humidity. These measurements are used not only to control the machine itself, but also to monitor and verify its correct functioning. During configuration and fault diagnosis, it is essential for the operator to visualize these values in real time in order to properly set up the machine in one case, and correctly identify the root of the fault in the other. Using an AR device, the operator can visualize patterns directly from these real-time measurements on the components while the machine is operating, allowing for instantaneous functional diagnosis. DAQRI implemented a similar solution to help engineers at KSP Steel to visualize data from heavy machinery directly on the shop floor.
3. Data-driven job documentation and quality assurance:  Job documentation as well as product certification and testing usually involve long procedures during which operators test structural and functional variables of the equipment. These tests are then documented in lengthy manually written reports that are sent to a central database to serve as the basis for certification and quality assessment. The whole process can be made faster and more accurate using AR devices; the operator goes through the procedure in a step-by-step fashion, approving or rejecting the measurements taken using IoT-enabled equipment. Using AR interfaces, measurements can be visualized on the component being tested and any anomaly can be reported using automatically generated non-conformance reports sent directly to the central database alongside the related IoT data coming from the machine itself or the measurement equipment.

4. Product design visualization:  during the process of designing electro-mechanical objects, testing prototypes is very important to identifying design flaws as early as possible. However, many of the objects of analysis during this process are variables not visible to the human eye that, after being measured through embedded sensors, are analyzed to provide feedback for the following design iterations. In some cases, AR can provide instantaneous visual feedback on these variables so that design teams can discuss the issues during the test phase and simultaneously tune the object settings at run-time, accelerating the decision-making process. This video presentation by PTC president Jim Heppelmann includes an example of how CAD tools and IoT can be combined with AR to provide real-time feedback on design choices for physical objects.

5. Smart urban infrastructure maintenance:  similar reasoning can be applied to the public sector. Most urban infrastructure is located outdoors and in hard-to-access areas, making embedded screens very difficult to use. Operators can use AR to scan large objects and detect the point of failure from real-time data visualizations. In addition, they can easily document the status of infrastructure in a digital, data-rich manner, just by pointing the device at the system.

6. Enhanced operator safety:  AR can also be used to provide safety information to operators interacting with machines that can cause physical harm if improperly handled. DAQRI shows how a thermal camera can be used not only to visualize a thermal map, but also to indicate to the operator where it is safe to touch the object. Although the technology used by DAQRI involves the use of a thermal camera mounted on a hard hat, the same result can be easily obtained using thermal (and other types of) sensors installed directly on the machine to inform the operator of potential hazards.

The challenges

Despite being a suitable solution for the unsolved problems of IoT data consumption on the shop floor, AR still provides challenges that AR providers are currently working on in to make it more practical and useful in real life scenarios.

The first challenge is related to the way IoT data is displayed using AR devices. As mentioned earlier, sensor data can be displayed in new, intuitive modalities using bespoke 3D visualizations, facilitating the decision-making process on-site. However, it is difficult to automatically create and scale up this type of visualization. Providers are working on systems that integrate 3D CAD models with IoT real-time data to automatically generate “datafied” 3D models that can overlay on top of physical objects to display extra layers of information.

In addition to this, the problem of visualizing multiple data points in one single visual entity is still an open issue. While there are consolidated methods that work for traditional displays (like sub-menus or scrollable areas), UI/UX designers are currently working on techniques to condense large amount of data and make it interactive using AR displays.

Another important challenge has to do with data security and integration. As operators are performing their jobs with mobile-connected AR devices that access sensitive data, providers must be sure that these devices are not vulnerable to threats using both software and hardware security protocols. The AREA has recently issued a Request for Research Proposals to members in order to foster an investigation into the issue and propose some solutions.

The future

IoT data is currently used mostly for offline processing. Many techniques allow the creation of very accurate mathematical models of the production line that enable not only cost reduction and production optimization, but also predictions of equipment performances. However, the value of this data resides also in its real-time consumption. The valuable insights generated from the real-time information produced by machines and equipment can greatly accelerate many procedures and integrate human labor even further into industrial information systems. Not taking advantage of this side of IoT means a partial waste of the deployment investment.

AR is considered one of the best tools for workers and engineers to access real-time IoT data on the shop floor, directly where it is needed. AR devices are aware of the spatial configuration of the environment around the worker and can intuitively visualize real-time data, filtering out unnecessary information. As these devices get smaller and lighter, the number of use cases to which this combination of technologies can be applied is growing rapidly, covering scenarios that could not be addressed before.

Eventually, the convergence of AR and IIoT will empower human operators with greater efficacy and will add to their skills in a knowledge-intensive working environment. With the advent of fully integrated automatization and robotics, AR provides a great opportunity for workers to retain the indisputable value of human labor and decision-making.

What the AREA is doing

The AREA is a great supporter of the integration of AR with the rest of the Industry 4.0 technologies. For this reason the AREA recently partnered with the Digital Manufacturing and Design Innovation Institute (DMDII) for a two-day workshop on AR requirement for Digital Manufacturing. The result of this workshop – a list of hardware and software requirements for the introduction of AR technology in the factory of the future – will guide both providers and users towards efficient AR adoption.