Visualization of Data About or From Tools in AR Display

Combining live sensor data and tools with AR displays enables many use cases. This topic studies how to meet requirements of Industrial Internet of Things (IIoT) use cases in new and intuitive ways. In the IIoT, and as part of Industry 4.0, machines and tools run software and may be connected to corporate infrastructure with cables or radios. Frequently, there are consoles on which a user receives tool status and diagnostics. Sensors may be integrated or attached to a tool, thereby enhancing the tool’s capability. A tool without direct connection to enterprise IT may be detected with the AR system or paired with a user’s AR device. Data from tools (or from a technician’s AR system) may be sent to work order management systems, SCADA, and asset tracing and tool use or management systems (e.g., the torque on a screw needs to be increased or decreased).

When an AR-assisted user is controlling connected tools (or interacting with unconnected tools) to perform a task, they do not need to turn their attention away from the main task to receive useful information. However, it is critical that the information provided in AR does not interfere with the user’s awareness of the tool and is of value to the user. There is the potential for back-end IIoT systems to contribute to the user’s productivity by providing clear indicators of status.

This research topic explores and studies the usability of different approaches to visualize data from or about tools (that may be generated by a user’s own device) in an IIoT context. The research could also include making, with AR device pairing, those tools without sensors or that are not connected, more detectable and/or intelligent. The user may also have the ability to receive and to see in their AR display assistance from a remote tool or system expert. The research can also examine integration of data from tools and users’ devices with operational software to automatically track processes, apply quality control, and access recommendations for users and managers.

Stakeholders

Operators of manufacturing, repair, maintenance facilities, quality managers, managers of factories, providers of repair and maintenance services, oil and gas, power and energy, medical practitioners, experience designers, enterprise IT, systems integrators specializing in IIoT or Industry 4.0

Possible Methodologies

An experimental environment using a range of connected or smart tools and procedures will provide a suitable environment for this research. Users of Industrial Internet of Things (IIoT) or Industry 4.0 systems will perform tasks under controlled conditions. Motion and time studies, quality inspection, and other measurements of performance will be used to quantify impacts. Users will complete surveys about satisfaction and feedback on various visualizations.

Research Program

This research topic is broad in scope. It can include exploring different approaches to visualize data about tools to better understand and document or compare usability. This would benefit the industry by establishing Industry 4.0 best practices or guidelines. The research could also include making, with AR device pairing, those tools without sensors or that are not connected, more detectable and/or intelligent. This would expand the types and number of existing tools that would be tracked, without the cost of replacing tools that were developed prior to Industry 4.0 adoption. It also can examine the integration of data from tools and users’ devices with operational software to automatically track processes and provide remote assistance, quality control, and recommendations for users and managers. Further, this topic can be combined with studies of AR-enabled guidance, integration of AR with IoT, finding parts and supplies in a large space with AR, 3D user interfaces, live sensor visualization for other use cases, and other use cases in factory or field settings.

Miscellaneous Notes

Two AREA members with financial support from both industry and government [MxD (in US), the Manufacturing Technology Center (MTC), and the AMRC (in UK)] have the required experimental environments and have already begun studying the approaches in this topic. In December 2014, there was on the Plex web site a blog post about the specific use case of a connected torque wrench.

Keywords

Connected tools, manufacturing, AR display, industrial Internet of Things (IIoT), SCADA, asset tracing, asset management, maintenance, diagnostics, usability, perception, real.

Research Agenda Categories

Business, Technology, Use Cases, Industries

Expected Impact Timeframe

Medium

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Norman, M., Lee, G., Smith, R. T., & Billinqhurst, M. (2019). A Mixed Presence Collaborative Mixed Reality System. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)
• Cavallo, M., & Forbes, A. G. (2019). CAVE-AR: A VR Authoring System to Interactively Design, Simulate, and Debug Multi-user AR Experiences. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)
• Hoffmann, C., Buettner, S., Prilla, M., & Wundram, K. (2020). Impact of augmented reality guidance for car repairs on novice users of AR. Proceedings of the Conference on Mensch Und Computer
• Chen, Z., Su, Y., Wang, Y., Wang, Q., Qu, H., & Wu, Y. (2020). MARVisT: Authoring Glyph-Based Visualization in Mobile Augmented Reality. IEEE Transactions on Visualization and Computer Graphics, 26(8), 2645-2658
• Nebeling, M., & Speicher, M. (2018). The Trouble with Augmented Reality/Virtual Reality Authoring Tools. 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)

More publications can be explored using the AREA FindAR research tool.

Author

Christine Perey

Last Published (yyyy-mm-dd)

2021/8/31

Go to Enterprise AR Research Topic Interactive Dashboard