In many industries, workplaces contain a plethora of hazards. When known or anticipated, hazard management protocols reduce the risks associated with a user’s encountering a hazard when performing tasks or fulfilling a work order. However, there are also hazards which, even if made aware of them, the user is untrained to treat or has insufficient time to avoid or deescalate. Alerts can provide the user time to react. On the other hand, there may be hazards that do not require any specific user actions.

Through data collected by user location sensing technologies on devices, and potentially on users’ PPE, as well as maps of known hazards, data generated from sensors on stationary or moving machines, and other methods, artificial intelligence algorithms could be used to continuously maintain and monitor a dynamic 3D map of hazards in a user’s proximity. The user may be provided the hazard proximity map at intervals or request to visualize hazards in proximity. When the user reaches conditions with respect to the hazard that suggests appropriate actions are needed, an alert on an AR device can spatially anchor the source of risk or hazard in the user’s perception (see Automated Alert to Dangerous Settings [[ra-Salert5-dangerosity]]). If and when needed, guidance for risk mitigation can be provided.

Stakeholders

Safety managers, workplace designers, risk managers

Possible Methodologies

This research can include studying appropriate definitions of proximity and risk in diverse industries and workplaces and/or using existing risk management tools, capturing data sets and training algorithms for types of hazards and testing reliability of AI in diverse conditions. The study of user interface and user experience for hazard notification systems will contribute to this field. Further, user studies will be required to measure cognitive load and user responsiveness to notifications of hazardous or potentially hazardous circumstances.

Research Program

The scope of this research can span many industries and workplaces. It could be tailored to any industry in which AR is introduced and demonstrated with many workplace use cases. It is closely related to other proposed topic concerned with Automated Alert to Dangerous Settings, and a topic focusing on dangers due to chemical or radiation in employee vicinity. It could be combined with research on visualization of IoT data streams, 3D maps of known risk and other safety management programs. Further, it also can include or be an extension of numerous 3D user interface and user experience topics.

Miscellaneous Notes

This peer-reviewed article was published in December 2015 entitled “Proximity hazard indicator for workers-on-foot near-miss interactions with construction equipment and geo-referenced hazard areas” pertains to the topic of this research.

Keywords

Hazard detection, hazard management protocol, hazard warning, location-detection, 3D spatial mapping, artificial intelligence, user interface, user experience, risk assessment, risk management, situational awareness, occupational risks, risk assessment, risk perception, accidents, occupational health, occupational safety, safety, health and safety, health hazards, safety devices, safety factor, safety systems, fault detection, monitoring, system monitoring

Research Agenda Categories

Industries, Technology, Business

Expected Impact Timeframe

Near

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Aromaa, S., Väätänen, A., Kaasinen, E., Uimonen, M., & Siltanen, S. (2018). Human Factors and Ergonomics Evaluation of a Tablet Based Augmented Reality System in Maintenance Work. Proceedings of the 22nd International Academic Mindtrek Conference.
• Lutz, R. R. (2018). Safe-AR: Reducing Risk While Augmenting Reality. 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE).
• Mauricio Gomes de S Ribeiro; Isabel Lafuente Mazuecos; Fabiano Marinho; Alice Neves Gomes dos Santos.. Agile Explorations in AR. IECON 2019 – 45th Annual Conference of the IEEE Industrial Electronics Society.
• Schmidt, S., Steinicke, F., Irlitti, A., & Thomas, B. H. (2018). Floor-Projected Guidance Cues for Collaborative Exploration of Spatial Augmented Reality Setups. Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces.
• Miyazaki, M., & Komuro, T. (2018). Extended Workspace Using a Smartphone with a Depth Camera. 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

More publications can be explored using the AREA FindAR research tool.

Author

Christine Perey

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard

New AR display devices encounter significant resistance from enterprise IT teams who consider the new hardware platforms increase security threats, consequently, increasing the need for an elevated security posture.

Driven by a human-centric approach, a critical step in ensuring compliance with existing security policies and systems is to balance the security with accurate and rapid user authentication and ultra-low-friction user input.

Biometric identification methods, ranging from palm-prints, voice-prints, iris scanning, gait to heartbeat detection offer a plethora of opportunities for identification of wearable enterprise AR device users before providing access to enterprise work orders and data.

This research topic compares different modalities of biometric identification and classifies them based on accuracy, cost and ease-of-use.

Stakeholders

All stakeholder in corporate security organizations but primarily CISOs, CIOs, IT and security managers. On the vendor side, OEMs, solutions providers, system integrators and independent software vendors will be impacted by this research.

Possible Methodologies

This research will require rigorous laboratory tests, deployed via multiple cells of different modalities. This will be followed by human factors and security research, culminating in field trials. Once baselines are available and validated, best practices can be established.

Research Program

This topic is closely related to another proposed AREA Research Agenda topic on cleaning and authenticating multi-user devices end user [ra-Tsecurity5-multiuserdisplays]. The topics could be combined with other AR security topics to develop a broader research program. In addition, the topic could be expanded to use the sensors on devices of other AR users in a workplace to confirm user identities.

Miscellaneous Notes

The field of biometric authentication in cybersecurity is vast and there are many highly reputable research centers that could contribute to this research. Hundreds of publications appear each year in journals and proceedings. This paper describes results of studies to connect AR users with sensitive personal information derived from on-line platforms and use of these data to predict AR user interests and preferences. In the https://dl.acm.org/doi/proceedings/10.1145/3457339[proceedings of the 7th ACM on Cyber-Physical System Security Workshop] (May 2021) an article compiles recently published work on this topic and describes MoveAR. The goal of MoveAR is to distinguish between a legitimate user and potential adversaries based on the signatures detected by the on-device sensors as the user interacts with an augmented reality environment.

Keywords

Biometric, palm print, voice print, gait, retina scanning, iris scanning, heartbeat detection, skin conductivity, access control, data protection, security systems, authentication, message authentication, authorization, data security, access protocols

Research Agenda Categories

Displays, End User and User Experience, Technology

Expected Impact Timeframe

Medium

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Garae, J., Ko, R. K. L., Kho, J., Suwadi, S., Will, M. A., & Apperley, M. (2017). Visualizing the New Zealand Cyber Security Challenge for Attack Behaviors. 2017 IEEE Trustcom/BigDataSE/ICESS.
• Lebeck, K., Ruth, K., Kohno, T., & Roesner, F. (2018). Towards Security and Privacy for Multi-user Augmented Reality: Foundations with End Users. 2018 IEEE Symposium on Security and Privacy (SP).
• Wazir, W., Khattak, H. A., Almogren, A., Khan, M. A., & Ud Din, I. (2020). Doodle-Based Authentication Technique Using Augmented Reality. IEEE Access, 8, 4022-4034.
• Epstein, J. (2017). An NSF View of Multimedia Privacy and Security. Proceedings of the 2017 on Multimedia Privacy and Security.
• Shen, Y., Wen, H., Luo, C., Xu, W., Zhang, T., Hu, W., & Rus, D. (2019). GaitLock: Protect Virtual and Augmented Reality Headsets Using Gait. IEEE Transactions on Dependable and Secure Computing, 16(3), 484-497.

More publications can be explored using the AREA FindAR research tool.

Author

Peter Orban

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard

Stereoscopic vision has long been considered the best type of visualization in terms of matching physical and simulated realities. While stereoscopic vision is the goal, producing a perfect 3D visualization and registration of AR assets is difficult using current technologies. Addressing the shortcomings of current AR displays will be prohibitively high and present a financial barrier to AR adoption in enterprises.

Leveraging advancements in Digital Signal Processing (DSP) and audiology, a new class of devices are emerging. Spatially-aware audio transducers can help determine the exact position and posture/pose of the wearer as well as generate a simulated sound field that matches the physical environment. Such systems could be combined with existing vision-centric displays for high fidelity enterprise AR experiences.

The scope of this topic includes measurement of the spatial audio technology resource requirements and impacts of combining visual cues with spatial audio on user performance. Comparative studies of human cognitive performance aided by varying blends of spatial technology ranging from “audio-only” to “video-only” and various combinations of both are also in scope.

Stakeholders

AR experience designers, developers of integrated sensor and world capture components, human factors researchers

Possible Methodologies

This research topic will require development of visual and audio AR experiences to be produced in a highly controlled laboratory environment within which a series of experiments can be conducted and reproduced. Studies will compare spatial audio requirements to vision-only AR experiences on the basis of accuracy, speed, battery life, bandwidth requirements, processor performance, wearer comfort and pricing. In addition to user perception assessments through surveys and interviews, methods could be expanded to include time-motion studies using standardized, public and well-documented processes typical of industry verticals, use cases and horizontal use case categories.

Research Program

This topic is at the intersection of both 3D visualization and 3D audio. The methodologies and tools developed for this research could be used in the study of perception, presence, and lead to new guidelines for AR developers and manufacturers of HMDs for enterprise AR.

Miscellaneous Notes

In 2016, the Sound of Vision consortium, which focuses on the construction of a new prototype electronic travel aids for the blind published a report about audio-assisted vision. A peer-reviewed article presenting a novel technique for reproducing coherent audio visual images for multiple users, only wearing 3D glasses and without utilizing head tracking was published in 2011 in the Journal of The Audio Engineering Society.

Keywords

Spatial audio, effectiveness, spatial vision, 3D audio, perception, audio signal processing, acoustic waves, active noise control

Research Agenda Categories

Technology, End User and User Experience, Displays

Expected Impact Timeframe

Medium

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Kim, H., Remaggi, L., Jackson, P. J. B., & Hilton, A. (2019). Immersive Spatial Audio Reproduction for VR/AR Using Room Acoustic Modelling from 360° Images. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
• Heller, F., & Schöning, J. (2018). NavigaTone. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
• Schmalstieg, D. (2018). VIS Keynote Address : When Visualization Met Augmented Reality. 2018 IEEE Conference on Visual Analytics Science and Technology (VAST).
• Erkut, C., Holfelt, J., & Serafin, S. (2018). Mobile AR In and Out: Towards Delay-Based Modeling of Acoustic Scenes. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
• Abdelrazeq, A., Kohlschein, C., & Hees, F. (2019). A Cloud Based Augmented Reality Framework – Enabling User-Centered Interactive Systems Development. Advances in Intelligent Systems and Computing, 417-422.

More publications can be explored using the AREA FindAR research tool.

Author

Peter Orban, Christine Perey

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard

In terms of collaborative robotics, the widespread adoption of robots in historically manual manufacturing environments (which are subject to high product turnover, short production runs, and high variability in equipment configurations) is limited by the robots’ inability to effectively and safely integrate and interact with the existing human labor. Instead, so-called collaborative robots are relegated to secluded operations with minimal contact with the workforce. The robots’ inability to communicate with, understand the intention of, and establish a mutual understanding of the environment and situation with human coworkers decreases the robots’ usefulness in collaborative teams consisting of both robots and people. This limitation is driven by both the absence of tools and protocols needed for effectively describing and measuring human-robot interactions, an incomplete collection of metrics for assessing human-robot teaming performance, and insufficient protocols for enabling more intuitive interfacing with robotic tools. These challenges are compoudned when augmented reality technologies are used at the interface between the robotics and human workers.

This research topic focuses on providing the methods, protocols, and metrics necessary to evaluate the interactive and teaming capabilities of robot systems. It uses a task-driven decomposition of manufacturing processes to assess and assure the safety and effectiveness of human-robot collaborative teams.

Stakeholders

Manufacturers will benefit from the products generated as a result from this research project. Robotics providers can also benefit in that standard testing protocols for human-robot interaction will generate new sales tactics. End users will benefit in that the end state will be much safer in complex manufacturing environments.

Possible Methodologies

This collection of methods, protocols, and metrics will enable integrators and end-users to maximize the effectiveness and efficiency of collaborative human-robot teams in production processes, impacting both large-scale companies designing and repurposing hybrid manufacturing workflows, and smaller companies looking to begin introducing automated tools into manual processes.

Research Program

This research topic mirrors an existing project at NIST. Inspiration can be driven from the existing work generated by that team. Furthermore, IEEE is a leader in curating academic work in this area. Refer to IEEE RAS for related publication venues, including IEEE CASE, IEEE ICRA, and IEEE IROS.

Miscellaneous Notes

This topic requires significant hardware, middleware, and software integration. One open source framework is ROS-Industrial

Keywords

Robotics, human-robot interaction, human-computer interaction, remote monitoring, remote control, collaborative robots, autonomous agents, communication, computer vision, control systems, cooperative systems, grippers, human factors, human-robot interaction, industrial robots, industry 4.0, intelligent robots, multi-robot systems, occupational safety, robotics, safety

Research Agenda Categories

Standards, Technology, End User and User Experience

Expected Impact Timeframe

Long

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Muhammad, F., Hassan, A., Cleaver, A., & Sinapov, J. (2019). Creating a Shared Reality with Robots. 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). </li>
• Jana Jost; Thomas Kirks; Preity Gupta; Dennis Lünsch; Jonas Stenzel (2018). Safe Human-Robot-Interaction in Highly Flexible Warehouses using Augmented Reality and Heterogenous Fleet Management System. 2018 IEEE International Conference on Intelligence and Safety for Robotics (ISR).
• Bolano, G., Juelg, C., Roennau, A., & Dillmann, R. (2019). Transparent Robot Behavior Using Augmented Reality in Close Human-Robot Interaction. 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).
• Whitsell, B., & Artemiadis, P. (2017). Physical Human-Robot Interaction (pHRI) in 6 DOF With Asymmetric Cooperation. IEEE Access, 5, 10834-10845.
• Xue, C., Qiao, Y., & Murray, N. (2020). Enabling Human-Robot-Interaction for Remote Robotic Operation via Augmented Reality. 2020 IEEE 21st International Symposium on “A World of Wireless, Mobile and Multimedia Networks” (WoWMoM).

More publications can be explored using the AREA FindAR research tool.

Author

Bill Bernstein

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard

Significant AR experience development effort and time is dedicated to linking domain-specific data to AR engines and presentation systems. While many industries invest in AR scene development, the need is especially high in the manufacturing industry. Product Lifecycle Management (PLM) software systems curate design, manufacturing, and sustainment information related to a product system (ideally) throughout its entire lifecycle.

Leveraging such data across PLM systems for industrial AR has mainly become a platform-specific exercise. Commercial PLM software vendors provide industrial AR solutions that leverage their application programming interfaces (APIs) to help establish AR scenes. However, most large original equipment manufacturers (OEMs) operate in complex, global, and heavily distributed supply chains. In other words, in many cases, OEMs subscribe to every major commercial PLM software platform to handle the variety of data representations provided by their suppliers. This is especially important if the developers aim to truly create a digital twin of a product or production system.

This research topic will develop and test a standard data model that permits the vendor-neutral exchange of AR-critical data to produce updatable, sustainable, and maintainable AR scenes. One particular area of interest is understanding the proper handling of industrial animations. Though AR standards provide representations for presenting animations within AR scenes, for example, they do not directly relate to product system animations stored within PLM systems. Rather than relying on a particular PLM platform’s data representations, if such a data model was successfully developed, software developers would be able to exchange data across PLM software and to platform-agnostic AR engines more readily.

Stakeholders

OEM manufacturers, integrated solution and software developers, CAD/CAM providers, engineering design teams, PLM software publishers.

Possible Methodologies

This research topics could focus on testing existing PLM platforms and their AR-related competencies. Reporting on gaps bewteen the interfaces across related software tools will be a strong contribution. Existing standard data representations provide a strong starting point for investigation. Focusing on dealing with sustainment for digital twin models would cover lots of the use cases.

Research Program

OEMs with complex and heavily distributed supply chains should be a center point for the project. Program managers and technical advisors in such organizations understand the issue of cumbersome technical data packages. This research topic significantly overlaps with the Digital Model Interoperability research topic. [BB: issue with sentence grammar] There are distinct in that this research topic is specific to the manufacturing industry. The Digital Model Interoperability research topic relates more generally to 3D asset translation and can be applied to other domains, such as construction.

Miscellaneous Notes

There exist resources to help position research efforts. For example, the CAx Implementor Forum provides a number of test cases. The Khronos Group is continously updating glTF, the latest de-facto standard for lightweight model presentation. ASME Y14 is a working group that focuses on the standard presentaiton of GD&T annotations.

Keywords

Standards, data interoperability, digital twin, product lifecycle management, platform-agnostic AR solutions, application programming interfaces, distributed supply chains, manufacturing, sustainment, industrial Internet of Things, IIoT, ontologies (artificial intelligence), cost engineering, knowledge engineering, information management, project management, supply chains, team working, asset management, metadata, application programming interfaces, open source software, design engineering, systems engineering, cad/cam, quality management, supply chain management, manufacturing industries, product life cycle management, standardization, interoperability

Research Agenda Categories

Standards, Technology, Industries

Expected Impact Timeframe

Near

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Eckertz, D., Berssenbrügge, J., Anacker, H., & Dumitrescu, R. (2019). Work-in-Progress: Enhancing Collaboration Using Augmented Reality Design Reviews for Product Validation on the Example of Additive Manufacturing. Cyber-Physical Systems and Digital Twins, 244-254.
• Speicher, M., Hall, B. D., Yu, A., Zhang, B., Zhang, H., Nebeling, J., & Nebeling, M. (2018). XD-AR. Proceedings of the ACM on Human-Computer Interaction, 2(EICS), 1-24.
• Tobias Küster; Nils Masuch; Johannes Fahndrich; Gudrun Tschirner-Vinke; Jan Taschner; Markus Specker; Hendrik Iben. A Distributed Architecture for Modular and Dynamic Augmented Reality Processes. 2019 IEEE 17th International Conference on Industrial Informatics (INDIN).
• Bellalouna, F. (2020). Industrial Use Cases for Augmented Reality Application. 2020 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom).
• Zigart, T., & Schlund, S. (2020). Evaluation of Augmented Reality Technologies in Manufacturing – A Literature Review. Advances in Human Factors and Systems Interaction, 75-82.

More publications can be explored using the AREA FindAR research tool.

Author

Bill Bernstein

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard