AREA: Are you an AR guy who got into safety, or a safety guy who got into AR?

Dr. Garrett: It’s the latter. In 2017, I was supporting the Boeing 767 tanker program when a couple of colleagues approached us in the Safety organization looking for safety and ergonomics guidelines on an Augmented Reality project using HoloLens for wiring work. We looked at each other and said, “What’s a HoloLens?” (laughs) I did some looking around and I couldn’t find any research on the safety ramifications of AR. I finally landed on some ergonomic recommendations for helicopter pilots using night vision goggles. That was the closest thing I could find, but at least it was a starting point. I put some recommendations together and very quickly became the subject matter expert for AR safety.

AREA: It sounds like everybody involved in studying safety requirements in enterprise AR has had to learn as they go along.

Dr. Garrett: It has been a very hands-on learning experience, but the technology is still a hands-on learning experience in a lot of ways. And as we’ve gone along, my interest has been pushed more into fully immersive technologies, not just the AR space. Once I became known as the AR guy, people started coming to me and asking me to help them with their VR projects. So that’s become part of my work now.

AREA: What is the AREA Safety Committee focused on right now?

Dr. Garrett: The past few years have been largely project-focused. There was the AREA Safety and Human Factors Assessment Framework and Best Practice Report. Things have changed a lot since that was published, so we’ll be doing a refresh of it. And then we put together the AREA Safety Infographic. We’ve now moved into the development of a playbook of sorts, a general guide to things to be aware of when you’re implement AR solutions from a safety perspective. What kind of infrastructure do you need? What kind of issues should you be aware of? How should you assess the environment? We’ve also brought in outside experts from academia and industry to provide their viewpoints and lessons learned. For example, at our next meeting in November, the CEO of Design Interactive will present some of the things they’ve been working on from a product design perspective, but also some of the research they’ve been involved in with their customers on usage requirements. We’ll be learning about the impact they’re beginning to see on the individuals who use AR.

AREA: What are the top AR safety issues that people are concerned about?

Dr. Garrett: Situational awareness is a big one. The restricted field of view. These are of particular concern in environments that have potential hazards. If you’re interacting with the system, you may not hear emergency or other messaging going on in your area. And with a restricted field of view, you might trip over something or bump into someone. Those are probably the top two. Cyber sickness is not generally a concern with AR, but we are starting to see some research that there are some impacts among those who are exposed for two hours or more. There is a correlation between the amount of usage and how much downtime you should have. As that research continues, we’ll be able to develop some requirements to address that issue.

AREA: What can we look forward to from the AREA Safety Committee in the near future?

Dr. Garrett: Last year, we entered into a partnership with the National Safety Council. We’re going to be working with them on the further refinement of the framework tool. It will give new AR adopters a checklist whereby they answer a series of yes/no questions to evaluate the job or their work environment from a safety perspective. In addition to the AREA sharing that framework tool with the AR ecosystem, the National Safety Council will be able to share it with their membership. We’re currently waiting for the NSC to arrange the resourcing of that work, but I expect we’ll see that completed next year.

AREA: Why should AREA members consider joining the Safety Committee?

Dr. Garrett: It’s really about having a voice and a say as to what content is being delivered to protect all employees. International standards are another area where we need a lot of support. There are standards development efforts underway right now at Underwriters Laboratories, IEEE, and ISO, and we need AR users to be represented in the room. There’s a lot of manufacturers and academics involved, but not enough AR customers, and their voices need to be heard.

If you’re an AREA member and would like more information about joining the AREA Safety Committee, contact Dr. Greg Garrett or AREA Executive Director Mark Sage. If you’re not yet an AREA member but care about ensuring safety in enterprise AR, please consider joining; you can find member information here.

• Magic Leap raised US$500 million off a US$2 billion valuation and unveiled its Magic Leap 2 AR headset that is set for release in 2022

• Chief executive Peggy Johnson said the headset would be the industry’s “smallest and lightest device built for enterprise adoption”

• Much like Microsoft’s Hololens, the goal for this headset is to help remote workers connect and train away from physical office

When Magic Leap was founded 11 years ago, the company set out to be a pioneer in augmented reality and mixed reality technologies. It even received almost US$3 billion to fund its first consumer-friendly AR headset, the Magic Leap One, which was launched in 2018 after a long delay. The US$2,300-priced headset eventually flopped, having sold only 6,000 units — a figure far removed from the one-million sales goal set initially.

The startup eventually narrowed its focus to professional applications, tried unsuccessfully to sell the company, and fired more than half of its workforce during the challenging economic climate of 2020. Plans to make mixed reality glasses mainstream were pushed back.

Amidst this whirlwind of shifting expectations, co-founder and CEO Rony Abovitz decided to leave the company in July 2020. Replaced by Peggy Johnson, the company then unveiled the Magic Leap 2, dubbing it as the industry’s smallest and lightest device built for enterprise, “designed to increase business adoption of AR.”

Johnson, formerly with Microsoft, revealed the new headset during a CNBC interview and in a blog post this past week. In a sign of investor confidence in the burgeoning enterprise AR space, Magic Leap further announced that it has raised US$500 million in funding at a post-money valuation of roughly US$2 billion. “The new capital will further Magic Leap’s focus on delivering best-in-class AR solutions including the roll-out of its second-generation product, Magic Leap 2, in 2022.”

Additionally, as claimed by Johnson, “this more advanced headset boasts critical updates that make it more immersive and even more comfortable, with leading optics, the largest field of view in the industry, and dimming – a first-to-market innovation that enables the headset to be used in brightly lit settings, in addition to a significantly smaller and lighter form factor.”

The short article mentions a number of providers, some of whom are AREA members one of which is Microsoft’s HoloLens 2. The HoloLens 2 smartglasses can connect to remote experts, holographically overlay patient data, and consult MRI images in 3D. They combine existing features—such as high-resolution images, better performance, and eye-tracking—to deliver an interesting full package.

In conjunction with Case Western Reserve University and the Cleveland Clinic, Microsoft has developed a HoloLens app, HoloAnatomy, that allows medical students to see every aspect of the human body, from muscles to veins, in 3D on a dynamic holographic model. Microsoft is not alone in this space.

Facebook has the wireless Oculus Quest 2 VR headset, just one of multiple types that Oculus is producing. As more technology and applications become available, exciting new opportunities await healthcare professionals.

Accuvein uses projection-based AR in its handheld device that illuminates peripheral veins on the skin’s surface with the goal of improving venipuncture.

Amazon and Intel are getting into this space, as is a Chinese company called Nreal (a former AREA member) which is developing mixed reality technology in the 5G era.

A mixed reality viewer from Brainlab, whose software and hardware create and enhance data, aims to improve critical surgeries and make the operating room more efficient.

The article does state that “down the road, AR healthcare apps could help surgeons become more efficient at surgeries”, however, that is very much a reality already, as many customers are already using Augmented Reality in surgery, such as AREA member Vuzix whose smartglasses have been used in surgery. See News Pixee Medical Surgery Solution Has Received Clearance to Enter U.S. Knee Surgery Market with Vuzix AR M400 Smart Glasses

“The worldwide AR and VR market will experience continued growth, thanks to a slowly warming reception and an increase in the number of new devices arriving to market,” says Ramon Llamas, research director, IDC’s Augmented and Virtual Reality team.

Through this partnership, TeamViewer and Google Cloud are co-developing and co-marketing enterprise AR solutions built on Google Cloud to better serve customer needs.

The first jointly developed solution ‘Assisted Order Picking’ is a hands-free order picking application that leverages Google Glass Enterprise Edition 2 smart glasses equipped with TeamViewer’s vision picking software from its Frontline suite, focusing on improving omnichannel fulfillment for grocery stores and retailers.

Carrie Tharp, VP Retail & Consumer at Google Cloud: “Shopping behavior has dramatically changed, especially in the current pandemic, and retailers are having to hire new associates while rethinking their picking and fulfillment processes.  Retailers are looking for off-the-shelf technology they can implement today that will also prepare them for long term success as they develop new approaches to the modern store.”

“We are excited to introduce this first solution with Google Cloud as retailers and grocers struggle to adjust to omnichannel shopping trends.  Our goal is to provide them the tools and technology to drive a higher level of service to their customers and to streamline their ever-changing business models,” says Alfredo Patron, executive vice president of business development at TeamViewer.  “As a Google Cloud partner, we are now developing solutions to improve both the front and back-end of retail operations as a first step.  In the very near future, we are looking to extend our partnership and apply this same technology stack to improve manufacturing, field service and supply chain processes within other industries.”

Connected to a retailers’ or grocers’ order fulfillment systems to update inventory in real-time, associates receive the information they need to fulfill the orders within the display of Google Glass.  This heads-up display capability enables associates to use both hands to locate and pick the correct items, improving pick rates anywhere from 15-40 percent.  At the same time, it provides new data insights to retailers servicing the increasing demand from customers for buy-online and pick-up in store (BOPIS) and same-day delivery options.

This announcement builds on a partnership initiated with Upskill, a pioneer in industrial AR solutions, that was acquired by TeamViewer in March 2021.

For an animated explanation of how the solution works, check out this video.

CMPC is a multinational corporation that produces and markets wood, pulp, packaging products and household and personal use products. “We implemented the use of Vuzix Smart Glasses in our company because we cannot travel sporadically to the more than 40 countries where we sell our products.

The objective of implementing this cutting-edge technology is to have greater access to our clients and take immediate resolutions,” explained Pedro Silva, Cardboard Technical Assistance Supervisor of CMPC’s Technical Deputy Manager.

Statkraft is an international energy producer of hydropower, wind power, solar power and gas-fired power and supplier of district heating. “Our plant is 45 minutes away, and each specialist has to constantly be traveling to operations.

Benefits of Smart glasses

With the implementation of Vuzix Smart Glasses in our facilities, we were able to realize a reduction of trips and travel costs. Recently, we were able to remotely obtain the advice of an expert in Brazil thanks to these glasses,” stated Mirko Ojeda, Dispatch Coordinator at Statkraft Chile.

Read more at https://www.vuzix.eu/News

Utilities can invest a million dollars and more to train an individual operator, over 18 months and more. The goal is exceptional competence and procedural adherence, and a solid knowledge of plant design and theory. But even with the best training, there is no replacement for operators with hands-on experience, who possess a level of knowledge unobtainable in a classroom or training programme. Utilities need put this invaluable knowledge to work.

Simultaneously, energy companies need to upskill frontline veterans. Additional skillsets are frequently at technology’s cutting-edge and it is essential to provide ongoing and regular training to operators.

Technological advances are providing innovative ways to tap into the knowledge and skillsets of experienced workers and train new generations of operators on core procedures and new protocols. Specifically, advanced augmented reality (AR) systems that enable nuclear facility operators to keep fully up to speed on best practices, and empower them to access expert instruction on rarely used procedures when necessary.

The role of AR can be tied to a SMART approach to operation fundamentals:

• Solid knowledge – Does what you’re doing deliver powerful, actionable knowledge to the individual, and empower them to make the right decision at the right time?
• Monitor – Do you have the ability to monitor the facility effectively?
• Act with a conservative bias – Are you installing technology for technology’s sake? Do you need it? Does it help?
• Rigorous control – Are you considering all the risks that are involved, both the physical risks and the cyber risks? Are you taking proactive steps to control and manage these risks?
• Teamwork – Have you considered more than just the obvious users? Account for technologically-advanced operators, as well as team members who may prefer pen and paper. Make the technology easy to use and indispensable for all individuals.

Capturing valuable organisational knowledge

Unique information — equipment-specific maintenance and repair procedures, system insights, insights into available organisational resources, even awareness of management style and company culture  is invaluable company-specific knowledge.

This unique knowledge set is supplemented by new and rapidly expanding SCADA/ industrial internet of things (IIoT) and related data, provided by a new generation of sensors. Sensors are being incorporated into legacy equipment, as well as being standard on new equipment.

Advanced AR systems can address the need for ‘Solid knowledge’ (above), capturing utility-specific knowledge and leveraging system data, turning it into procedures all frontline staff can use. The result is improved operational efficiency, increased safety, and consistent completion of complex tasks, which raises employee productivity and allows utilities to operate safely and profitably.

Deliver real-time access to authored content

With instruction provided through AR, operators can walk up to a piece of equipment and follow step-by-step instructions to get the job done using a phone or tablet, or a headset when hands-free operation is needed. Performing complex tasks consistently and efficiently is a critical goal.

The ability for experts to author content themselves is essential, especially when it comes to communicating procedures for infrequent maintenance tasks such as reactor head venting.

With this need in mind, AR systems such as Manifest from Taqtile are designed to be intuitive. Knowledge experts are able to create step-by-step digital work instructions overlaid on facility equipment, which other operators view via their device or headset. The author can also make available supporting materials assisting in the repair or maintenance process, such as photographs, videos, PDFs, or manuals and documents, delivered in a hands-free heads-up display when needed.

Real-time access to content, ranging from step-by-step video instruction walking frontline staff through complex procedures to detailed holograms, will increase AR’s value. And when needed, live remote assist from the facility’s operations centre, utility headquarters or equipment OEMs, can be made available.

To assist in auditing, capturing evidence — a picture or video showing the proper performance, or visuals demonstrating completion of a task — is critical. Capturing evidence also helps orchestrate operations across teams, where one operator may be located at a piece of equipment, another at an electrical panel and a third in the control room. Each has their specific task in the executing a complex procedure, and AR platforms can help guide each individual while keeping other team members apprised of status and progress in real time.

The ability to enable two-way video, real-time remote guidance, and on-site access to repositories of knowledge, helps remove ambiguity from the communication process and empowers workers to complete complex tasks at the machine.

Leverage the power of 5G

Nuclear facilities are considering adoption of 5G, driven by the edge-computing capabilities, support for remote operations and improved security and reliability. With 5G, the capabilities and value of AR applications increase, as 5G makes data more portable and AR systems faster, more secure and more resilient.

Maximising expertise within organisations is also a supported by AR and optimised by 5G. AR platforms can capture and disseminate knowledge from experts and distribute it to other staff, regardless of location, making employees more productive more quickly, with reduced classroom time.

The integration of AR with IIoT technologies will be greatly enhanced by 5G and the incorporation of sensors into legacy equipment. It will provide transparent information on demand, right at the location of frontline workers, enabling them to directly and securely access critical data. 5G also reduces latency, increases throughput and enables the deployment and connection of more sensors, providing more data for machine learning to help inform maintenance tasks.

Meanwhile advances in measures for device authentication, encryption of user traffic between devices and the network, and additional security protocols provide a new level of security required by utilities.

The Manifest AR solution

The Manifest platform is an end-to-end augmented work instruction platform that allows nuclear utilities to capture knowledge that enables interactive work instructions, and deliver ‘over-the-shoulder’ support when needed.

The platform functions like an advanced help centre for frontline workers. Benefits include increased staff efficiency and accuracy during complex operations and maintenance tasks.

Real-time interactions between remote experts and field operators have traditionally been limited to simple audio communication. Manifest changes that by fully leveraging the power of AR, sharing the complete experience of the onsite operator with the remote expert through enhanced AR communication on most popular AR-enabled devices – HoloLens, Magic Leap, iPad, Trimble XR10, and Android phones. For example, an expert working from a PC or tablet in a remote location can follow the operator through a first-person view of a valve line up verification in containment, seeing precisely where operators are looking with tracked eye movement. This capability mitigates the risk of mispositions and human error. By tracking the operator’s view, a remote expert is able to guide the operator with real-time instruction, including annotating directly in the operator’s environment, using directional arrows or highlights to guide the operator.

Help-centre-like interactions available via the Manifest platform between remote expert and frontline worker enhance the capabilities and efficiency of all staff:

• Virtual ink – Experts can draw and place targets over the operator’s field of view, and incorporate spatial audio, producing sounds that emanate from a position within the operator’s environment for easier location finding.
• Hologram sharing – In addition to the video feed delivered from the headset, operators can share holograms, including any Manifest user interface or 3D models in their view.
• Wayfinding – Remote experts can render holographic directional arrows in the operator’s field of view to guide them within the onsite environment.
• Eye tracking – By tracking the operator’s eye position and projecting a directional image to the expert’s desktop, the expert can follow what the operator is looking at, enabling more accurate remote assistance.

For example, Taqtile industrial customers have experienced remarkable improvements in trainee performance, with some companies decreasing training times by 700 percent. Simultaneously, it has helped increase accuracy, reducing operator errors to zero in some cases.

Manifest can also capture evidence and operational data as part of regular inspections and preventative maintenance, supporting regulatory requirements and meeting audit and related oversight needs.

From improving training performance to upskilling frontline workers on new technologies, AR will enable utilities to capture the knowledge of their most seasoned operators and share that knowledge in-real-time across the organisation.

Reinventing Intelligence Operations for the British Army

Masters of Pie was proud to showcase their work on the Digital Bird Table solution for the British Army at DSEI in London. The Digital Bird Table solution complements Project Hydra, a project about collating multiple data feeds and processing this information so that Command Centres are able to make better and more timely decisions.

The Masters of Pie’s software (Radical) has been integrated directly into multiple data sources such as SAP HANA and ESRI ArcGIS. The Radical real-time collaboration server pulls the Digital Bird Table experience together, synchronising Hololens and desktop users. The time-lapse video highlights how the solution was polling live data and tracking assets on the move. Watch Video

XR Collaboration for CAD/PLM and BIM Data

The XR collaboration for manufacturing and engineering consisted of a live demonstration of the Radical software integration into the CAD software application to enable the seamless transition to a single user or collaborative VR experience without the need for exporting complex and sensitive data into a standalone application. The solution is ideal for remote assistance, remote diagnostics, design review, design to manufacturing handover, design to supplier engagements, and engineering training purposes. Watch Video

DSEI was a great opportunity to showcase the Radical’s software ability to deliver such contrasting use cases, integrate into a wide array of data sources, deliver multiple end-user experiences (Hololens (AR), VR, and desktop), and support a range of deployment options (cloud, on-premise and stand-alone). Radical is the future of collaboration delivering any data securely to any device.

When the stakes are high, the tools you use to communicate your point or value proposition will matter the most: Augmented reality has transformed how global engineering innovator Danfoss Climate Solutions has approached product presentations and trainings for its internal stakeholders and customers. The company uses the AR software AR 3S to visualize complex products in 3D to provide a captivating experience of learning and understanding.

Choosing the Right Augmented Reality Software and Partner~Keeping in line with the company’s DNA of “engineering tomorrow”, Danfoss Climate Solutions was looking to drive digital transformation by implementing augmented reality into its processes.

The company reached out to various AR suppliers, seeking high flexibility and a strong support culture. “We chose Holo-Light as our entryway to augmented reality engineering. It was easy to discuss use cases and solutions,” remarked Kabileshkumar Cheetancheri, Senior Project Manager at Danfoss.

Holo-Light’s software AR3S (Augmented Reality Engineering Space) is a new tool to improve and simplify workflows in planning, development, and presentation. It enables users to visualize, manipulate, and work collaboratively on 3D designs. “One part I was always interested in, is how we can add a new dimension to interacting with and explaining complex products – internally and externally for our customers,” said Cheetancheri. The product presentation and training use case emerged quickly as the AR software showed itself to be very easy and intuitive in use. With AR3S, Danfoss Climate Solutions was able to add real value to internal product demos with company stakeholders and help customers have a better understanding of its products.

Full original article on Holo-Light website

Find out more about Holo-Light on their AREA member profile 

The T2 Augmented Reality (T2AR) project demonstrates how station crew members can inspect and maintain scientific and exercise equipment critical to maintaining crew health and achieving research goals without assistance from ground teams.

To kick off the T2AR activities in orbit in April, astronaut Soichi Noguchi of the Japan Aerospace Exploration Agency (JAXA) was tasked with maintenance for one of the space station crew’s pieces of exercise equipment, the T2 Treadmill. The inspection procedure is typically available as a PDF document to be accessed on a computer or tablet, which can be hard to hold while also operating tools or flashlights or examining equipment in a tight space. This time, no extra handheld instructions or communication with ground teams in Mission Control were necessary since the information was all in plain sight. Using the Microsoft HoloLens augmented reality (AR) goggles and armed with novel procedure tracking software NASA developed, Noguchi had step-by-step guidance and cues to assist in the work without referring to a separate screen.

T2AR is the first in-space operational use of the HoloLens in combination with custom-built AR software, which enables an astronaut to perform unassisted maintenance and inspections on a major piece of crew support hardware. This investigation builds on the Sidekick experiment former NASA astronaut Scott Kelly conducted in 2016.

This novel demonstration used 3-D directional cues to direct the astronaut’s gaze to the proper work sites and displayed the procedure instructions. The device followed an astronaut’s verbal instructions to navigate procedures and displayed AR cues and procedure text over the hardware as appropriate for the procedure step being performed. The system also provided supplemental information, such as instructional videos and system overlays, to assist in performing the procedure.

“AR tools hold the promise of allowing us to pre-package guidance and expertise,” says International Space Station associate scientist Bryan Dansberry at Johnson. “The space station is the perfect platform to test out AR systems and refine these tools so they will be ready when future astronauts need them. Closer to home, these tests help to mature software and AR technology now so expertise and support are available in remote locations around the world.”

NASA astronaut and Expedition 65 Flight Engineer Megan McArthur wears the specialized Sidekick headset and tests using augmented reality aboard the International Space Station. Credit: NASA

Since that first activity with Noguchi, astronaut Thomas Pesquet of ESA (European Space Agency) and NASA astronaut Megan McArthur have also used the AR application aboard the space station. The tests provided researchers with information about how the technology does and does not work to assist crew inspection maintenance procedures. With the completion of this maintenance activity, nine more test sessions remain in the technology demonstration plan.

While this demonstration was currently limited to the critical T2 Treadmill, the platform is designed to be used across a wide variety of space station needs in the future, leading to increased crew efficiency and activity execution accuracy. In addition to use on the space station, this technology could aid astronauts on future journeys to the Moon and Mars, when communication delays are greater, by allowing them to conduct tasks without waiting for further direction from Earth.

NASA’s Advanced Exploration Systems division sponsors this technology demonstration aboard the microgravity laboratory to advance future human and robotic exploration missions as part of NASA’s Moon and Mars exploration approach.

Situation

To maximize uptime of its customers’ heavy trucks and other commercial vehicles, Renault Trucks needed a more effective way to remotely support a network of 72 service outlets across the UK & Ireland. They needed collaborative video technology that would be easy to use, deploy, scale, simplify training and quality control processes, and supply a safe, secure connection even in low bandwidth environments.

Solution

Using Librestream’s Onsight augmented reality software and HMT-1 hands-free headsets from RealWear, Renault Trucks developed Optiview, a remote support platform that has since been deployed throughout the UK and Ireland. The system allows licensed service centers to virtually connect with the OEM’s support desk or even with other service centers.   

Results

Since implementation, there is less need to send Renault Trucks representatives onsite. In-person warranty part inspections have gone down by a factor of five, resulting in 10 to 15% fewer miles driven and a corresponding reduction in greenhouse gases. Warranty claims and payments are much faster, customer satisfaction has improved, and asset uptime is significantly higher—for example, complex service procedures that once took 24 to 48 hours can often be performed in just 20 to 30 minutes, greatly increasing uptime value for vehicle operators.  

Read more on Librestream use cases