Measuring Impacts of Enterprise Augmented Reality
The single most compelling reason to invest in enterprise Augmented Reality is to improve the productivity of people while on the job. Workplace performance is a broad concept that can be measured in many ways, and the impacts of a new user interface providing contextually sensitive information are new to most technology managers.
During the ARise ’15 conference, Matt Kammerait, Vice President of Products at DAQRI, presented some of the methodologies currently in use for gathering metrics and assessing the full impact of introducing Augmented Reality. This post builds upon experiences shared in the presentation and offers project leaders suggestions for how to quantify the impacts of enterprise AR.
Consider the Organizational Attitudes and Experience with Innovation
Before selecting the metrics for an AR project, consider the attitudes and experience of the various stakeholders in your organization. Aligning the project objectives with stakeholder goals and adapting available best practices or existing metrics from other projects can boost the chance of success. It will help when you want to expand the project beyond the pilot stage.
In parallel, those defining the parameters of an internal study need to take into account industry-wide metrics or constraints. Time savings is very important, but may not be the most important or best metric for all industries. In industries where workers’ lives are at risk, safety is almost surely a more important organizational metric. If an industry is heavily regulated, then compliance is likely to be at or near the top of the list.
It’s important to prioritize potentially valuable metrics. Take into account the time requirement and cost of studying each type of metric as an organizational constraint during the design phase.
If capturing detailed or complex metrics will increase the need for specialized staff and otherwise exceed the study resources, compromises based on the original set of priorities will need to be made.
Detect the Broad Patterns
Human-computer interfaces are key to the digital economy. Screens, keyboards, speakers, cameras and microphones provide faster and more accurate ways to acquire information or knowledge or, conversely, to quickly develop and communicate it. Knowledge workers build value with computers and networks by manipulating pixels that form meaningful symbols such as numbers, letters, lines and even virtual 3D objects.
In contrast to the tools in use by knowledge workers today, Augmented Reality is most useful when and where people interact with, or perform tasks with objects in the physical world. In the use cases that guide workers, Augmented Reality-assisted visualization provides individuals who lack information or encounter obstacles a means to retrieve and use digital symbols directly, without losing focus on the physical world.
Before any Augmented Reality introduction project begins, the objects with which people interact and the interactions themselves can be inventoried. The most frequently manipulated objects are going to be most familiar and the least likely to benefit from an AR-assisted interface. On the contrary, it is those processes or objects that are infrequently encountered and yet complex where the greatest potential for Augmented Reality can be tapped.
Which tasks or objects frequently present obstacles in terms of inexperience, limited human cognition or memory lapses? For example, in a warehouse, nearly every order or the contents of every truck is unique. There’s very little that past experience or strength can do to help humans perform their job better, but a digital guide for where to find an object or how to pack it on a palette, can reduce the need for search, trial and error.
In a field service scenario, every part has had a unique life history with respect to use, environmental conditions and other factors. Workers can better use past experience for rapid diagnostics when the track record of the part is rapidly and clearly available.
Learning or training organizations are often good partners for project managers who want to document patterns across a workforce when performing key processes. These groups have a unique perspective on the tasks that are the most difficult to teach or retain, and may also have well established methods for measuring performance in the lab and on the ground.
Capture Ground Truth
Prior to introducing Augmented Reality, perform a systematic measurement of the un-assisted process. Interviews with those who will participate in the project are always beneficial to assess attitudes about the new technology, but the documentation of ground truth must include actual task observations.
Accompanying a person and observing all their activities is one way to document their existing processes but this is likely to introduce a variety of errors in the data. If possible, automatic measuring tools that are completely invisible to the subject and do not interfere in any way with the normal flow of tasks should be explored.
Frequently, when a person needs assistance and cannot easily find the information in a manual, there is a need to consult another worker. Since AR could reduce the need for a person to seek assistance from others, the impacts on other workers’ productivity should also be considered.
A representative sample of people with different training or experience levels is key to getting good ground truth data. By observing the methods of a novice as well as a highly skilled journeyman and people between the two ends of the spectrum, it may be possible to narrow down a limited number of steps that are most likely to benefit from AR support.
Build Recording and Capture Tools into the System
When designing AR experiences, it may be possible to record the achievement of specific steps or the entire session of use. This may require mounting an independent camera into a workspace, or adding components to the AR delivery platform. To get the highest fidelity recording may require adjustments to ambient lighting since the AR experience setting may not be suitable for the camera that is recording activities. If using a mobile platform to capture activities, the additional task of recording interactions will impact battery life and, if network-based storage is part of the design, the communication needs (in terms of coverage and bandwidth) will certainly be different than those of the AR-assisted application itself.
Another key component when recording the user’s interaction is to have permission in advance for the project to use the recording in documenting the impacts of the AR-enabled system. Usually a simple release is adequate but in a unionized work setting, having the cooperation and support of the union might be necessary for successfully documenting the impacts.
Be Flexible
In some projects, the additional information provided by an Augmented Reality-assisted system introduces new opportunities to save resources, to catch un-discovered errors or even to document entirely new methods to complete a task. It may also enhance the work experience of the employee, which may be an important “soft metric” that is difficult to assess. In general, these are all important factors to consider, even if difficult to quantify.
In order to reduce the likelihood of overlooking the qualitative (as well as unanticipated quantitative) impacts, projects should include an in-depth exit interview. This can be conducted either online or face-to-face. In the interview with study participants, invite discussion and feedback on all aspects of the experience. Something valuable is likely to shed light on metrics collected as well as other obstacles to, or drivers of adoption.
Recommendations
Every organization has a unique approach to new technology introduction and different industries place emphasis on different performance metrics, but there are some basic best practices to follow, based on past experience of AREA members. When designing an AR introduction impact measurement system, project leaders can apply these best practices:
- Consider the business setting and management priorities in order to design metrics that matter most to those making the final decisions.
- Collaborate with different stakeholders and groups to identify and thoroughly document characteristics of bottlenecks or pain points that are common or similar across diverse professional skill sets, tasks, groups, products or facilities in the organization (e.g., transit time, down time, assembly errors and inspections).
- Capture existing processes that are part of the proposed AR introduction use case, as performed by both novices and senior members of the workforce, without the assistance of new technology.
- Build in or set up recording systems that do not interfere with or impact the user’s performance or the AR experience delivery.
- Perform an exit interview and keep an open mind about impacts that the user may have perceived that were not originally part of the study’s measured parameters.
How have you designed your pilot to capture metrics, and have the measurements helped to estimate the impact the introduction of Augmented Reality will have in your organization? Please leave your feedback below.
Want to hear more? Watch this video…