In his presentation on the InsideAR 2014 stage, AREA member Carl Byers of NGRAIN Corporation shared with the audience his conclusion that Augmented Reality is “all about people.” When in the middle of a technology-centric event taking place in the center of the densest AR-populated region of the world (Munich, Germany), it is important to reframe why all activities and investments matter: Augmented Reality helps people to see digital data in context.

The “all about people” guideline applies in medicine as well. Improving patient outcomes is at the heart of Dr. Kavin Andi’s research at St. George’s Hospital at the University of London.

Dr. Andi is an oral and maxillofacial surgery consultant who also practices microvascular reconstructive facial plastic surgery. In his InsideAR presentation Dr. Andi explained how Augmented Reality could provide value to:

• Designing and communicating tumor removal and reconstructive processes
• Detecting airway obstruction
• Planning bone and tissue harvesting

The presentation also introduced some of the many tools surgeons use to achieve positive patient outcomes. Some tools are physical: scalpel, saw, clamps and hoses of many types. And others use software. In addition to the many credentials he has earned in his journey from molecular biology to reconstructive surgery, Dr. Andi has invested heavily in mastering the use of a dozen different software products in a graphics pipeline.

Beginning with scans of patient bodies, the processes he has defined permit surgeons to better prepare for surgery. By studying and planning procedures to be performed in the operating theater in minute detail in advance, the time spent in the actual theater is reduced. 

Patient scanning is highly sophisticated, involving measurements of tumor and bone density, blood flow and other critical factors. But as in other fields where big data is accessible to the expert, the data needs to be accompanied with analytical tools, and in the case of surgery with real time visualization.

The first gap Dr. Andi needs technology to address is advanced segmentation. Better segmentation of the data will separate the volume of the tumor from the surrounding area that is affected. Then, with this data superimposed on the patient, Augmented Reality can help surgeons and assistants—and ultimately the patient—to visualize the proposed treatment.

Leaving diseased tissue in the patient, or removing too much tissue due to low accuracy visualization can impact patient outcomes. That is why surgeons also need (for registration and tracking with hands-free displays) to have sub-millimeter accuracy on deforming surfaces.

When this can be achieved, Dr. Andi envisages that he and other surgeons will be able to perform complex procedures with 3D digital overlay on the patient instead of constantly referring to a display on the side.

To learn more, watch Dr. Andi’s InsideAR 2014 presentation below.

Augmented Reality will be ubiquitous in the year 2025, according to one of the predictions shaped from the input of over 1,400 people and described in a new Pew Research Internet Report entitled “Killer Apps in the Gigabit Age,” released on October 9, 2014.

How can We Capitalize on New Bandwidth?

The respondents were asked to share their views on new killer apps in the gigabit age: will there be new, distinctive, and uniquely compelling technology applications that capitalize upon significant increases in bandwidth in the US between now and 2025?

The replies were then distilled into seven themes (Figure 1).

Source: Pew Research Center, Sept. 2014, “Killer Apps in the Gigabit Age”

“In 2025, Augmented Reality will enhance people’s sense and understanding of their real-life surroundings and virtual reality will make some spaces, such as gaming worlds and other simulated environments, even more compelling places to hang out.”

Many of the changes described by the expert respondents will emerge from a decade of maturation of the technologies we currently refer to as the Internet of Things. There will be much more than fast networks involved. In 2025, everything is continually connecting, capturing, storing and transmitting observations, as well as receiving data from other sources.

 Augmented Reality in 2025

The three key components of Augmented Reality—hardware, software and content—are directly impacted, even redefined, by the advance of technology and bandwidth. The biggest trends in hardware for AR-assisted experiences will be miniaturization and use of harvested power. With smaller sensors and processors, there is an increased ability to embed and distribute the components of an AR solution into multiple objects, both on humans and in the environment. With the ability to harvest locally generated or locally stored power, batteries will become smaller and their capacities greater.

Surprisingly, personal display technologies—a necessary hardware component for Augmented Reality experiences—are not often discussed by the respondents of the Pew study. Perhaps there is tacit agreement that there will be personal head-worn displays; the emphasis is greater on the use light and lasers producing high-resolution digital objects and representing physical world features, including people, with real time holography.

Respondents frequently describe software, the second key component of AR, as being less distinct and visible as part of computer-assisted systems than in 2014. Many experts predict that the “app economy” will be a distant memory. Software will run in the background, barely reaching the user’s awareness.

Impacts on People

In addition to technology changes, the respondents recognize that there will be enormous societal changes combined with rapidly evolving economic and cultural shifts. The study explores the human elements of life surrounded by sensors and actuators.  Visions converge on many points: improved healthcare services, more engagement between people at a distance and discreet “apps,” such as are prevalent today, will disappear.

Concerning other dimensions of life in 2025, there is controversy. Some describe increased security and privacy and others the opposite.  Perhaps this dichotomy is not reflecting of contradictions, but is rather a reflection of the deepening digital divide, between those who are more digitally empowered having greater privacy and the rest being more exposed.

Aside from how travel behavior will differ, this study does not shed light on the professional side of life in 2025, nor how Augmented Reality and its enablers will impact business due to streamlined commercial transactions, greater human productivity and lower risk to material assets.

How would you respond to the study’s key question? How will your AR-assisted business capitalize upon significant increases in bandwidth over the next decade?

Big Data is not just for IBM. Many organizations with important requirements are also benefiting from Big Data projects to improve the quality of their products and services, detect and take advantage of new business opportunities and accelerate decision making with fewer errors.

Earlier this year, Dell Software released a study conducted by Competitive Edge Research Reports, a subsidiary of Triangle Publishing Services Co., on Big Data projects and planning. The study, “BIG DATA: Midmarket Companies See Early Success,” concludes that even medium-size firms are now able and eager to benefit from Big Data initiatives.

The report’s key take-home messages could easily apply to Augmented Reality projects in the same and other enterprises. First, Augmented Reality is not just for large organizations.

Second, the report makes a recommendation for Big Data and AR project advocates, regardless of the size of the organization: visionaries who want to use data to change their businesses must have strong senior management as well as IT department support to succeed.

A closer look at their words of wisdom also bear out the need for IT departments to partner with those who ask for big technology investments to make sure they are targeting outcomes that will push business forward.

Lessons Learned

Partnership and collaboration are important in any business, but this study reveals that for big, unproven IT projects, the support and collaboration of senior executive stakeholders is critical to success. Executives have to be willing to go to bat for the best team, to obtain financial resources and data access and to make changes in their businesses to take full advantage of the project’s outcomes.

Another key to success related to the first is for the project to use real time data acquisition and analytics to improve business performance. For some that means making an organization more agile and responsive to its customers. For others, performance is measured in improved product quality (and error reduction). Prototypes and projects in development do not show the great benefits that can come about once technologies are fully deployed and embraced. Once Big Data projects are in production, respondents report significant improvement in key business intelligence metrics. Cost reduction justifications were not among the top six metrics.

There is definitely a lot to learn when designing and putting in place Big Data projects. Much of the necessary talent for complex big data project success can be sourced from within an organization, the study revealed. The respondents said that investing in staff—both to bring specialists in-house when needed and to train existing employees—was quickly justified over outsourcing parts or all of these projects.

The Big Data study participants said they could do more and recommended that others following their steps invest deeply in five enabling technology groups:

• Real time processing
• Predictive analytics
• Data cleansing
• Data dashboards
• Visualization

AR-enabling business systems will improve data acquisition speed and quality, potentially reducing the need for data cleansing and real time data visualization.

Learning from Others

In this study, 300 executives in companies with 2,000 to 5,000 employees shared how they are implementing data-driven approaches to decision making processes. Though not the focus of the study, some decisions must be made with physical world objects and are often taken by people lacking the time and skills to request or perform data analyses. The data on which to base decisions needs to be readily available and meaningfully communicated if it is going to be useful. That is where AR-assisted systems become important.

AREA charter members are reducing the roadblocks to Augmented Reality introduction in their organizations through appropriate collaboration with their IT groups. They are also sharing lessons learned in their projects with one another. Lessons documented in the Competitive Edge Big Data study fit Augmented Reality projects perfectly and there are probably many more Big Data project lessons that will guide enterprise AR project leaders in the future.

Have you been speaking with the Big Data advocates and project leaders in your organizations? What lessons have they shared with you?

Big data does not solve all problems in an organization. There are many myths that should be examined more carefully before selling these to your senior management. Read this article about Gartner Group’s view on big data myths.

Many visitors to the AREA web site seek to prepare themselves and their teams to sell the benefits of Augmented Reality to the senior executives of their organizations. The goals of their campaigns are to receive the executive stakeholders’ support for financial resources, to ensure that the project or initiative is in alignment with the primary business goals and to get the executive allegiance to resolve major (and minor) headaches when they arise in the course of the project.

To achieve these goals, they will need data, proof of others having achieved great results, as well as strong convictions. Passion and convictions may even be more important and certainly more readily available than tangible results reported by others at this early stage of Augmented Reality’s evolution. But they also lead to myths forming in the minds of audiences that are later proven wrong.

Avoid planting or maintaining illusions about these three common myths.

Augmented Reality will Work with Anything

Conceptually, Augmented Reality works on any target in the physical world. To bring digital information into alignment with a person, place or thing, however, requires two important precursors:

1. The target needs to have an experience associated with it using data that has been extracted from the target. The unique set of features associated with the physical world target need to be stored digitally
2. The system the user has available must be able to detect the same set of unique features and those are the basis for identifying the experience

Then, to continue the AR experience, the system must also track the same target over time.

In a laboratory or another controlled environment, demonstrations are based on targets that have features that can be reliably extracted and matched. Many objects are unsuitable because they are reflective, or they deform or change in one or more dimensions over time.

In the real world, many other factors can interfere with reliability of target detection and experience delivery even when the target is well known and optimal. There can be changes in lighting when the detection system relies on the camera. There can be interference from large metal objects and bodies of water when the detection system relies on a compass or orientation of the user. The network-based data may not be available for a variety of reasons.

When describing AR to senior management, or to anyone for that matter, do not promise that it will work with anything and under any condition. It simply does not and probably never will.

Augmented Reality Confers Wisdom

When describing the benefits, it is tempting to attribute new and important powers to the users of Augmented Reality-enabled systems. After all, everyone wants to use new technology, right?

While people using well-designed and delivered experiences should be able to perform their jobs better and to make data-driven decisions, the value of AR to the challenge it is designed to address relies heavily on two factors:

1. The raw data that was used to originally design the experience. If the information shown to the AR user is incorrect or misleading, the use of AR does not make the information correct
2. The design of the total experience from the point of view of the user’s interaction as well as the integration of the experience into a larger workflow

The data needed for the employee or customer to be able to complete a task or make a decision may simply not be available to the AR-enabled system. In this case, the user’s work is no further enhanced than it would be without the AR-enabled system. Raw data may also be too large or lack analysis, making it difficult to use.

A great deal of study and experience goes into planning the data access, ensuring data quality (e.g., fresh data is important but so does data based on longitudinal studies), and data processing prior to delivery to the user. Here the organization’s history and know-how with Big Data initiatives can be very valuable.

In general, the synchronization of information with the real world can accelerate business processes and decision making, but the user remains in control and must use common sense.

Augmented Reality Reduces Costs

In general, early studies strongly suggest that when compared with traditional methods, use of AR can reduce time and lower error rates, both factors that impact production and delivery costs. At scale, the costs could be very significant and have profound impact on business performance.

What this statement fails to take into account is the cost of designing systems, equipping people and environments, testing and implementing AR-enabled systems in the enterprise. Begin with the costs associated with testing and prototyping AR-enabled procedures. These will be far higher than simply adding a Web page or a link to the user manual.

Total cost of ownership must include everything from the beginning to the final training of users. No one has accurately measured all of these costs, although AREA members are in the process of developing the tools and systems to estimate them.

Cost reduction in IT development is never a sound argument to use with management. The better metrics pertain to the return on investment that may be measured in weeks, months or years, depending on the project size and scope.

Proceed with Passion and Caution

In addition to citing the results of others who have conducted studies on the use of Augmented Reality in enterprise, advocates who seek to build support among executives for new AR projects need to use caution when describing the potential impacts of Augmented Reality. The most common myths should be avoided at all costs or the proposed project’s support may wane or completely evaporate.

What are some of the myths you’ve heard about enterprise Augmented Reality? Share them with others so they can avoid them.

Among use cases to which Augmented Reality brings value, an increasing portion focuses on training. Training use cases fill an immediate need for user education in a rapidly changing industrial environment and can be successfully implemented with current AR technologies.

Training requirements differ across industries and employee roles. Metrics with which organizations can track training benefits also vary widely. AR-assisted on-the-job or classroom training of assembly and maintenance personnel can ultimately reduce the cost and time of doing tasks, leading to higher operational efficiency.

Assembly

Some manual production tasks such as spot welding and riveting are still prevalent despite the fact that many assembly workflows and procedures are automated. These manual tasks may require not only particular skills to fulfill them, but also demand expertise in a particular area of assembly, as well as personal adaptability to changing production processes.

Agile manufacturing is particularly demanding on workers, as the high cost of modifying production environments means that it often falls to humans to remain flexible in the face of changes. The less automatization and more customization in production environments, the more experienced workers need to consult manuals, work instructions and fellow experts to perform challenging tasks, which in turn can effectively become problem-solving exercises. This puts greater emphasis on training.

Maintenance

Training is also important in maintenance organizations. While many aspects of preventive maintenance have become automated via data collection, some types of corrective interventions require resolution by experts, who might be off site. Redundant systems may be available to provide fail-safe operation and minimize risk of service degradation but human intervention is usually still necessary. Whether preventative or prescribed, assessing risks, organizing a task force and scheduling the intervention according to escalation protocols take time.

Training for system maintenance in order to minimize downtime is a significant endeavor as well as time consuming, especially in cases requiring certification and where an employee’s equipment service credentials must be maintained with repeated training. Often, cross training is necessary to acquire expert knowledge for diagnosing a variety of problems.

Current Technological Limitations

Despite their potential, Augmented Reality technologies are still in their infancy. Mobile devices well-suited to AR-assisted tasks, such as Amazon’s Fire Phone and Google’s upcoming Tango Project, are making their way to the mass market. However, in a shop floor or factory setting, the acceptance and implementation of AR faces significant barriers:

• Workplaces are often dynamic environments, with frequent repositioning and changing of tool and equipment layouts
• Factory floors are often fully utilized with little open space
• Natural light enters the workplace and its intensity and angles change over the course of the day and the seasons

These realities pose challenges for AR implementations in various ways. For example:

• Technologies for object tracking and image recognition in mainline workflows are not optimized. Although edge-based and markerless tracking capabilities are available in Augmented Reality SDKs, they are not sufficiently powerful or flexible for tracking reliably in all conditions. For example, edge-based tracking can work well under varying lighting conditions, but is still susceptible to engineered surfaces presenting repetitive edges or many smooth surfaces.
• Marker-based tracking can produce consistent results, but fiducial position and quality in changing and cluttered work environments is difficult to maintain or guarantee.
• Capital investment and costs for customized development of pilot projects can be high. For example, prices for dedicated AR devices possessing novel features such as multiple cameras or sensors for depth perception, sufficient processing for real time capture of camera pose in a 3D coordinate system and realistic rendering based on image-based lighting are correspondingly steep. Customization requirements such as ruggedization for specific business environments must often be added to the cost.
• As a result of technological shortcomings in features such as tracking, some companies must explore and develop their own solutions, such as employing multiple cameras or sensors in a “six degrees of freedom” configuration in a controlled environment.
• Form factors and user interfaces of available head-mounted displays that are powerful enough for Augmented Reality tend to be bulky and provide an immersive experience. This can also create safety risks in busy and cluttered environments.

Although such factors introduce barriers to mainstream adoption of AR in enterprise, many companies are developing pilot projects to assess benefits at a reasonable cost. Such projects are being implemented in controlled environments, which provide opportunities to introduce AR-based training programs.

AR-based Training for Efficient Learning

Well-designed, AR-based training procedures using objects and environments that are as close as possible to real experiences can enhance learning by capitalizing on the following benefits:

• Reduced cognitive load promotes better focus on both procedures and the object being studied
• By overlaying AR instructions on objects, associations between the instructions and the object’s features and visual cues are more highly synchronized, thus promoting spatial learning
• Collaborative features of AR can connect remote experts with trainees, providing instructor-led guidance

Generally, faster learning increases worker availability and encourages better problem solving. An additional benefit to enterprises is that they can use pilot project facilities and set ups for dual project assessment and training purposes.

Is your organization using or contemplating AR-assisted training programs?

Benefits attributed to use of Augmented Reality are not just marketing hype; they are borne out in studies over the past decade. Despite requirements that still impose costs and other obstacles on AR implementation in the enterprise, the studies reveal that having AR-assisted systems guide users in performing complex tasks and support their collaboration is beneficial to performance.

This article summarizes some academic research findings and explains how AR can improve performance for scenarios where an AR system guides users through assembly and maintenance tasks.

Studies

Many studies highlight the differences in task completion speeds and error rates between two groups of users, with one group using a device such as a head-mounted display or watching a screen with AR instructions, and the other relying on traditional media such as a paper manual to complete tasks.

In four such studies, both groups performed identical, but relatively simple assembly and maintenance tasks. The table below summarizes the tasks on which the studies focus and the type of AR tested.

The studies found statistically significant differences in the performance of users, with AR-enabled groups having the edge. To cite Augmented Reality Efficiency in Manufacturing Industry: A Case Study:

• The group using AR instructions completed tasks 13% faster on average than the group using paper
• When using paper, the probability of using inappropriate tools was six times higher than with AR
• Also when using paper, the probability of putting a part in the wrong place was twice as high than with AR

These are the findings of one study and readers are invited to peruse the selection of linked studies above for specific information.

How AR Impacts Task Performance

The cited studies amply describe the positive impact of Augmented Reality on task performance, but how exactly does AR work?

Workers see instructions precisely overlaid on, or associated with, the parts to be handled or manipulated. These instructions can take the form of graphics, text or even audio. By delivering instructions when and where they’re needed, Augmented Reality reduces the cognitive work of part and tool recognition and allows users to concentrate more fully on the task at hand.

Furthermore, AR:

• Reduces body movements—workers tend to move around less when all the information is in one place
• Reduces attention switching—no need to switch between doing tasks and thumbing through a manual
• Promotes learning through spatial memory—it provides a frame of reference for fast and effective learning of new tasks, processes and equipment

Overall, Augmented Reality reduces both physical and cognitive efforts which makes for more efficient task completion than with traditional media such as manuals or on-screen help.

Caveat of AR

As any coin has two sides, these studies also have raised shortcomings associated with AR-assisted processes. A user’s exclusive focus on one area of view may reduce situational awareness of the periphery. This is known in industrial literature as “attention tunneling,” and is of acute concern in fighter pilots using head up displays.

Some research in Augmented Reality has uncovered potential issues with attention tunneling emerging from excessive focus on a single task and overreliance on AR cues. As one study mentions, this is primarily a design issue:

“Designers seeking to make use of the performance gains of AR systems also need to consider how the user manages their attention in such systems and avoid the over-reliance on cues from the AR system.”

Conclusion

Research into human factors of Augmented Reality reveals valuable findings that can be applied directly to the design of AR-assisted procedures for enterprise. The studies conclude that users can complete assembly and maintenance tasks more rapidly and with fewer mistakes with Augmented Reality. These conclusions will have significant impacts on business process design and operational costs.

Which human factors studies have you found helpful to guide your AR project design?

As technological advances in software and devices continue to drive adoption of Augmented Reality, the need has grown for proven, global standards for designing AR-specific products and processes. New and unfamiliar technologies always bring risks of higher project costs being introduced as a result of miscommunication, unfulfilled user expectations and safety, as well as having to invent or “reinvent” processes and taxonomies that have been used elsewhere.

A global standard that is open and promotes clear communication and a common conceptual framework mitigates such risks, and offers many benefits to accelerate the growth of the AR ecosystem of providers and customers by:

• Improving development of new AR-based products and services
• Allaying the fear of vendor lock-in
• Encouraging innovation
• Sharing best practices
• Safeguarding customers and users of products and services

There is a track record of successful adoption of reference models leading to industry growth. The Distributed Computing Reference Model (DCRM) provides enormous benefit to vendors and society served by information technology, having contributed to the growth of the knowledge economy as we know it today. In the domain of Augmented Reality, the AR Community a grassroots organization of AR professionals dedicated to the development of open and interoperable standards, has been contributing to the development of a new conceptual framework: the Mixed and Augmented Reality (MAR) Reference Model. The MAR RM is being prepared for publication using the process defined by the International Standards Organization (ISO). This collaborative effort has produced a working draft of the world’s first comprehensive reference model for Augmented Reality.

Mixed Reality

As the title of the reference model document indicates, Augmented Reality must be understood within the larger context of Mixed Reality, which is the area between our physical world and a 100% computer-generated virtual reality.

http://commons.wikimedia.org/wiki/File:Reality-Virtuality_Continuum.svg Title: Reality-Virtuality Continuum / Giovanni Vincenti

Mixed Reality is more a continuum or spectrum than a box with sharply defined borders, so it makes sense to define standards for the entire area, rather than for Augmented Reality alone. Regarding AR, you can think of it as watching the real world through your device camera, with graphics and other information overlaying your field of view. Farther along, Augmented Virtuality is more like watching a computer-generated environment like a video game, with some of the elements on your screen being real world images captured by the camera.

The concepts, systems and roles of actors within this technological continuum are the focus of the MAR Reference Model.

MAR System and Classes

Besides providing global definitions and terminology, the reference model describes a high-level representation of typical components in a MAR system that recognizes a real world context, registers target physical objects, displays MAR content and handles user interactions. Using the same high level architecture, the reference model defines three viewpoints of the MAR system:

• The Enterprise view describes the purpose and scope of the system, as well as the objectives of different actors and users of MAR components
• The Computational view describes system components, their functions and interfaces
• The Information view describes high-level data flows among components

Each viewpoint, presents a different version of the AR system architecture along with a detailed description of components, data flows, interfaces and actors. Requirements, as well as quality and technical criteria are also defined.

The reference model also presents a helpful classification framework for mapping MAR components to real world applications. Possible applications and services employing features such as camera or haptic user interfaces are organized as system classes, with each class being described as the one below.

MAR Criteria and Use Cases

The document additionally describes criteria for the following qualitative features:

• Performance
• Safety
• Security
• Privacy

The reference model illustrates real world applications by presenting nine Augmented Reality applications with use cases. In the last section of the document, a number of technologies and relevant standards are described.

Conclusion

Based on years of practical application development and industry consensus, the MAR Reference Model will become a valuable guide for AR designers and technology providers to define their own systems and business models. It is currently in ISO draft form with official publication foreseen in the first half of 2015.