Many companies have made significant investments in developing 2D and 3D content for applications such as product design, mechanical and electrical engineering, manufacturing, training, and customer service. As these organizations begin adopting enterprise Augmented Reality (AR), they often seek to reuse or build upon their existing content assets for their AR applications. This article explores and explains the use and repurposing of 2D and 3D content for use in enterprise AR applications.

Given that AR is often used to bridge the physical and digital worlds by overlaying digital content over a live view of the 3D world that the user inhabits, the level of congruence and context required of the display of digital data must be considered.

This document is not intended to be a detailed treatise on this matter, but rather attempts to serve as a high-level overview for those new to AR.

Introduction

Taking the simple example of displaying informational text within an AR application, there are several possible ways in which to display this content:

• 2D on-screen display (in “screen space”). Here, text is simply displayed at a location on the user’s screen without consideration of a location in the 3D physical world and often independent of the orientation of the camera. Typically, this text would appear overlaid over the camera view at top or bottom of the screen.

Usage: when the spatial context of the information is not important

• billboarded 2D text in “world space”. Often, AR applications provide a higher value of information presentation and interaction by displaying information at the spatial location to which the information relates. A common example is to be able to display multiple informational items at specific locations of a physical product (e.g., status information of various subsystems of a factory machine). In such examples, the focus is on delivering content that is always visible to the user (hence the “billboarding” – the information display is always parallel to the screen – you can’t walk around the “back” of the data and see it reversed). Clearly, there is an additional level of complexity in this scenario as the application must “know” where in the 3D world to position and orient the data. These 3D locations may be pre-determined or defined in various ways.

Usage: Display of in-context information that is easily viewable

• Non-billboarded 2D text in “world space”. Some AR applications call for information to be displayed at both a specific location and in a specific orientation. Such examples include overlays over digital displays on the physical product, amongst others where the viewing orientation of the user is important. Again, the precise location (and orientation) of this information within the physical world must be determined or pre-defined.
• Usage: Applications where the orientation and position of the text is important (and often the view orientation of the user).
• 3D text content in world space. An AR application may also use 3D geometrically-defined information (e.g., 3D characters, displays, etc.). Whether this is of value to the user is a matter of aesthetics, complexity and usability.

If this information is to be displayed on stereoscopic devices, then clearly the correct positioning of the items above in world space is important.

3D content: models, 3D scans, “scientific data”

The use of various forms of 3D content can provide a unique content presentation and interaction paradigm to radically improve contextual awareness or the understanding of information. 3D content can be used during the application content authoring phase (i.e., creating the AR experience and using the 3D model as “scaffolding”) in addition to the data consumption experience (i.e., within the AR app). The 3D data may also be used only during the authoring phase (i.e., it is not delivered to the end user’s device), its purpose being to provide a backdrop of spatial positioning of other content.

Here are some examples:

1. 3D models can be used to:
   1. Contextually inform users in step by step procedures by overlaying partial or whole CAD models over a live view of the physical product (e.g., showing animations of components to be adjusted, removed, operated, etc.)
   2. Place 3D models in a physical setting to study, present or review spatial, aesthetic or other aspects of 3D products in the physical world.
   3. Represent supporting graphics (e.g., tools used to perform an operation)
2. 3D scan data offers a good alternative for obtaining 3D context where no CAD models exist or are not available (or simply do not represent the current physical state of an object). Scan data can be used to:
   1. Support the authoring and positioning of digital AR information items in context (e.g., machine status in a factory environment)
   2. Place “real world” content into another physical setting
3. “Scientific data” is an informal term referring to the use of CAE (stress, thermal, etc.), CFD (particle flow, streamlines, etc.) and other (typically) simulation data (which in certain use cases may be data extracted from physical measurements) that are presented within the physical world view to aid communication and understanding to what may be very complex data. Some examples are:
   1. Visualizing the flow of heat within a physical environment to help the understanding of various options for heating and ventilation
   2. Visualizing the results of stress simulations overlaid over the physical products
4. Supporting graphics 3D geometrical items may be used to:
   1. Represent tools used to perform an operation in instructional materials
   2. Provide visual markers (e.g. arrows to show location, direction, etc.)

A related 3D topic is the ability to create 3D “virtual interfaces” that enable humans to interact with physical objects by means of a digitally rendered user interface that is delivered by the AR display device, in effect adding or extending the ability to operate the product by hand gestures or other input methods.

Important Considerations for 3D data

Irrespective of the target AR use case, careful consideration must be made about several aspects of leveraging 3D content:

• Data complexity. Large datasets (e.g., rich scan data or complex 3D models) can negatively impact the user experience by:
  • Exhibiting long download times
  • Reducing graphical performance
  • Consuming excessive memory on the target device
• Data and intellectual property. The delivery of 3D content “beyond the firewall” often raises concerns over the potential leaking of intellectual property.
• Automated content creation and delivery. Often, the process of using 3D content within an AR application is underestimated. AR app and content developers often do not have access to a suitable pre-existing representation of the digital product that satisfies the needs of size, security and complexity for real-time AR applications. This often results in the re-modelling of a CAD model that already exists in a new form, with obvious cost and time implications. More importantly, manual creation of special-purpose 3D models is difficult to support the scalability needs required by most enterprises.

An alternative approach for many industrial enterprises is to re-purpose their existing 3D engineering models for other 3D related applications.

To support the underpinnings of scalable deployment of AR across large numbers (and configurations with options and variants) requires careful planning and consideration of:

• An automated processing pipeline to convert often heavyweight CAD models to lighter-weight derivatives (often via standard 3D formats)
• Automated removal of “unnecessary” complexity within the 3D model. The AR app content must be “good enough” and several optimizations of the 3D content will likely be needed. These include:
  • Removing “internal” parts if they do not form part of the use case content
  • Reducing polygonal and triangular details (if used)
  • Restructuring of data topology (inter-surface relationships and so forth) to reduce complexity

Perspectives of various content types in AR apps

The following table summarizes the author’s perspectives and experience with various forms of digital product data and where it is most applicable within enterprise AR apps.

Applicability of 2D and 3D to use cases

Given that AR is used to overlay digital content over a real-time camera feed of the physical world, there is an inherent sense that the application is “3D” – i.e., it “understands” the world and presents information within it.

The following table illustrates the applicability of 2D and 3D content to a selected number of use cases and is intended to provide an overview of where specific domains of content are required or where a use case may benefit from the use of 3D data.  In the bulk of these use cases, 3D refers to the use of 3D CAD model derivations or 3D models purpose-built for the application.

Conclusion

This document has attempted to provide some initial guidance and context on the use of 2D and 3D content in various enterprise AR scenarios. Clearly, many AR-driven use cases will benefit from the use of 3D content. The benefits include reducing instruction time, helping transition expertise, improving quality of operations, reducing risk and similar value propositions.

However, the exact applicability of 2D and 3D content will, by and large, be determined by:

• Availability of content that may be re-purposed and supported by an appropriate automation infrastructure
• The cost and feasibility of custom-generating content for the specific application
• The risk, cost, value, time and other considerations of the target use case
• The availability of the relevant capabilities within the selected technology for the use case of interest

In summary, depending upon the specific use case, 3D content is either not required, a must-have or, perhaps most often, a future enhancement once the viability and value have been established.