Many companies have made significant investments in developing 2D and 3D content for applications such as product design, mechanical and electrical engineering, manufacturing, training, and customer service. As these organizations begin adopting enterprise Augmented Reality (AR), they often seek to reuse or build upon their existing content assets for their AR applications. This article explores and explains the use and repurposing of 2D and 3D content for use in enterprise AR applications.
Given that AR is often used to bridge the physical and digital worlds by overlaying digital content over a live view of the 3D world that the user inhabits, the level of congruence and context required of the display of digital data must be considered.
This document is not intended to be a detailed treatise on this matter, but rather attempts to serve as a high-level overview for those new to AR.
Taking the simple example of displaying informational text within an AR application, there are several possible ways in which to display this content:
- 2D on-screen display (in “screen space”). Here, text is simply displayed at a location on the user’s screen without consideration of a location in the 3D physical world and often independent of the orientation of the camera. Typically, this text would appear overlaid over the camera view at top or bottom of the screen.
Usage: when the spatial context of the information is not important
- billboarded 2D text in “world space”. Often, AR applications provide a higher value of information presentation and interaction by displaying information at the spatial location to which the information relates. A common example is to be able to display multiple informational items at specific locations of a physical product (e.g., status information of various subsystems of a factory machine). In such examples, the focus is on delivering content that is always visible to the user (hence the “billboarding” – the information display is always parallel to the screen – you can’t walk around the “back” of the data and see it reversed). Clearly, there is an additional level of complexity in this scenario as the application must “know” where in the 3D world to position and orient the data. These 3D locations may be pre-determined or defined in various ways.
Usage: Display of in-context information that is easily viewable
- Non-billboarded 2D text in “world space”. Some AR applications call for information to be displayed at both a specific location and in a specific orientation. Such examples include overlays over digital displays on the physical product, amongst others where the viewing orientation of the user is important. Again, the precise location (and orientation) of this information within the physical world must be determined or pre-defined.
- Usage: Applications where the orientation and position of the text is important (and often the view orientation of the user).
- 3D text content in world space. An AR application may also use 3D geometrically-defined information (e.g., 3D characters, displays, etc.). Whether this is of value to the user is a matter of aesthetics, complexity and usability.
If this information is to be displayed on stereoscopic devices, then clearly the correct positioning of the items above in world space is important.
3D content: models, 3D scans, “scientific data”
The use of various forms of 3D content can provide a unique content presentation and interaction paradigm to radically improve contextual awareness or the understanding of information. 3D content can be used during the application content authoring phase (i.e., creating the AR experience and using the 3D model as “scaffolding”) in addition to the data consumption experience (i.e., within the AR app). The 3D data may also be used only during the authoring phase (i.e., it is not delivered to the end user’s device), its purpose being to provide a backdrop of spatial positioning of other content.
Here are some examples:
- 3D models can be used to:
- Contextually inform users in step by step procedures by overlaying partial or whole CAD models over a live view of the physical product (e.g., showing animations of components to be adjusted, removed, operated, etc.)
- Place 3D models in a physical setting to study, present or review spatial, aesthetic or other aspects of 3D products in the physical world.
- Represent supporting graphics (e.g., tools used to perform an operation)
- 3D scan data offers a good alternative for obtaining 3D context where no CAD models exist or are not available (or simply do not represent the current physical state of an object). Scan data can be used to:
- Support the authoring and positioning of digital AR information items in context (e.g., machine status in a factory environment)
- Place “real world” content into another physical setting
- “Scientific data” is an informal term referring to the use of CAE (stress, thermal, etc.), CFD (particle flow, streamlines, etc.) and other (typically) simulation data (which in certain use cases may be data extracted from physical measurements) that are presented within the physical world view to aid communication and understanding to what may be very complex data. Some examples are:
- Visualizing the flow of heat within a physical environment to help the understanding of various options for heating and ventilation
- Visualizing the results of stress simulations overlaid over the physical products
- Supporting graphics 3D geometrical items may be used to:
- Represent tools used to perform an operation in instructional materials
- Provide visual markers (e.g. arrows to show location, direction, etc.)
A related 3D topic is the ability to create 3D “virtual interfaces” that enable humans to interact with physical objects by means of a digitally rendered user interface that is delivered by the AR display device, in effect adding or extending the ability to operate the product by hand gestures or other input methods.
Important Considerations for 3D data
Irrespective of the target AR use case, careful consideration must be made about several aspects of leveraging 3D content:
- Data complexity. Large datasets (e.g., rich scan data or complex 3D models) can negatively impact the user experience by:
- Exhibiting long download times
- Reducing graphical performance
- Consuming excessive memory on the target device
- Data and intellectual property. The delivery of 3D content “beyond the firewall” often raises concerns over the potential leaking of intellectual property.
- Automated content creation and delivery. Often, the process of using 3D content within an AR application is underestimated. AR app and content developers often do not have access to a suitable pre-existing representation of the digital product that satisfies the needs of size, security and complexity for real-time AR applications. This often results in the re-modelling of a CAD model that already exists in a new form, with obvious cost and time implications. More importantly, manual creation of special-purpose 3D models is difficult to support the scalability needs required by most enterprises.
An alternative approach for many industrial enterprises is to re-purpose their existing 3D engineering models for other 3D related applications.
To support the underpinnings of scalable deployment of AR across large numbers (and configurations with options and variants) requires careful planning and consideration of:
- An automated processing pipeline to convert often heavyweight CAD models to lighter-weight derivatives (often via standard 3D formats)
- Automated removal of “unnecessary” complexity within the 3D model. The AR app content must be “good enough” and several optimizations of the 3D content will likely be needed. These include:
- Removing “internal” parts if they do not form part of the use case content
- Reducing polygonal and triangular details (if used)
- Restructuring of data topology (inter-surface relationships and so forth) to reduce complexity
Perspectives of various content types in AR apps
The following table summarizes the author’s perspectives and experience with various forms of digital product data and where it is most applicable within enterprise AR apps.
|Content type||2D value||3D value||Commentary|
|Text||High||Low||Display information, instructions, etc.|
|Numerals||High||Low||Display data values|
|Symbols||High||Low||Display warnings, anomalies, instructions|
|Redlining||High||Medium||Support communications, capture issues|
|Schematics (2D) and engineering drawings||High||Low||Provide context from the physical object to other data abstractions|
|3D Models||N/A||High||Overlay within the “empty” camera view or overlay partial models over a physical product (often animated within step by step instructions.|
|Navigation aids||High||High||Show the user where to find a specific location both within and beyond the current field of view|
|‘Scientific’ data||Low||High||Present the results of simulated or processed captured data over a physical product|
|UI controls||High||High*||Enable interaction with the physical product or the workings of the app itself
*The user interface may be a purely 3D construct to implement a virtual user interface to a physical object.
|Avatars and spatial awareness of others||Low||High||Support multi-participant AR scenarios|
|Scene interaction (picking, etc.)||Low||High||Component selection for subjection information retrieval and display or part manipulation (e.g., position, scale and orientation).|
Applicability of 2D and 3D to use cases
Given that AR is used to overlay digital content over a real-time camera feed of the physical world, there is an inherent sense that the application is “3D” – i.e., it “understands” the world and presents information within it.
The following table illustrates the applicability of 2D and 3D content to a selected number of use cases and is intended to provide an overview of where specific domains of content are required or where a use case may benefit from the use of 3D data. In the bulk of these use cases, 3D refers to the use of 3D CAD model derivations or 3D models purpose-built for the application.
|Importance and value|
|Remote assistance||High||Low||AR-enhanced remote assistance apps typically offer various forms of duplex redlining and markup over the shared camera stream. The redlining typically uses computer vision to “lock on” to physical world objects. Overlaying redlining in 3D (i.e., on the physical product location) offers benefits for viewing the redlining from multiple angles.
3D data (models, etc.) are not typically used.
|Training||High||High||AR-based training apps vary in their richness of 3D content usage. Typically employing an awareness of the 3D “shape” of the object or scenario of the exercise (e.g., to point to or provide information on specific aspects) complemented by 2D information, AR-based training apps often leverage 3D content to enhance the experience.
The 3D content may include the use of 3D models to overlay over the physical product to show, for example, how to perform an operation by animating the 3D model. This may be enhanced by using the 3D models in step-by-step sequences.
|Field service||High||High||Examples of the usage of AR in field service include:
· providing visual context on the physical product (e.g., where to replace a component)
· displaying real-time data from the product (e.g., overlaying IoT data streams over the camera view)
· instructing the user how to perform the procedure
The latter point offers significant opportunities to improve the instructional process by incorporating 3D models, step-by-step sequences, and animations.
|Space planning and placement||Low||High||AR can be used to determine the form, fit (and function) of a physical object within its environment. This may be to determine the ability to design or configure data for a specific physical environment (e.g., placement of machinery within an factory space) or to present an aesthetic view of how the digitally rendered product appears in the environment (e.g., furniture within an office setting). This is inherently a 3D problem and therefore is highly reliant on the use of 3D models.|
|Logistics picking||High||Low||AR can be used in warehouse environments to improve the efficiency of the human “picker” in locating the correct location and product type. Whilst the primary use is to assist “navigation” within the 3D environment, there is very little requirement for 3D models in this use case.|
|Design review||Low||High||AR-enhanced design review apps aim to present a dynamic and configurable view of a product in a shared setting (often with multiple participants in multiple physical settings). 3D models are used as the core medium in which to present a product and discuss alternatives.|
|Product sales demonstrator||High||High||AR-based product sales demonstrators seek to motivate a purchase by placing a digital rendering of the object into the target physical setting or by digitally enhancing the live view of a physical product. In both cases, AR is used to highlight features, functions and aesthetics of the product. This may include:
· displaying textual “sales” information
· providing 3D-based animations of product operation overlaid over the physical setting or product
In either scenario, 3D content plays a key role and may be custom-created for the purpose.
|Factory operations||High||Med||One key use of AR in factory operations is to present a contextually-rich view of the (present or predicted) operating status of a factory machine. Whilst 3D models (or scanned data) may have been used as “scaffolding” in the creation of app content (i.e., assisting the author in the physical placement of information locations about the machine) there may be little use made of the 3D content within the end user’s app.|
This document has attempted to provide some initial guidance and context on the use of 2D and 3D content in various enterprise AR scenarios. Clearly, many AR-driven use cases will benefit from the use of 3D content. The benefits include reducing instruction time, helping transition expertise, improving quality of operations, reducing risk and similar value propositions.
However, the exact applicability of 2D and 3D content will, by and large, be determined by:
- Availability of content that may be re-purposed and supported by an appropriate automation infrastructure
- The cost and feasibility of custom-generating content for the specific application
- The risk, cost, value, time and other considerations of the target use case
- The availability of the relevant capabilities within the selected technology for the use case of interest
In summary, depending upon the specific use case, 3D content is either not required, a must-have or, perhaps most often, a future enhancement once the viability and value have been established.