AREA-Directed Research: RFP Submission

Effectiveness evaluation of Augmented Reality (AR) for delivering highly-dexterous work instruction

Customer Problem

Many enterprises are exploring – or currently deploying – AR for displaying work instruction. Prior research projects gauged the effectiveness of using AR in learning and for facilitating task-based operations. This research project seeks to expand the current state of knowledge and shed light on AR-support for highly dexterous tasks, i.e., tasks that involve high levels of variability and flexibility in completion of a set of manual operations. An example of such tasks can be found in composite manufacturing, where typically a multitude of steps is required for task completion. Even more, each process step involves high levels of variability and there is more emphasis on the operator fulfilling subjective tasks rather than pre-specified operations.

In industries such as Composite Manufacturing, there is a drive for customers to produce ‘Right First Time’ products with an emphasis on the reduction of scrapped parts and time-consuming part reworking.

The results of this project will inform companies with these goals in their decision-making regarding adoption of AR-delivered work instructions for manual tasks involving high degrees of dexterity.

Suppliers of AR-assisted operator work instructions and learning solutions require evidence regarding key performance indicators to monitor and demonstrate effectiveness of their platforms to industry partners. This research project is expected to quantify impact and develop methods to predict potential benefit for companies delivering work instruction for highly dexterous tasks to employees regardless of their prior knowledge (for rapid re/upskilling).

Project Goal

The project aims to study impact of AR-assisted instruction for performing complex tasks that require high levels of dexterity. For this, the project will use a between-groups research design to contrast findings against those of a control group instructed with traditional means (PDF). It is expected that both quantitative and qualitative methods will be used in a mixed-method approach.

The project would use three tasks with both AR and non-AR work instruction each. This research project will evaluate effectiveness and efficiency in task guidance with the help of a big enough group of voluntary participants to an experiment, requiring about one hour per participant to complete.

Outputs of the learning tasks can be assessed with the help of an external assessor, who will mark task fulfilment with respect to accuracy, quality of the resulting work outputs, waste produced, and errors in task execution.

The system or observation capture environment for this project will provide time codes for start, each step, and end of the operation, possibly also linking motion capture data with body posture or finger movement. Observations should be complemented by a survey, setting additional evaluation focus on technology acceptance, spatial user experience, usability, or user engagement. Interaction with demographic characteristics could be of interest.

Bias (e.g., from priming effects) should be avoided. Part of the project proposal is to determine the required number of participants for the proposed exact research design and methodology. The proposer should explain and defend the chosen research design for measuring the intended KPIs and for answering the stated guiding research questions.

Guiding Research Questions

This project will answer questions including but not limited to the following:

• How does AR for high dexterity tasks differ from other instruction delivery methods?
• How are users impacted by the delivery of instructions using AR in high dexterity tasks?
• What are the key factors (e.g., cost structure) informing decision-making and driving return-on-investment in delivering work instructions for particularly dexterous, manual tasks?
• Can AR-assisted work instructions for variable, flexible manual tasks help improve:
  • Quality improvement
  • Productivity increase
  • Reduction of waste / rework of manufactured parts

The research will expose participants to different methods of information delivery:

• Digitized static instructions viewed on tablet device
• AR-assisted instructions viewed on a tablet device
• AR-assisted instructions viewed via an Augmented Reality (AR) Head mounted display

Deliverables

This AREA research project will produce the following deliverables:

1. D1: Report on the efficiency and effectiveness of AR work instruction for tasks requiring high levels of dexterity, including:
   1. Summary of key findings of relevant related prior research into AR training
   2. Assessment of the effect strength from training task to transfer task.
   3. Comparison of AR vs non-AR results with relevant statistics, significance level, and estimated confidence interval (as needed)
   4. Descriptive stats of the questionnaire data, with structural relations investigated where required by models, and demographic effects
   5. Recommendation for a return-on-investment formula to quantify potential cost savings for enterprises (with clear specification regarding task types and required conditions)
2. D2: Research Data Set (including task content, time codes, motion capture and other observation data, questionnaire data) and analysis source code (allowing for further exploration)
3. D3: Video summary highlighting the key findings
4. D4: Interactive members-only webinar presenting the findings from the research to the constituency of the AREA

Note: The research conducted can be published with a 6-month embargo period.

Proposal

The research project proposal will include (max 7 pages):

• Description of the research design and methodology (incl. explanation how it will measure requested KPIs and answer the research questions)
• Project milestones and timeline
• Project risks, bias, and mitigation measures
• Potential or confirmed partners in the research
• Background and qualifications of the organization to perform research and any proposed partners
• Examples of other completed projects/reports (not included in page count)
• Names and contact information for up to three references

Selection Criteria

All proposals will be evaluated by the AREA research committee chair and research manager on the following criteria:

• Experience in work instruction delivery using AR technologies
• Track record in research and development
• Research methodology and ability to deliver statistically sound research
• Number of devices provided for the experiment
• Timeline and scheduling information for participant management to reach proposed numbers
• Feedback of references

Timeline and Deadlines

Please use the form below to submit your proposal on or before 12 PM Eastern Daylight Time July 1 ^st 2022.

The AREA will provide detailed replies to submitters on or before July 7^th 2022.

Unless otherwise negotiated in advance, the research project is expected to take approximately 120 days. Research will be completed, and finished deliverables provided to the AREA by October 28^th 2022.

Budget for this Project

The AREA Research Committee budget for this project is $15,000.

Questions

For answers to any questions concerning this project and the AREA Research Committee, please send an email to the Research Committee.