Intelligent personalized diagnosis modeling in advanced medical system for Parkinson's disease using voice signals

Abbreviations: HHD: Handheld Device; HMD: Head-mounted Display; PC: desktop/laptop; SAR: Spatial Augmented Reality

1.   Introduction

1.1.   Goals

While there are already existing surveys of Augmented Reality (AR) user studies ^[1,2,3,4], there is still no systematic review of the current state of collaborative Mixed Reality (MR) research to the authors' knowledge. To help the MR community determine opportunities for future research, this paper provides a high-level overview of collaborative MR studies from 2013 to 2018. The goals are to provide readers with the overview of collaborative MR research, highlight papers which are making an impact in their respective application areas, identify research gaps, and layout interesting research directions. Through this, researchers might be encouraged to explore and pursue research directions which will further develop and improve collaborative applications in MR. Most importantly, it summarises how collaborative MR environments have changed and improved the communication and interactions of collocated and remotely placed users, and how MR environments might change the future of research discovery, exploration, and informed decision making across several research disciplines.

Section 1 touches on the reality-virtuality continuum, describes the characteristics of collaboration in MR, and explains the motivations for developing this technology. Section 2 provides a high-level overview of the reviewed papers categorised based on application areas, types of display devices used, collaboration setups, and user interaction and experience aspects. Section 3 details the current state of collaborative MR research in each application area and highlights remarkable papers which are making an impact in their respective application areas. Section 4 focuses on user interaction and user experience aspects, which are gaining importance as previous problems (i.e. tracking and registration) are now being solved with the advances in technology. Section 5 describes research gaps in some areas and directions that require further research. Finally, Section 6 concludes the paper with a summary.

1.2.   Definition

Figure 1 depicts the reality-virtuality continuum ^[5]. Reality is the perception of the real environment without the use of any technology. Augmented Reality (AR) is a technology which allows near-seamless, real-time integration and interaction of virtual and physical objects in the environment ^[6]. Augmented Virtuality (AV) captures real objects and integrates them into a virtual environment. Virtual Reality (VR) is a technology which immerses users in a complete virtual, often computer-generated environment to simulate an experience. This paper focuses on technologies that operate somewhere in between real environments and complete virtual ones and are collectively called Mixed Reality (MR) ^[5,7]. Recent advances in display technologies are driving down the cost of MR devices which were previously only available within specialised facilities. As a result, there has been a significant adoption of MR devices across different application areas, such as Architecture, Engineering, Construction, and Operations (AECO) ^[8], Education and Training ^[9], Industrial ^[10], and Entertainment and Gaming ^[11].

Although recent MR developments have been proven valuable in single user applications ^[1], collaborative MR applications present a promising area of research. It is believed that collaborative MR applications addresses two major issues: seamlessness and reality enhancement, which are usually present in traditional collaborative computer applications ^[12]. There are existing taxonomies that divide collaborative MR systems into different dimensions ^[13,14,15]. In collaborative MR systems, the interaction space is an important dimension. It is subdivided into collocated setup where users are in the same location, remote setup where users are in different locations, and variable setup where different collocated users can collaborate with remote users. The temporal reference of the interaction is another crucial dimension. This differentiates systems which are limited to the use of synchronous interactions where users are present at the same time and which are usable for asynchronous interactions where users are present at different times. In this paper, a clear focus on the interaction space and temporal reference are given to the reviewed papers.

1.3.   Motivation

For the past years, industries and research domains have been rapidly growing in terms of scale, complexity, and interdisciplinarity. Thus, complex problems now require more knowledge than any single person possesses because the experience and expertise relevant to a problem is usually distributed among different professionals ^[16]. Bringing different and sometimes opposing points of view together to create a shared understanding of the problem among different stakeholders can lead to new insights, innovative ideas, and interesting artefacts. With the advances in information and communication technologies (ICTs), shared environments that facilitate multidisciplinary collaboration are now being developed.

Mixed Reality (MR) enhances a user's perception of the real world and allows near-seamless interaction between virtual objects and the real world. Information overlay allows remote collaborators to annotate the user's view and it enhances communication between collaborators by providing visual, auditory, or haptic cues. In this way, supporting collaboration in MR applications provides even greater advantages, especially when distributed stakeholders are necessary to complete a certain task. To help the MR community determine opportunities for future research, this survey provides a systematic review of the current state of collaborative MR research published from 2013-2018. It includes researches in different application areas that utilise different display configurations, collaboration setups, and user interaction and experience aspects (see Section 3). While the application areas are not exhaustive, they do cover the areas explored so far.

2.   Methodology

This section presents the research methods employed to carry out the systematic review process. Section 2.1 details the search process done to collect papers for this review and Section 2.2 explains the review process employed to ensure that the papers follow our review criterion.

2.1.   Search process

This systematic review was made as inclusive as practically possible. Using the search terms: (ⅰ) "Augmented Reality" AND "Collaboration" and (ⅱ) "Mixed Reality" AND "Collaboration", we collected papers from the Scopus bibliographic database, IEEEXplore, and ACM Digital Library. The search for the terms was made in Title, Abstract, and Keywords fields. All search results published in conferences and journals between 2013 and 2018 were taken into consideration.

2.2.   Review process

The search results were scanned individually to identify whether or not it supported collaboration in MR. Only 259 papers satisfied the criterion. The number of citations received by each paper to date was obtained through Google Scholar and was used to determine its impact on each application area. The papers which received the highest citation count were discussed to understand exemplary applications from each domain. An Excel spreadsheet was created in order to systematically keep track of all the reviewed papers. The reviews of each paper focused on the following attributes:

● application areas and keywords,

● types of display devices used,

● user collaboration setups, and

● user interaction and experience aspects.

2.3.   Limitations

Although a considerable amount of effort has been given to be systematic and thorough during the selection and review process, there are still limitations with the described method. Although multiple bibliographic databases have been used to cover a wide range of publication venues and topics, it remains likely that there are papers which were not included in this review. In addition, the search terms used might be limiting, as other papers could have used different keywords to describe "Augmented Reality" and "Mixed Reality".

3.   High-level overview of reviewed papers

Overall, 259 collaborative MR papers published between 2013 and 2018 were reviewed and categorised. Figure 2a categorises the papers into application areas. Figure 2b analyses the types of display devices used in collaborative MR applications. Figure 2c summarises the factors usually considered to establish collaboration in MR. Figure 2d shows the user collaboration setups in the reported papers.

3.1.   Categorisation

The papers were categorised into different application areas (see Figure 2a): (ⅰ) Architecture, Engineering, Construction, and Operations (AECO) (21 papers, or 14%), (ⅱ) Education and Training (42 papers, or 29%), (ⅲ) Entertainment and Gaming (40 papers, or 28%), (ⅳ) Industrial (22 papers, or 15%), (ⅴ) Medicine (12 papers, or 8%), and (ⅵ) Tourism and Heritage (8 papers, or 6%). The application areas emerged from the inductive analysis of the obtained papers. Alternatively, there were a number of papers which did not fall into any of the application areas mentioned above but were still included in this review because their focus was primarily on user interaction and user experience aspects. Figure 2a shows the change over time in the number of collaborative MR papers in these application areas. Education and Training, Entertainment and Gaming, and AECO seem to benefit from collaborative MR systems as they constituted more than half of the reviewed papers. Although there were fewer studies in Medicine and Tourism and Heritage, recent researches in these application areas already showed promising benefits of collaborative MR systems, suggesting a need for further studies. Finally, the analysis shows that there is an increasing interest in applying collaborative MR systems in certain application areas. These will be discussed in a separate section of the paper (see Section 6).

3.2.   Types of display devices used

The types of display devices used in the papers were also recorded (see Figure 2b). A significant number of papers (65 papers, or 25%) reported using head-mounted displays (HMDs), such as Oculus Rift, HTC Vive and Microsoft HoloLens, for their collaborative MR setup. This was followed by the use of a combination of different types of display devices that were mentioned in this subsection (60 papers, or 23%) and handheld displays (HHDs) (58 papers, or 22%). 38 papers (or 15%) reported using at least one kind of PC display (desktop and laptop) and either HMDs, HHDs, or a projector system. Our survey shows that desktop displays are often used by remote collaborators as they are often stationary and require more computing power to guide or assist a local user. There is an increasing interest (22 papers, or 8%) in using spatial AR (SAR) through projector systems to enhance collaboration. This might be due to the fact that using SARs affords users the ability to view and interact with digital information without being tethered to a display device. Finally, there were relatively few papers (16 papers, or 6%) which used a combination of HHDs and HMDs.

3.3.   User interaction/user experience aspects

The types of user interaction and user experience in collaborative MR were also considered. These were categorised as: (ⅰ) annotation techniques (89 papers), (ⅱ) cooperative object manipulation techniques (112 papers), and (ⅲ) user perception and cognition studies (55 papers) (see Figure 2c). Annotation techniques allow instructions and directions to be overlaid in the environment. They were found to be useful and effective in providing information to users during collaboration (as a 'guiding voice') ^[17]. Cooperative object manipulation techniques in collaborative MR environments were determined to be a promising way to decrease completion times of various tasks as multiple users can manipulate the same object at the same time ^[18]. There is also a need to handle privacy issues and view management of virtual objects in collaborative MR ^[19]. Finally, user perception and cognition studies, which aim to lessen cognitive workload and increase users' perceptual (e.g. situational, social, and task) awareness and presence, were also found to be an important factor to foster collaboration in MR environments ^[17].

3.4.   Collaboration setup

The collaboration setups used in the papers were also reported. Users can be remote in different locations as they collaborate in MR. Alternatively, users can be collocated during collaboration. Finally, a combination of both setups was observed in a number of papers. Figure 2d shows the number of papers across different collaboration setups. 129 papers (or 50%) used a remote collaboration setup, 103 papers (or 40%) used a collocated collaboration setup, and 27 papers (or 10%) used a variable setup with both collocated and remote collaborations. This analysis shows that there is an unexplored area for collaborative MR systems which can support both collocated and remote collaboration setup. While there have been numerous research studies on synchronous collaborative MR applications which require users to be present at the same time, there has not been the same interest in pursuing asynchronous MR applications.

4.   Application areas

In this section, the reviewed papers are categorised based on their respective application areas. In addition, the types of display devices used and collaboration setups for each paper are discussed. A high-level overview of the work done in each application area is also provided. Research papers that are making an impact in their respective areas are highlighted. Finally, interesting research directions for each application area are laid out.

4.1.   Architecture, engineering, construction, and operations

Architecture, Engineering, Construction, and Operations (AECO) were the main application areas of twenty-one papers. Expert coordination and discussion are found to be one of the keys to success for all projects in this application area ^[20]. For every design phase, architects, engineers, and subcontractors need to work closely to meet the requirements and deadlines. By incorporating experience and expertise of different professionals, the project team can realise high-quality decisions and innovations ^[21]. It is also evident that group discussions contribute to a better and more efficient outcomes when compared to individual decision making ^[22]. As projects become enormously complex, certain disciplines become increasingly specialised, and immense amounts of information are included during meetings, current researches are exploring a multidisciplinary and multi-organisational discussion environment that can support the planning process between project teams. The recent advances in collaborative MR technologies have the potential to offer new opportunities to provide such environments by offering immersive experiences, physical embodiment, and immediate feedback to its users that would be difficult to obtain through traditional design media ^[23]. Fourteen papers used a collocated MR setup, four used a solely remote MR setup, and three used a variable remote and collocated MR setup. Five papers used HHDs, five used HMDs, five used a combination of different display devices, three used a PC and HHDs or HMDs, one used HHDs and HMDs, and two used SARs (see Table 1).

4.1.1.   Exemplar paper

LinLiuTsai et al. ^[20] provided an example of how collaboration in MR can create a visualisation environment to facilitate discussion processes during project meetings. They employed AR technologies to display public and private information. Public information can be viewed in a stationary display, called a Building Information Modelling (BIM) Table, while private information can be viewed using mobile devices. By viewing public and private information separately, users can clearly grasp the whole picture of the construction project. In addition, the complexity of discussion-related information was reduced while keeping the necessary information available during the project meeting. The authors conducted a comparison test (with 36 participants) between their system and the conventional paper-based method to validate how users can benefit from their system. It was found that the completion time was significantly shortened using their system in both data-finding and problem prediction during the discussion process.

4.1.2.   Discussion

Even with a high amount of planning and coordination efforts during meetings, misunderstandings which can result in a decrease in efficiency and cause enormous additional costs and delays are still likely ^[24]. Due to the increasing complexity of projects, relevant information and important authority are often distributed across multiple locations and parties. Although it is ideal that experts are all present during project meetings, it might not always be the case due to varying schedules. In order to avoid the aforementioned problems, current researches are utilizing MR devices to support and improve coordination, discussion, and collaboration between different AECO experts. MR allows users to see digital information, such as construction plans, design sketches and blueprints, and 3D Computer Aided Design (CAD) models in a shared environment. In addition, MR allows contextual information to be placed relative to the user's location in the environment. As a result, different experts can actively participate in discussions about strategies to meet the deadlines, search for information more efficiently, and foresee potential problems faster. It will be interesting to investigate asynchronous collaboration in this application area. Asynchronous collaboration allows users to create and retain digital information which can be used for later consumption. This provides an opportunity for members to revisit previous project meetings and brainstorming sessions while keeping annotated documents visible. Through this, team leaders can monitor their teams' progress and members can review the project timeline. Finally, it will be interesting to see different AECO firms remotely collaborating in MR environments during project meetings.

4.2.   Education and training

Education and Training are well-explored application areas in collaborative MR research. Forty-two papers (29%) focused on this application area. As expected, all studies reported teaching applications with a few studies focused on areas like military and sports training. A majority of the papers focused on improving learning through collaborative engagement and participation in subjects like natural science, history, computer science, and mathematics. Thirty-five papers used a collocated setup, six papers used remote setup, and one used a variable setup. Eighteen papers reported using HHDs, nine papers used a combination of different display devices, six used SARs or projector system, four papers used a combination of desktop computers and HHDs, HMDs or SAR, three used HHDs and HMDs, and two papers used HMDs (see Table 2).

4.2.1.   Exemplar paper

Bressler and Bodzin ^[49] investigated factors affecting student engagement during a collaborative mobile AR learning game. They conducted their study using a mixed method approach through pre- and post-surveys, field observations, and group interviews with 68 urban middle school students. The sample population included 35 male (51.5%) and 33 female (48.5%) students, aged between 11 and 15 years old. Students teamed up in groups of 3 or 4, and played a forensic science mystery game where they analysed fingerprints, hairs, and other trace evidence. During the game they collaboratively solved investigative problems such as decoding locker combinations and determining suspects' intentions. The findings demonstrated that a collaborative mobile AR learning game increased interest in science, made learning fun and enjoyable, and facilitated teamwork and engagement as students learned from each other.

4.2.2.   Discussion

Collaborative MR environments were extensively used in the education and training application areas. By presenting learning content in 3D perspective, MR educational applications can provide an environment where difficult and complex subjects, such as engineering concepts, are easily taught ^[78]. This feature adds valuable help especially to the applications which support constructivism that require authentic context ^[86]. All MR applications support collaborative and situated learning to gain a social flavour, provide high interactivity, and increase students' engagement into learning activities. The work done in education is mostly directed towards collocated collaboration where students collectively learn from active participation in the group. In addition, learning through gamification is seen to be an innovative way to promote engagement in learning. Research on the interaction between users and virtual objects in MR environments are important to collaborative training. Different interaction techniques have already been developed but the most effective one is real hand interaction ^[87]. By utilising the affordance provided by the human hand, users can manipulate virtual objects quickly and precisely, with little conscious attention ^[88]. In addition, annotations is effective to convey spatial information as compared to using arrows or pointers ^[89].

4.3.   Entertainment and gaming

There was a total of forty papers reviewed in this application area. A majority of the reviewed papers reported how collaborative MR was used to play games, such as solving a 3D jigsaw puzzle, competing in board games, and consuming multimedia content. Twenty-four papers reported using a collocated collaboration setup, twelve papers used a remote setup, and four papers used a variable setup. Thirteen papers used a combination of different display devices, eight paper used HMDs, six papers used a combination of PC and either HHDS or HMDs for their setup, five papers used SARs, four papers used both HHDs and HMDs, and four papers used HHDs (see Table 3).

4.3.1.   Exemplar paper

BallagasDuganRevelle et al. ^[90] developed a multi-player augmented reality game which is layered on top of an Emmy Award winning television show, The Electric Company. The authors developed the game to facilitate collocated collaboration and learning between siblings during joint consumption of media. They used different prototypes that combined mobile phones and web-based video. In the final game design, siblings must collaborate to collect and return words stolen by the prankster Manny who is a character in the award-winning show. The authors observed nine pairs of siblings aged between 6 to 10 and took video recordings while the siblings were playing the game. After the game, the children were interviewed in a semi-structured way by a researcher. The video recordings were then reviewed to analyse interactions and conversations made by the siblings. During the pilot test, the authors found that siblings made sense of media content better when using their application. In addition, they noticed that siblings displayed physical coordination by gesturing, verbally referencing physical objects, and guiding each other.

4.3.2.   Discussion

Although gaming applications in MR have already been developed, most of them are mainly accessible through expensive devices (such as HTC Vive, Oculus Rift, or Microsoft Mixed Reality devices and HoloLens) with different platforms and frameworks. The need to develop applications that allow players to join together regardless of the platform they own makes it hard to support collaborative scenarios in MR ^[91]. However, several factors, including advances in mobile connectivity and computing power and an increasing number of technology companies providing APIs (e.g. Apple ARKit, Vuforia and Google ARCore) with collaborative features for developers, make collaborative MR more accessible and attract more people to use MR for entertainment and gaming. Furthermore, companies like Facebook are enabling shared experiences between users at remote locations, in which they can interact using virtual avatars that typically reflect user movements captured by the user's HMD, external sensors, or controller input. In addition, the proliferation of collaborative MR gaming applications can be attributed to the increasing availability and accessibility of support and creation of complex networked applications by game engines (such as Unity3D, Unreal Engine, etc.). This opens a huge area for exploration on collaborative MR gaming especially driven using mobile platforms, paving the way for more widespread adoption. Natural user interactions without causing user fatigue should be researched and developed for sustained usage of these technologies. Both collocated and remote collaboration environments could potentially help drive the Entertainment and Gaming application area.

4.4.   Industrial

There was a total of twenty-two papers which used collaborative MR environments for industrial applications. All papers aimed to improve tasks during repair and maintenance of equipment, as well as manufacturing and assembly-related tasks. In this application area, collaboration where local users are being assisted or guided by remote users was the most common setup with a total of seventeen papers. Three papers had a collocated setup and only two papers have a variable setup. Nine papers used a combination of different display devices for their collaborative MR setup, six papers had a collaborative setup where remote users used desktop displays, while local users used HHDs, HMDs or projector systems, seven papers allowed users to collaborate using the same type of display device (HHDs or HMDs) (see Table 4). It is notable that less intrusive display devices were favoured so that local users can use both of their hands to accomplish tasks in industrial applications.

4.4.1.   Exemplar paper

One of the papers which is making the most impact in this application area was written by GauglitzNuernbergerTurk et al. ^[129]. The authors developed a system which allows live mobile remote collaboration on car repair tasks. Local workers use a lightweight tablet, while remote experts use a commodity PC. Remote experts can communicate with local workers and place spatial annotations which are automatically reflected in the local user's view. In addition, remote experts can independently navigate through the local worker's scene. The authors used proxy tasks, that would allow users to communicate while doing little or no physical labour, in order to create a controlled study setup. They evaluated the effectiveness of their collaborative application through an extensive outdoor within-subject design with 60 participants. They recorded the number of user errors and obtained user feedback through post-study surveys. In addition, they compared their system with two baseline interfaces: a video-only interface and a video interface annotated with static information. Their application was found to be preferred and usable by most (80%) of their test participants.

4.4.2.   Discussion

A majority of the papers in this application area focused on the repair and maintenance of equipment, as well as manufacturing and assembly-related tasks. Many of these studies have explored a remote expert collaboration setup where remote expert users provided help to local, usually less experienced users. In such collaboration setup, the main concern is how to establish an effective way of communication between the remote user and the local user. Promising progress has already been made to convey information, especially spatial information, from drawing annotations and pointer cues to reconstructing part or the whole body of the remote expert. It has been found that the projection of the remote user's hands gesture to a local user's environment is an effective way of providing task awareness and decreasing cognitive load ^[151]. It is promising to explore improvements in the non-verbal communication cues used during remote collaboration. Further work needs to be done to easily convey complex instructions during industrial tasks. In a remote collaboration scenario, the remote user can have either an independent view or a dependent view of a shared environment of the local user. Investigating the effects of the remote user view on remote collaboration is another important research topic. It has been found that the independent view had several benefits over the dependent view ^[17]. Our survey showed that less intrusive devices, which allow local users to freely use both hands for task completion, are desirable for this application area. Finally, this application area can also benefit from asynchronous collaboration. The workflow of people working in the industry usually involves shifts where outgoing workers hand over their work to incoming workers. Outgoing workers need to be able to recount all completed tasks to provide incoming workers insight of their overall progress. Retaining their actions through spatial information within the workplace will be a revolutionary step for a more effective and efficient workflow.

4.5.   Medicine

One of the most promising areas of collaborative MR is in medical sciences. There were twelve papers in this application area. Different medical experts can visualise the same information and share their knowledge to generate more meaningful insights from medical information. Nine papers had a remote collaboration setup where care is provided to a patient through the use of a remote specialist, two papers had a variable collaboration setup, and one paper had a collocated collaboration setup. Six papers used HMDs, three papers used HHDs, one used PCs and HMDs, one used HHDs and HMDs, and one used a combination of the aforementioned devices (see Table 5).

4.5.1.   Exemplar paper

DavisCanPindrik et al. ^[152] developed an iPad-based tool, called Virtual Interactive Presence and Augmented Reality (VIPAR), which allows experienced surgeons to provide remote, real-time, and virtual guidance to local surgeons. Local and remote surgeons from Vietnam and the US can perform endoscopic third ventriculostomy with choroid plexus coagulation with an aid of a composite image of video feeds with VIPAR. Fifteen procedures were performed with the use of VIPAR between Vietnam and the US, with no significant complications. The authors performed subjective and objective evaluations of the system performance through questionnaires. The survey showed that local and remote surgeons found VIPAR to be very useful for operating neurosurgeons.

4.5.2.   Discussion

Collaborative MR applications can support and enhance communication between medical specialists and remote patients. It will be promising to develop more platforms which will allow remote experts to guide or assist a local expert in performing surgeries, rehabilitation and recovery, and other medical procedures. E-consultation services could prove to be useful in improving patient services especially for patient in rural, remote, and under-serviced regions, allowing patients to receive higher quality services, and providing greater access to better healthcare services ^[86]. Similar to the effect of the internet-based telemedicine which provided a cost-efficient form of communication ^[157], collaborative MR applications may drive down medical costs. However, this will require studies to include more quantitative measures on performance measures, equipment alignment accuracy, and latency during collaboration. These need to be taken into account as poor performance can lead to deaths. The effectiveness of using HMDs, when compared against HHDs and traditional desktop configurations, has shown that they can combine the real world and the virtual world, and allows for interactions in-situ at the positions of 3D virtual models ^[164]. MR technology is efficient in the aspects of affecting depth perception, task completion, and social presence ^[165]. Utilizing these aspects can be applied to the improvement of medical education and healthcare services ^[162]. Through this, MR applications can assist medical practitioners in making more informed decisions, for instance when deciding whether to carry out surgery or not. This can be driven both by real medical data supported by simulation and feedback through MR environments.

4.6.   Tourism and heritage

Tourism and heritage are the application areas with the least number of collaborative MR applications with only eight papers. In this application area, providing navigational aids that are overlaid on the real environment is essential to help users plan their movements using spatial knowledge they have gained about the environment. If the provided navigation support is insufficient, users become disoriented and get lost. Therefore, developing efficient techniques for guiding the attention of users towards virtual objects or points of interest in an environment is the main focus of the current research in this application area. Out of eight papers, four focused on scene exploration, two focused on museum-based applications for cultural exploration, one focused on collaborative wayfinding, and one which focused on land navigation. Five papers used a remote collaboration setup, three papers used a collocated setup. Three papers used HHDs, two papers used PC and either HHDs or HMDs, two papers used SARs, and one paper used HHDs + HMDs for their collaboration setup (see Table 6).

4.6.1.   Exemplar paper

ChenLeeSwift et al. ^[167] developed an interaction model for supporting live remote collaboration between users. They presented a cost-effective system where remote users can use inexpensive devices to see the local users' view. In their paper, the authors illustrated how their system can be used during cave exploration. Local users can scan the physical space and create a 3D reconstructed model which can be annotated by remote users. The novelty of this system is that only a single HoloLens user is required to support collaboration in MR. In addition, they introduced a screen lock mechanism which allows remote users to create accurate and stable 3D annotations even though local users move their head around.

4.6.2.   Discussion

Just like entertainment and gaming applications, this application area will greatly benefit from using HHDs, as opposed to HMDs, since these devices are convenient for people to use while traveling. A novel direction in this application area is to provide a shared MR environment where users can create, share, and collect helpful information about physical objects and interesting locations and to help users thoroughly explore a new location or discover previously unknown features of familiar environments. By using their mobile phone cameras, users can point at a certain location, see reviews written by different people, and ask for directions when they get lost. This can be helpful for tourists visiting a new country and for people visiting a museum or a heritage site. One major advantage of using MR technology in this application area is that different positional information can be overlaid on the real environment to provide spatial awareness and guide the attention of users. However, MR environments can potentially contain a huge number of virtual objects at different locations. This problem is further complicated by the limited field of view of current MR devices, making it difficult to explore and navigate an MR environment. Although different techniques have already been developed, it still remains a challenging task as to how to convey information about surrounding virtual objects to the user ^[174]. Recent research has shown the potential of the use of multimodal feedbacks, such as audio, visual, and haptic cues, in enriching the communication and assistance between remotely located users ^[144]. It is promising to explore the effectiveness of multimodal feedbacks in providing instructions and directions during a collocated or remote collaboration in an MR environment.

5.   User interaction and experience aspects

Our study finds that there are three complementary factors to support and enhance collaboration in MR environments: (ⅰ) annotation techniques, which provide non-verbal communication cues to users, (ⅱ) cooperative object manipulation techniques, which make complex 3D object manipulation easier by dividing different tasks, such as scaling, translation, and rotation, between users, and (ⅲ) user perception and cognition studies, which aim to lessen cognitive workload for task understanding and completion and to increase users' perceptual awareness and presence. In this section, the reviewed papers are analysed based from these three complementary factors.

5.1.   Annotation techniques

Traditional approaches to remote guidance through phone or video calls limit how a remote expert can provide instructions and convey spatial references to a local user. Using speech to describe spatial locations and actions can be ambiguous or vague, leading to confusion and error ^[175]. In contrast, MR environments enable a remote expert to overlay information for spatial referencing on a local user's environment and to allow a local user to view the remote expert's annotations directly overlaid on the environment. Remote collaborative MR setups leverage annotation techniques to improve communication between remote users. A total of 89 papers reported that they explored annotation techniques to support and enhance collaboration among users in MR environments. Interestingly, 28 out of the 89 papers (or 31%) reported using a combination of either desktop computers and HHDs or HMDs, 22 papers (or 25%) used just HMDs, 13 papers (or 15%) used just HHDs, 11 papers (or 12%) used a combination of the different display devices, 8 papers (or 9%) used SAR through projector systems, and 7 papers (or 8%) used a combination of HHDs and HMDs (see Table 7).

5.1.1.   Exemplar paper

SodhiJonesForsyth et al. ^[176] presented a proof-of-concept design to explore 3D gestures and spatial inputs in collaborative MR. Remote users can perform a variety of virtual interactions which can be displayed in local user's environment. The authors used depth sensors to capture the 3D shape of objects in front of the sensor, as well as to track the location of the user's fingers. They provided qualitative user feedback from a preliminary study which indicated that users could perform collaborative tasks easily using their system.

5.1.2.   Discussion

The survey showed that recent research explores alternative ways to improve communication between remotely located users. Although providing instructions can be made through voice or video calls, it is usually hard to convey complex instructions which require spatial context. Annotation is one of the mostly studied visual communication cues for presenting spatial information in MR environments ^[17]. In a collaborative MR environment, annotations can include individual gaze directions, pointers from collaborators, hand gestures, and even avatars that reflect collaborators' actions. It is important to support annotations and non-verbal communication cues in collaborative MR environments as they require fewer inputs on the expert side and require less cognitive load on the local worker side ^[151]. It is highly recommended to display stabilised annotations at the real world where they were drawn to enhance mutual collaboration in MR environments ^[279]. Manual and automatic freezing of the shared video are the usual approaches to stabilise annotations in the real world. Manual freeze function allows a remote user to manually freeze the live video, to draw on the still video frame, and then to return back to the live video feed ^[129,279]. With the automatic freezing, the freezing and unfreezing interface was seamlessly integrated with the drawing interaction so that the live video automatically freezes when the remote user starts drawing and unfreezes when they stopped drawing ^[240,283]. The techniques are used to prevent annotations being anchored at a wrong place when a local user unexpectedly changes the viewpoint while the remote user is drawing.

A majority of the papers used mobile touchscreen displays to create annotations in 3D. Touchscreen displays allow direct interaction and provide instant haptic feedback. However, current mobile phones do not have advanced sensors for 3D depth perception. To support annotations in 3D, researchers must develop robust and efficient techniques to automatically infer depth for 2D drawings and create world-stabilised annotations in 3D. With the advent of new HMDs such as the Microsoft HoloLens, new types of annotations can be created.

5.2.   Cooperative object manipulation techniques

A total of 55 papers reported the implementation of cooperative object manipulation techniques. Out of the 55 papers, 19 papers (or 35%) used a combination of the different types of display devices, 16 papers (or 29%) used HMDs, 13 papers (or 24%) used HHDs, 3 papers (or 5%) used HHDs and HMDs, 2 papers (or 4%) used PC and SAR and 2 papers (or 4%) used SAR (see Table 7). Again, a trend of using HMDs is prominent in this application area. A majority of the reviewed papers used collocated MR setups.

Interactions in MR environments may be very complex, depending on the degrees of freedom (DOFs) required for the task. 3D object manipulation can be accomplished through different tasks such as scaling, translation, and rotation. Recent research proposes that collaboration can be used to solve this problem. Users can choose which transformations they want to perform on the object, and the effect of each transformation is combined to produce the final transformation. Through user studies, collaborating users were found to perform better than individual ones ^[18]. This strengthens the idea that collaborative MR experiences contribute positively towards task completion.

The lack of interaction and object manipulation techniques during collaborative problem-solving leaves outstanding challenges that need to be addressed before collaborative MR become widely accepted by the community. Further work needs to be done to provide the ability to smoothly and naturally interact during face-to-face and remote collaborations in shared workspaces. More natural interactions, such as new gesture and gazed-based interactions, are novel directions in this area.

5.2.1.   Exemplar paper

Cortés-Dávalos and Mendoza ^[187] developed a novel approach to the collaborative modelling of Digital Elevation Maps (DEMs) that are commonly used to model geometric assets (e.g. terrains on a landscape). In comparison to traditional applications for editing DEMs, their application allows a group of collaborators to use HHDs and easily visualise and modify 3D representations in an intuitive way. Their studies have shown that the perceived workload was considerably small because the sense of structure, shape, and size of DEMs were improved through the AR technology.

5.2.2.   Discussion

Object manipulation greatly affects user experience in all application areas. It is important that object manipulation techniques are intuitive and seamless so that users can interact with virtual content effortlessly. The implementation of different manipulation techniques that are universal and work across MR is an interesting research direction. In addition, it is important to make sure that they do not cause fatigue to users under prolonged engagement with the technology. Qualitative and quantitative evaluations must be done to assess their usability. User attitudes towards interaction techniques must also be taken into account when designing user interaction with collaborative MR systems.

5.3.   Perception and cognition studies

During collaboration, collaborators make a joint effort to align and integrate their activities in a 'seamless' manner to achieve a common goal whilst not interrupting each other ^[284]. In this process, it is key to be aware of what is going on in the shared workspace and in understanding the collaborators' activities. With the different types of display devices available, collaboration setups, and varying user interaction and experience, collaborative MR environments present different communication channels and different level of awareness ^[285]. Perception and cognition studies are interesting topics in collaborative MR research. Current researches investigated how different factors, such as types of display devices used and collaboration setup, affect the feeling of enjoyment and togetherness of users in a collaborative MR environment. In addition, recent work has explored the level of awareness and understanding of the participants collaborating in a shared workspace ^[17].

A total of 112 papers studied how MR enhances the sense of presence and the perception of social awareness, situational awareness and task awareness during collaboration. There was also a considerable amount of research which studied how collaboration reduces cognitive workload through MR environments. 40 papers (or 36%) used HMDs, 18 papers (or 16%) used a combination of either desktop computers and HHDs or HMDs, 18 papers (or 16%) used HHDs, 17 papers (or 15%) used a combination of different display devices, 10 papers (or 9%) used SAR, and 9 papers (or 8%) used HHDs and HMDs (see Table 7). There was a significant amount of perception and cognition studies observed in remote collaborative MR setups.

5.3.1.   Exemplar paper

KimLeeSakata et al. ^[279] conducted a user study on how the experience of sharing remote tasks and collaborating can be improved by adding visual communication cues in the environment. They developed a live remote collaboration system where local users use either HHDs or HMDs and remote users use a desktop computer. To investigate the experience of sharing a remote task space and collaborating with someone, they compared three video-conferencing conditions with different combinations of communication cues: shared live video only, shared live video with a shared pointer, and shared live video combined with annotations. They found that adding visual cues, such as pointers and annotations, significantly improved the sense of presence, togetherness, and connectedness between users.

5.3.2.   Discussion

Choosing the types of display devices, collaboration setups, and user interaction and experience aspects to be used in a collaborative MR environment is a crucial step as different media provide different communication channels and different levels of awareness ^[285]. A number of papers have explored ways to improve the sense of presence and social awareness during remote collaboration. The analysis showed that remote users usually feel disconnected to local users due to several factors. Non-verbal communication cues, such as gaze ^[286,287], pointing and hand gestures ^{[150,151,250,266,277]}, are important to give instructions since they are usually limited during remote collaboration. In addition, the limited perspective of the local users' environment greatly affects task awareness for remote users. Recent research provided an independent view for the remote user by physically controlling the camera. The results showed the effectiveness of this technique leading to better awareness and understanding of the shared activity in a collaborative MR environment.

6.   Other applications

6.1.   Crime scene investigation

Crime scene investigation usually requires the involvement of different organisations. Police officers and fire fighters are often the first to respond to a crime. They usually scout the area to make sure that it is safe for crime scene investigators to enter the scene. Once declared safe, the scene will be searched for crucial traces and evidence. During this time, it is important to not pollute the crime scene for better investigation. This application area can benefit from collaborative MR system.

DatcuLukosch and Lukosch ^[196] reported on the development and evaluation of a mobile AR system which supports collaboration among collocated and remote forensic investigators. Local investigators run the system on a smartphone strapped on their wrists, while remote investigators can see local users' view on a laptop. The authors evaluated the usability of their system and its effect on collaboration quality and situational awareness. It was found that although the mobile AR system addressed the limitations of HMD-based AR systems, the divided attention between the smartphone and the real environment greatly impacted the situational awareness.

Collaborative MR applications can improve collaboration between experts during crime scene investigation. This application area will also greatly benefit from asynchronous collaboration. With the advances of 3D depth sensors, it is promising to see MR applications where first responders in a crime scene can scan the room and ensure that evidence is free of tampering. The scanned crime scene can then be reviewed by multiple people in the head office, annotated during investigation, and archived for further investigation. As more evidence is added to the scene, new annotations can be created whilst previous annotations can be loaded to provide a bigger picture of the crime scene. Also, digitally recorded and reconstructed crime scenes can be investigated in completely new ways. For instance, measuring various distances, annotating timelines to enhance storytelling outcomes, and reconstructing possible scenarios by adding simulation into the mix.

6.2.   Collaborative data analysis

Decision-making based on data analysis often requires different experts to be effective. Collaborative MR applications can make data analysis more fluid by providing a shared environment where experts have a sense of each other's presence. Unlike traditional computer setups, collaborative MR applications provide a means to visualise large amount of data to be analysed simultaneously. Furthermore, they can provide more natural user interactions as compared to traditional computer setups.

ButscherHubenschmidMüller et al. ^[268] presented the Augmented Reality above Tabletop (ART) which is a collaborative tool designed for multidimensional, abstract data analysis. It uses multiple scatter plots and provides linked connection between data points by creating a 3D parallel coordinate plots (PCP). ART is designed to work in HMDs and is anchored to a multitouch tabletop, enabling users with familiar and fluid interactions. It was found that ART allows for a more natural communication and coordination between collaborators. Additionally, it can facilitate data immersion and foster a more fluid analysis process.

Collaborative MR applications seem to naturally support collaboration as they provide a shared environment where users can discuss and analyse information. An interesting research direction is to support non-linear analysis workflows where users can save analysis states for sharing and consumption at different times. In addition, the integration of basic statistical operations with visual representations is a novel direction in this area. Users should be able perform both manual and automatic operations, such as filtering, clustering, dimensionality reduction, to analyse and explore multidimensional data better and generate more meaningful insights.

7.   Discussion and conclusion

For the past years, industries and research domains have encountered a rapid growth in terms of scale, complexity, and interdisciplinarity. Thus, complex problems now require more knowledge than any single person possesses because the experience and expertise relevant to a problem is usually distributed among different professionals. Bringing different and sometimes opposing points of view together to create a shared understanding of the problem among different stakeholders can lead to new insights, innovative ideas, and interesting artefacts. Although it is ideal when different professionals are all present during discussions, it might not always be the case due to varying schedules. In order to prevent the aforementioned problems, current researches are utilizing Mixed Reality (MR) devices to support and improve coordination, discussion, and collaboration between different collocated and remote experts.

In this paper, 259 collaborative Mixed Reality (MR) papers published in a wide range of journals and conferences from 2013-2018 were reviewed. This was done to establish the current state of MR studies. The reviewed papers were categorised into application areas, types of display devices used, collaboration setups, and user interaction and user experience aspects. In the period given, collaborative MR applications were primarily used in application areas such as Education and Training, Entertainment and Gaming, Industrial, Architecture, Engineering, Construction, and Operations. Although there are relatively fewer papers in the field of Medicine and Tourism and Heritage, recent researches have proven that collaborative MR applications provide valuable impact in these application areas. In the period given, HMDs were the most popular display device used during collaborative MR. With the recent advances in mobile technology, a potential shift in the preferred display device (e.g. HHDs) and opportunities for increased studies in this area can be expected. Remote collaboration was seen to benefit from collaborative MR applications. However, a collaborative MR system which can support both remote and collocated collaboration can be a very powerful tool for discussion and analysis of complex problems and situations. Finally, our survey showed that attention has been given to synchronous collaboration setup to enhance the shared workspace. However, collaboration usually requires different experts who may not always be present. Asynchronous collaborative MR systems bridge this gap by providing the creation and retention of information and its consumption at a later time. Among other things, our study finds that there are three complementary factors to support and enhance collaboration in MR environments:

(ⅰ) Annotation techniques were found to be useful for providing spatial information and conveying instructions as they require fewer user inputs and less cognitive load to understand.

(ⅱ) Cooperative object manipulation techniques in collaborative MR were observed to decrease completion times of various tasks as multiple users can manipulate the same object at the same time.

(ⅲ) Perception and cognition studies were also improved in collaborative MR applications. MR applications provide more situational, social, and task awareness which made users more productive in both data-finding and problem prediction during collaborative discussion processes.

Acknowledgements

This research is supported by an Australian Government Research Training Program (RTP) Scholarship, and the Expanded Perception and Interaction Centre located at the UNSW Art & Design.

Conflict of interest

The authors declare that there is no conflict of interest.