State-of-the-art in Augmented Reality – A Review

Dr. RGS Asthana

Senior Member IEEE

Figure 1: Augmented reality in Defense and Military [15]

Summary

Augmented reality (AR) is a live, direct or indirect, view of a physical, real-world environment whose elements are augmented by computer-generated imagery or data.

We provide a few common definitions of AR, and show how AR fits into taxonomies of other related technologies. We briefly describe the key technologies, tools and SDKs used to implement AR. We also provide some examples of successful AR applications, particularly, in education, entertainment, training, military and retail. Finally, we conclude with description of portable AR and scope for future work.

Keywords

GPS, Augmented reality, mixed reality, virtual reality, Virtual Environment, Google Glass, HoloLens, head mounted displays, Portable AR System

Definitions

As per Wikipedia [1], “Augmented reality (AR) is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS [2] data.”

Another definition of Augmented Reality (AR) is a variation of Virtual Environments (VE) [10] or Virtual Reality (VR) as it is more commonly called. VR technologies completely immerse a user in a synthetic environment so that the user cannot see the real world around him. In contrast, AR allows the user to see the real world, with virtual objects superimposed upon with the real world. Therefore, AR supplements reality, rather than completely replacing it. Google Glass [38, 39] and Hololens are few examples of appliances, which use AR [7].

AR can be thought of as the "middle ground" between VE (completely synthetic) and tele-presence (completely real) [3]. Some researchers define AR in a way that requires the use of Head-Mounted Displays (HMDs) [14]. HMDs are display devices, worn on the head or as part of a helmet that has a small display optic in front of one (monocular HMD) or optic for each eye (binocular HMD). Major HMD applications include military, government (fire, police, etc.), and civilian-commercial (medicine, video gaming, sports, etc.). To circumvent limiting AR to specific technologies, some researchers define AR as systems that have the following three characteristics:

a) combine real and virtual,

b) interactive in real time and

c) Registered in 3-D.

This definition allows other technologies besides HMDs while retaining the essential components of AR. For example, this by definition will exclude films or 2-D overlays. Films like "Jurassic Park" which feature photorealistic virtual objects seamlessly blended with a real environment in 3-D, but they are not interactive media. However, this definition does allow monitor [4] based interfaces, monocular systems [5], see-through HMDs [6], and many other combining technologies. In monitor based system directions are overlaid right onto a video of the physical workspace and are showed on a standard monitor making the solution not only cost effective but also technologically acceptable because it meets most of the industrial needs. In monocular style, observers can acquire information without shifting their attention, so they have a wider useful field of view (UFOV). The UFOV is the area in which an observer can acquire visual information.

Motivation

Why is combining real and virtual objects in 3-D useful? AR improves a user's perception of and association with the real world. The virtual objects display information that the user cannot directly detect at all or accurately with own senses. A few potential AR applications viz.: medical visualization, maintenance and repair, annotation, robot path planning, entertainment, and military aircraft navigation and targeting are described here in brief.

Mixed reality [35] combines holograms into physical world so that it looks and sounds like a part of your desired world. To reiterate, VR immerses you in a simulated world whereas AR puts digital information on top of your real world. Zuckerberg [30], founder of Facebook, believes that VR is just the beginning and it will later merge with augmented reality to become a staple in our everyday lives. “VR has the potential to be the most social platform because you actually feel like you’re right there with another person,” he said, alluding to the Oculus Rift that allows two people to play together in some VR modes. As per view of Tim Cook, CEO of Apple [31], AR is bigger than VR. He said, "AR is the larger of the two, probably by far, because this gives the capability for both of us to sit and be very present talking to each other, but also have other things visually for both of us to see, maybe it's something we're talking about, maybe it's someone else here that is not here, present, but could be made to appear to be present with us. So there are a lot of really cool things there." VR is not completely written off by Cook but in Cook’s view it has lower potential. Although, Apple seems to have lot of interest and plans but have not opened its cards.

It is expected that by end 2020, worldwide AR market will grow up to USD 150 Billion.

Marker based and Marker-less Augmented Reality

In a marker-based AR application the images are provided beforehand. Thus, it is known what the application should recognize while acquiring camera data. Most of the AR apps dealing with image recognition are marker-based. It's comparatively simple to detect things that are hard-coded in your application. There is no need of accelerometer or compass in a marker-based app. The recognition library may be able to compute the pose matrix (rotation & translation) of the detected image relative to the camera of your device.

The black frame in an image delivers strength to marker tracking because it has high contrast. The interior of the image can be filled with application specific artwork, like a framed painting. Frame markers are ideal for branding as businesses can put their logo inside the marker. Split markers, on the other hand, are composed of two separate barcodes, which further reduces the occupied area. Similar to frame markers, the interior area is available for businesses for their use. One can hold a marker in the hand with the thumb covering part of the marker without affecting the tracking as only two of the four sides of the marker contain features required for tracking. Both frame and split markers are shown in figure 2. Once a marker is seen then what to do next is pre-programmed.

A markerless AR app recognizes images that were not known to the application beforehand. This scenario is difficult to implement because the recognition algorithm running in your AR application should identify patterns, colors or some other "features" that may exist in camera frames. For example, if your algorithm is able to identify image with cross, it means that the AR application will be able to trigger AR actions whenever a cross is detected in the camera image, without providing images with all the crosses one can think of. This may need training a database when developing the application.

Google and Qualcomm [24] are working on markerless AR projects viz. Tango [25] and Smart Terrain [26] (Hands On: Vuforia SDK) respectively. Marker-less AR systems are a better option for applications in areas such as gaming & entertainment, commercial product visualization and advertising as they use normal images or objects as targets and they are no invasive like marker-based systems.

Optical AR Systems

The physical world’s view is not obstructed by an optical AR system as against video AR system. Google Glass is one example of such a system as optical mixing is done of images obtained from Physical World and Computer. This system uses true resolution of the Physical World. Lighting conditions of the physical world can have some issues. There, however, can be delay in generating CGI depending on the application.

Figure 2: Frame and Split Markers

Video based AR system [29]

The system described here provides a driver-assistance by giving warning if he leaves a lane i.e. lane-position tracking. It does "video-based lane estimation and tracking". It is designed using steerable filters for accurate lane-marking detection. Steerable filters provide an efficient method for detecting circular-reflector markings, solid-line markings, and segmented-line markings under varying lighting and road conditions. They help in providing robustness to complex shadowing; lighting changes from overpasses and tunnels, and road-surface variations.

AR Tools and SDKs

Table 1 [20] gives information about the Product, Company name, License(s), under which each one of these tools is distributed, and the supported platforms.

Table 1: Main AR Tools and SDKs

Product	Company	License	Supported Platforms
ARPA SDKs	Arpa Solutions	Commercial*	Android, iOS (ARPA SDKs), Google Glass (ARPA GLASS SDK), Android, iOS, Windows PC (ARPA Unity Plugin)
ARLab SDKs	ARLab	Commercial	Android, iOS
DroidAR	–	Free and Commercial	Android
Metaio SDK	Metaio	Free and Commercial**	Android, iOS, Windows PC, Google Glass, Epson Moverio BT-200, Vuzix M-100, Unity
Vufoia SDK	Qualcomm	Free and Commercial	Android, iOS, Unity
Wikitude SDK	Wikitude GmbH	Commercial*	Android, iOS, Google Glass, Epson Moverio, Vuzix M-100, Optinvent ORA1, PhoneGap, Titanium, Xamarin

* There is also a free trial available.

** On 28 May 2015, it was reported that Metaio, GmbH was acquired by Apple Inc.

Applications

Medical Visualization

Medical visualization [8] uses computers to produce 3D images from medical imaging data sets. In the last five years, commercial CT scanners have become available that can take five 320 slice volumes in a single second. That’s fast enough to make 3D videos of a beating heart.

Doctors could use AR as a visualization and training aid for surgery. Using non-invasive sensors, 3-D images/ datasets of a patient in real-time can be obtained from Magnetic Resonance Imaging (MRI), Computed Tomography (CT) scans, or ultrasound imaging. These image/datasets could then be projected in real time with image of relevant part of the real patient. In effect, this will enable a doctor true vision inside a patient which is extremely useful for minimally-invasive surgery. The view can be zoomed to any level to make surgery easier. This approach avoids need of large incisions and surgery can be very precise. AR techniques are also useful for general medical visualization in the surgical room as Surgeons can easily find some features which are difficult to locate through bare eyes in a MRI or CT images, and vice-versa.

AR would enable surgeons to images from MRI or CT as well as real-time patient image of surgical area simultaneously. This will help surgeons in precision tasks, as system may guide him where to drill a hole into the skull for brain surgery or where to perform a needle biopsy of a tiny tumor.

Segmentation of volumetric medical imaging data [9] of CT or MRI scans is a process to select structures and areas within such data. An exemplary application using segmentation as a basis is the computation of surface models around the contour of segmented structures. In normal scenario one may use standard devices viz. mouse and keyboard or alternatively, a 3D mice and haptic devices or semi-automatic tools to segment 3D data, but there are always issues as all these user interfaces still require a 2D monitor showing either 2D slices of the volume data or a projection of a 3D visualization of the data. Imagine the process of segmenting 3D structures from the 3D data would be similar to modeling pottery. What if one can hold the data in one’s hand or place it on the table and then use scarpers and brushes guided with the other hand to model the regions of interest? What if we succeed in transferring the computer and 2D screen based segmentation tasks of today to a more analog 3D working environment? It’s a challenge which scientists are working to resolve. AR seems to have a role here.

These models can then be used for 3D visualization of the imaging data [9], measuring structures in 3D, building VR surgery simulators, additive manufacturing of anatomical models, designing patient specific implants, and much more.

Education, Entertainment [22, 41, 42] and Training

Both VR and AR [11, 12] are being used in education and training in many fields. AR in training can provide a rich, contextual learning environment to develop skills in a "no risk" environment, where the consequences from mistakes made are not the same as in real life.

In Medicine, AR is being used to enhance visualization of the human body, plan operations and train medical staff in various procedures. Both virtual reality and augmented reality have been used in digestive surgery. A combination of 3D modeling of patients from their CT-scan or MRI (as explained in the previous section) combined with simulation technology [12] is used to train the surgical gestures that will be used before carrying it out in reality. It can be combined to provide the surgeons with a transparent view of patient and can guide the surgeon with the virtual movement of their real surgical tools that are tracked real time during a surgical procedure.

The Military have long been using AR for their training [15] as their strength lies in use of best available technology. The US military has already begun employing AR technology in its training cycles [See Figure 1]. Young Marines recently used the Augmented Immersive Team Trainer (AITT) [16] a new AR-based system that trains troops in calling in airstrikes and artillery barrages. Traditionally, this form of training is difficult to conduct. Munitions and targets can only be used once and aircraft and artillery barges are prohibitively expensive and often unavailable during these training cycles. AITT enabled overlaying of the expensive components viz. aircraft, bombs and their resulting explosions on a real-world training battlefield. These elements would of course be required to behave realistically. Thus need to worry about limited munitions unavailable jets were circumvented. The added bonus is that unlike virtual reality, augmented reality in defence allowed troops to go through the actual physical motions of training. Soldiers require actual walls and stairs, real world obstacles to build up muscle memory. AR-based training systems are considered better than virtual reality based systems as AR allows the superimposition of synthetic images over the real world [17]. It’s great for a fighter pilot in a cockpit simulator, as he essentially deals with buttons, joysticks and screens. The pilot himself is static while it is his vehicle that moves in space.

Virtual instructions help a novice surgeon of the essential phases, without the need to look away from a patient to consult a guide/ manual. A case of breast cancer imaged using contrast-enhanced breast CT was viewed with the AR imaging, which uses a HMD and joystick control interface. The AR system demonstrated 3D viewing of the breast mass with head position tracking, stereoscopic depth perception, focal point convergence and the use of a 3D cursor and joy-stick enabled fly through with visualization {as depicted in Figure 3} of the mass’s speculations extending from the breast cancer [18].

When you combine AR with Internet of Things (IoT) you get something unique with many prospects {see figure 4}. For example, a new hire in any field may be given hands-on experience without the costs and risks of real hands-on. You can get best of both worlds and limitation, if any, is lack of your imagination [19].

Google Translate [27] is not intended to be an AR app, but it’s one AR feature but its camera mode is extremely useful. You need to simply snap a photo of the text you don’t understand, and the app will translate the text in your photo in real time.

The system developed by Israeli military [36] and described in section military apps of this article can also be used for training purposes. Pro sports have exploited both AR and VR – the use of down line and the early hockey puck glow are examples of some of the first uses of AR during live broadcast. The Pro Kabaddi World cup held in 2016 in India also used AR Technology to keep audience engrossed in game on TV broadcasts. Star Sports Pro Kabaddi app connects you with your favorite teams and heroes. This app is available on

iTunes i.e. for iOS devices [43] as well as on Google play for Android devices [44].

Figure 3: 3D Viewing Breast Cancer [18]

Figure 4: AR and use of HMD in Training [19]

VR and AR enable a better training experience, advanced analytics, and also a spectator immersion that, in turn, will enrich today’s broadcasts to great extent.

Teams like the Dallas Cowboys and Tampa Bay Rays use VR and AR systems to aid in training almost on regular basis. And it’s not just on-field performance NFL teams are after. Recently, the league began exploring the use of VR in diversity training.

AR apps build upon the world around us by displaying information overlays and digital content tied to physical objects and locations. For example, Niantic Labs' mobile monster-catching simulator has successfully turned the real world into a Pokémon [27] playground {see figure 5}.

Figure 5: Pokémon GO on Android Platform [27]

Use of Ink Hunter is recommended when it is significant where to put a pre-made or custom-made tattoo on the body. Tattoos placed on the body using the camera look as close to real life as you’re going to get — without actually going under the needle [27]. Figure 6 depicts some of the tattoos on a mobile screen.

Figure 6: Ink Hunter Tattoos [27]

Yelp in the year 2009 [27] was the first AR based app on the iPhone. Yelp app uses your smartphone’s GPS and compass to display AR markers for nearby restaurants, bars, and other businesses in real time. Augmented Car Finder [27] is an app designed to help guide you to your hiding vehicle in a parking lot. Once the car’s location is set, the app creates a visible marker showing the car, the distance you are from it and the direction you should walk to locate it.

Annotation and visualization [40]

AR technology could be used to annotate objects and environments with public or private information. AR helps to close the service knowledge gap and enables any person to effectively perform assembly, maintenance and repair jobs with minimal training. This is made possible by annotating physical objects by superimposing, in real-time, virtual information from documents, databases and sensors to assist technicians in performing complex tasks.

Applications using public information assume the availability of public databases to draw upon. For example, a hand-held display could provide information about the contents of library shelves as the user walks around the library. At the European Computer-Industry Research Centre (ECRC) [45], describes Collaborative Interior Design and Collaborative Engine repair where the AR system displays computer-generated graphics (lines in this case) and text (annotations) that describe the visible components or give the user hints about the object. Since we also track the engine, the annotations move with the engine as its orientation changes. The mechanic can also obtain assistance of a remote expert who can decide what information is displayed on the mechanic's AR system.

Assembly, maintenance, and repair of complex machinery are other areas of AR applications [21]. As we all know, instructions are little easier to understand, not in form of manuals with text and pictures, but rather as 3-D drawings superimposed upon the actual equipment, showing step-by-step the tasks that need to be done and how to do them. These superimposed 3-D drawings can be animated, making the directions even more explicit. Several research projects have developed prototypes in this area and established their viability.

Robot Path Planning

Teleoperation of a robot is often a difficult problem, especially when the robot is far away, with long delays in the communication link. Under this circumstance, instead of controlling the robot directly, it may be preferable to instead control a virtual version of the robot. The user plans and specifies the robot's actions by manipulating the local virtual version, in real time.

Robot 3D (three-dimension) path planning addresses the problem of finding an optimal and collision-free path in a 3D workspace taking into account kinematic constraints (such as geometric, physical, and temporal). This article [33] discusses few recently developed robot 3D path planning algorithms. The article deliberates on universally applicable implementable algorithm in aerial, ground, and underwater robots. This paper classifies methods into categories based on their exploring mechanisms and proposes a category, called multifusion based algorithms. These algorithms are analyzed from a time efficiency and implementation perspective. The article also analyzes merits and weaknesses of each of the algorithm.

Military Apps

Tanks - a moving fortress of steel, aluminum, and ceramic composite often several inches thick and armored vehicles traditionally have poor situational awareness with a limited view of the outside world. Vision openings and periscopes allow some view of the outside, but to really get a good look around, crews need to a hatch and expose themselves to enemy tank fire, snipers, and artillery.

Now a Ukrainian Company Limpid Armor [13] is using the Hololens[38, 39] to give armor vehicle crews a better idea of their surroundings in combat. Taking a cue from the Joint Strike Fighter's Distributed Aperture System (DAS), the technology will give crews a 360-degree view of their environment without compromising their safety. DAS uses infrared cameras installed on the fighter's fuselage are linked to the pilot's helmet, allowing the wearer to "see" in all directions without turning his or her head.

The Ukrainian version [13] uses a series of cameras installed on the outside of an armored vehicle that are then fed into the Hololens. The Hololens can also project information over the video feed i.e. AR, noting mission objectives, enemy vehicles, hidden from the view and other mission critical information.

At the army base outside Tel Aviv [36], army has developed an app that allows commanders to manipulate military terrain models and intelligence data to monitor troop positioning from enemy vantage points. Battlefield maps are superimposed on top of the real terrain, streamed in via satellite, to create a blend that can be interacted with via sight, voice and hand gestures {see figure 7}.

Figure 7: Microsoft Hololens and use of Gestures

The unit is now finding ways to allow HoloLens-wearing medics to operate on wounded with instructions directly from trained surgeons, and combat soldiers to fix equipment malfunctions. It’s far removed from hurling Pokeballs at Pidgeys, Rattatas and Zubats in Pokémon Go [46], but based on similar principles.

AR in Retail [23]

Trying on clothes can be pretty off-putting for many shoppers, especially those in a hurry. Always one to test the boundaries of technology, Topshop has partnered with Kinect to create AR dressing rooms. This allows shoppers to virtually try on their purchases quickly and easily.

Toshiba developed a virtual fitting room system [37] that allows consumers to select and virtually try on clothes. It uses a 3D scanner and a camera to get a picture of the customer's body, and then sizes the clothes to fit. The booth has an app which empowers your mobile to select outfits, or place an order.

Using this app, women can find the right shades of makeup before committing to a purchase. The AR makeup mirror from Shiseido {see Figure 8} takes an image of a shopper’s face, before showing them what the latest cosmetics products will look like on their face. Another beauty example is from the Burberry Beauty Box store in Covent Garden, London that uses AR in a number of imaginative ways. The most prominent is their nail bar. Here, customers can select their skin tone and then place different polishes on the bar. The display then shows how the polishes look in real life.

Figure 8: Magic Mirror [23]

A new Audi A3 app allows you to spin the car by full 360 angle {see figure 9}, further you can change interactively almost every part (rims, colors, accessories etc.) and even can park the car anywhere and make an AR photo with the final configuration.

In 2013, IKEA launched their augmented reality catalogue to enable shoppers to visualize how certain pieces of furniture could look inside their home.

Figure 9: Look and feel of every aspect of Audi car A3 through an AR App

IBM found that 58% of consumers wish to get product information in-store before a purchase, and that 19% of customers will browse mobile devices whilst shopping. To address this issue, IBM launched AR app to provide shoppers with personalized information whilst browsing the shelves.

Portable AR system [32]

The patent describes a portable device configured to provide an AR experience with a display screen, an image capture device and image recognition logic configured to analyse the image data representing the real world scene. The image generation logic is configured to incorporate an additional image into the real world scene. A computer readable medium and a system providing an AR environment are also provided {see figure 10}.

Figure 10: Portable AR System [32]

A few applications [34] showcased for Microsoft HoloLens which, in fact, is a full-fledged portable holographic computer include:

· “Holographic Workstation” for Citi Traders, a mixed reality evolution of the trading floor workstation

· An interactive digital human anatomy curriculum by Case Western Reserve University and Cleveland Clinic

· On-Sight and Sidekick, software projects developed by a collaboration between NASA and Microsoft to explore mixed reality applications in space exploration

Conclusion

AR hazes the line between the physical and digital zones. It brings computer-generated graphics to life, superimposing images on what one actually sees. The potential for AR as a powerful training tool is only limited by your imagination, especially when you blend it with employee collaboration, field tests, and presentations.

From the success of experiment described in [12], aim now is at developing automated image-guided robotic surgical procedures by combining AR and robotics.

Microsoft HoloLens [35] embraces VR and AR to create a new reality referred to as mixed reality. As we all know, VR immerses us in a simulated world. AR puts digital information on top of our real world. Whereas, mixed reality enables holograms to look and sound like they’re part of our world. Israeli Military is developing some interesting AR apps to be used in Battlefields using Hololens [36].

The evolution of AR technologies is strictly related to component miniaturization leading to portable AR. For instance, the availability of extremely small cameras now allows designers to provide users AR glasses almost unnoticeable from usual glasses. The issues may also arise in blending the real and virtual images with focus and contrast. In near future, contact lenses will be able to incorporate all functionalities now provided by AR glasses and this clearly introduces some privacy and security issues.

We have described host of examples of AR being used in various fields including retail. Whether it is to enhance sale of a product or use of technology to help buyer in selection process, AR is something all brands and retailers cannot ignore.

In this review article, Author has made attempt to cover many areas where AR is applicable but many areas are also not mentioned to keep article brief.

References

[1] Augmented Reality

https://en.wikipedia.org/wiki/Augmented_reality

[2] Global Positioning System

https://en.wikipedia.org/wiki/Global_Positioning_System

[3] Augmented Reality: A Class of Displays on the Reality - Virtuality Continuum

http://etclab.mie.utoronto.ca/people/paul_dir/SPIE94/SPIE94.full.html