Augmented Reality (AR) technology, which aims to ‘augment’ reality via the use of technology, can modify one’s perception of the world around them. For instance, a rotating virtual billboard could be placed on top of the Empire State Building in New York City, showing specific advertisements to different people using an AR system. This emerging technology has significant economic potential because it embeds virtual objects into physical reality at negligible marginal costs, while maintaining targeted advertising capabilities similar to those of the internet. AR will play an ever-increasing role in other fields too. For example, in cars AR improves human perception though additional sensory aids while still relying on visual-spatial acuity and the ability to navigate in dynamic 3D environments.
Smartphones and tablets have provided an important push to the daily use of AR systems. Such platforms have become ubiquitous, combining data connectivity, cameras and micro-electromechanical systems (GPS, accelerometer, compass) which are important to create AR content. However, even the most popular AR apps installed on smartphones or tablets only occasionally achieve continuous use for long periods of time. A range of new technologies, such as wearable equipment in the form of, for example, Google Glass, are aiming to solve this issue.
In particular, cars especially offer tremendous potential as highly-used, state-of-the-art AR platforms. Cars are particularly suited to AR as, on average, people use their cars for more than two hours per day and, while it is a challenge to compel people to constantly look at a few inches of their smartphone display, drivers tend to look constantly at their windshield screens. Cars offer other important features for the creation of AR content including very high power autonomy and highly immersive environments. Cars also offer control tools such as steering wheel and pedals, information systems, such as a sound system, and other electromechanical systems and sensors (GPS, speedometer, com- pass), which are interfaced with very high computing power. Furthermore, wireless connectivity is now be- coming embedded in cars, using dedicated frequencies and standards for vehicle- to-vehicle communications.
Historically, AR in cars existed even prior to the invention of electronic displays. Rear view mirrors were in- vented to augment the visual perception of drivers with a perspective of the road behind. However, the current use of AR systems in production vehicles has expanded far beyond such optical mirrors. In-car AR systems are now used in visual, acoustic and tactile forms. Starting with the latter, many cars now use what is known as drive-by-wire (or throttle-by-wire) technology, where the accelerator pedal is not mechanically connected to the throttle, but rather electronically connected to a throttle control unit. Similarly, in steering-by-wire, the steering wheel is not mechanically connected to the wheels, but rather to an electronic control unit that manages the steering.
Lane departure warning is conveyed to the driver through the vibration of electronically controlled steering wheels, mimicking the sensation caused by physical rumble strips
By using such technology, vehicles are able to reduce the lock–to-lock steering wheel travel as a function of the vehicle speed, and convey variable force-feedback to the driver through the steering wheel or pedals, replicating the traditional mechanical feel through AR. For instance, in some cars the lane departure warning is conveyed to the driver through the vibration of electronically controlled steering wheels, mimicking the sensation caused by physical rumble strips.
With throttle-by-wire, steering-by-wire, or even brake-by-wire, the set of pedals and steering wheels found in many modern cars is not very different from the PC gaming controls that are used to play racing games. All physical feedback conveyed to the driver through the steering wheel or pedals is in fact created by computer-generated AR.
In terms of acoustic AR, we find highly innovative systems currently deployed in production cars. In some German sport sedans, with powerful V8 engines, the noise isolation from the exterior is so effective that the engine is hardly heard. For some owners this can be considered a disadvantage and thus a clever solution based on acoustic AR has been developed and named Active Sound Design1. A pre-recording of the non-isolated engine sound is made at the different rotations- per-minute (RPM) of the engine, and played through the vehicle’s sound system, making the sound vary ac- cording to the actual sound of the engine. Using this AR technology it becomes possible to please both drivers who enjoy a quiet ride, as well as drivers who enjoy hearing the engine sound.
Acoustic AR is even becoming mandatory for some cars. Electric vehicles are too silent at slow speeds and have become dangerous to pedestrians that are not made aware of their presence. Legislation has thus mandated that electric cars now produce an audible warning of their presence at slow speeds. Car design now includes the figure of the composer, responsible for creating the acoustic signature of a rolling electric vehicle.
Electric vehicles are too silent at slow speeds and [...] electric cars now produce an audible warning of their presence at slow speeds
In addition to the optical rear-view mirrors mentioned above, other visual AR systems are also becoming common in modern cars. The video-see-through form is the most common, and examples of such systems include the rear-view cameras that display real-time video on a dashboard monitor, with super-imposed guidelines to help in a reverse parking manoeuvre. Other examples include night-time driving assistants, which display video captured by infrared or thermal cameras on a driver-centric dashboard monitor. This video can be augmented with computer-generated highlighting of pedestrians. Optical- see-through AR, where the windshield is used as a projection screen where digital content is merged with reality, is also emerging in production vehicles. Laser holographic projection is able to display navigation arrows that appear to be painted over the road pavement, or traffic signs as roadside virtual objects.
Vehicle-2-vehicle
The above examples show that AR systems are already heavily used in the production of automobiles. However, these current systems base the creation of AR con- tent on on-board sensors or information stored within the car. Wireless connectivity, which is important in the establishment of smartphones and tablets as AR plat- forms, has previously largely been absent.
Our research has focused on the design of novel AR systems that leverage the emergent standard for vehicular communications2, in the form of vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) systems. The transmission range for such communication can easily reach 500m on motorways but can drop to 150m or even less in urban environments. The frequency band is divided into several channels, including a control channel for coordination and transmission of traffic safety messages. These safety messages can be viewed as SMS that vehicles send to each other, which can contain information such as “ambulance approaching from the left” or “airbag deployed 500m in front”.
An important challenge is how to present this information to the driver in a relevant and timely manner. Other service channels allow for multiple applications, from infotainment services to real-time video-streaming.
Vehicular ad hoc networks allow digital content to be created based on sensors that reside in neighbouring vehicles or roadside infrastructures
This is only possible because regulations ensure that quality-of-service mechanisms give priority to certain delay-sensitive data streams. The same mechanisms are found in latest generation Wi-Fi routers that provide high quality audio in voice-over-IP applications.
In terms of AR systems, these vehicular ad hoc networks allow digital content to be created based on sensors that reside in neighbouring vehicles or roadside infrastructures. Instead of a video see-through system to assist in reverse parking manoeuvres, where the video feed comes from a camera installed on the back of the vehicle, we can design assistance systems that use perspectives from cameras installed in other vehicles.
This has been the inspiration for the design of the See-Through System (STS) overtaking assistant3, which works as follows. A vision- obstructing vehicle such as a bus or a truck is equipped with a dashboard camera and a V2V communication device. As a car equipped with a virtual windshield approaches the vehicle in front, it establishes a real-time video stream from the windshield camera to the virtual windshield. Based on the vision-obstructing vehicle’s dimensions and relative position to the car, the video stream is overlaid onto the vehicle using the AR capabilities of the virtual windshield and computer vision to seamlessly overlap the preceding vehicle. A virtual depth effect is added to the frame surrounding the video in order to account for the blind spots created by the vehicle’s length. The end result is a transparent vehicle that is inherently safer to overtake – and the ad shown on the back of the original vehicle can be replaced by a new virtual ad, specifically targeted at the overtaking driver.
While AR systems are especially relevant for the automotive environment, the road is shared with pedestrians, cyclists, emergency vehicles, buses and trucks. AR systems can be beneficial in improving the perception of other road users, especially with the increase of vehicle sensing capabilities, as well as ubiquitous connectivity of individuals and vehicles. AR can also create virtually segregated road segments in order to improve safety by separating road usage between different vehicles as well as creating virtual paths for emergency vehicles.
It should be noted that the adoption of AR systems is not only dependent on the technology, but also on the regulatory environment. On this front, AR systems have a straightforward adoption roadmap since they usually provide safety improvements or have no impact on driving behaviour. Furthermore, autonomous vehicles have a much higher burden-of-proof since they could endanger lives if not properly implemented.
Virtual Traffic Lights
An important aspect of AR is that the creation of digital objects is much cheaper than the creation of physical objects. The more expensive a physical object is, the more advantageous it is to create it as AR content. In terms of both installation cost and operational cost, traffic lights are found at the top in the road signage infrastructure. Such costs are raised if the traffic lights governing an intersection have the ability to adapt the cycle to the traffic conditions, based on inductive loop detectors. The ideal adaptation of a traffic light would be to have it actually disappear under some traffic conditions. However, such retractable traffic lights are too expensive to install in reality. An affordable and existing alternative is to attach a ‘Part Time Signals’ label to traffic lights that only work during some periods of the day.
With AR windshields and V2V communications it becomes possible to have virtual traffic lights, whose creation, cycle and phase durations are self-organized based on wireless communication between vehicles approaching an intersection4. No road infrastructure is necessary, as a stopped vehicle (a vehicle seeing a red light) is a very good and reliable temporary infrastructure to provide the computerized control for an intersection managed by a virtual traffic light. This vehicle uses V2V communications to broadcast the virtual traffic light messages, which are received by other vehicles and used to create the appropriate traffic light as a virtual object on the windshield. As the current cycle ends, this control is handed over to another stopped vehicle, which continues with another cycle.
Such virtual traffic lights are only created if they are necessary and they could be created only at times when they would be useful for current traffic conditions. A ubiquitous infrastructure of such retractable traffic lights is very much affordable using AR. It has been shown that the average travel time in cities with dense traffic conditions could be reduced by 60%4, while emissions and fuel consumption would be reduced by 20%5. This paradigm of traffic control based on virtual traffic lights has been shown to be deployable gradually, assuring compatibility between equipped and non- equipped vehicles6.
Virtual traffic lights are only created if they are necessary, and [...] it has been shown that the average travel time in cities with dense traffic conditions could be reduced by 60%, while emissions and fuel consumption would be reduced by 20%
Virtual Billboards
Augmented reality could not only reduce the costs of road infrastructure, such as physical traffic lights, but also provide a revenue stream to make major highways self-sustainable.
Most developed countries have an extensive highway network at the core of their communications infrastructure, with many indicators correlating economic development with the density and extension of the network. However, the costs of developing and maintaining this network can be quite high, and are usually supported by a mix of direct public funding, fuel taxes and tolls.
The concept of virtual traffic lights can be generalised to an entire range of virtual objects based on AR windshields and V2V communications. Virtual roadside billboards are a natural step in bringing internet-style targeted advertising to highways, and also a user- friendly alternative to tolls. Most internet services are fully or partially supported by advertising due to their ability to reach mass audiences at a negligible cost while providing high value targeted advertising. In a similar way, if highway users were offered a choice between having ad-sponsored free highways or paying tolls with static billboards, the end result would be overwhelmingly in favour of toll-free highways.
With virtual billboards, the vehicle employs V2I communications to roadside physical billboards with inter- net-connected V2I roadside units that fetch targeted advertising and transmit it to the vehicle so that it can be superimposed on AR windshields over the physical billboard. Eventually, the entire billboard could be re- moved and roadside units would only be necessary to create persistent virtual billboards. Overlaying with existing physical billboards guarantees that the visibility conditions that could affect the driver’s perception are maintained, as we only replace an existing ad with a more targeted one. For a highway service provider, just 2 cents per view per billboard would more than compensate for the toll revenue from a typical suburban highway7. On the other hand, advertisers are able to reach an adult market segment with a high car ownership ratio and provide targeted and localised ads to a captive audience. The idea of virtual billboards could be combined with STS overtaking assistance. Instead of a fixed ad painted onto the truck’s rear, the area could be used to display different ads, selected based on each drivers’ mobility profile. Note that the truck’s owner should also be part of the revenue chain, similar to a website that adheres to an internet-advertising display network.
Autonomous Vehicles
With the emergence of autonomous vehicles, virtual windshield technology presents a range of interesting possibilities. Currently, tactile AR systems in drive-by- wire cars try to augment the driving feel for the driver. In autonomous vehicles, the self-driving nature is assumed and this type of AR disappears. The driver assistance systems will also disappear, as there is no driver. Note, however, that the V2V communication part of systems such as STS or Virtual Traffic Lights still makes sense for autonomous vehicles.
With autonomous vehicles, AR content aims to benefit the comfort of the passengers. It is unlikely that the windshield will be used as a large Smart TV, completely obscuring the road and outside environment. But, as we do not need to convey information that is necessary for driving manoeuvres when a human driver is in command, a number of comfort-oriented enhancements could be designed. Other vehicles could be made in- visible. Curved roads could be made straight. Under- ground routes could become scenic and a Sunday drive could be made totally safe, with almost no fuel consumption, in the landscape of your choice.
Pacific Coast Highway, anyone?
