The automotive industry is already offering some ‘AI-powered’ experiences, but customers are still waiting—whether they know it or not; whether they want it or not—for the automotive industry’s iPhone moment, when technology transforms the user experience and eliminates pain points, which might include rule-based interaction (particular vocabulary, syntax, and actions required of the user); limited infotainment apps, and limited personalization.
Once that ‘iPhone moment’ happens, it will usher in more natural and multimodal in-car HMIs using images, videos, and sounds, to improve user experience. Natural-language communication will entail automakers using prompt engineering and fine-tuning methods to enable the ‘car-pilot’ of the future to interact with users in a way that resonates with their brand culture. Visual content will be generated based on the system’s ‘understanding’ of the context and user preferences. Metaverse (AR/VR) content will be generated for entertainment.
Preferred in-car settings—climate, etc—will be predicted and selected. Safe-driving tips will be proactively suggested under difficult conditions. Occupants will be informed about functions they aren’t using, and user feedback and data will be collected and leveraged. Through in-car cameras and sensors, the ‘car-pilot’, whatever it eventually winds up being called, will detect the user’s mood and mental and physical state. It will cheer them up (whether they like it or not?) with peppy music, a conversation, or a light message.
Android Auto and Apple CarPlay as First Step
Android Auto and Apple CarPlay are services developed for iOS and Android to allow the modernization of the automotive world by displaying phone applications directly on the car’s screen in compatible vehicles.
Navigation, apps, and infotainment are always at your fingertips, and updates pushed over the air ensure your vehicle is always running the latest software (as with smartphones, if you preferred the way a previous version worked, you will have to use the new version anyway). A voice-control system makes interacting with the apps while driving your car easier and safer.
Android Auto, for example, provides access to Google Maps for navigation, YouTube Music for streaming, and Google Messages for communication. Similarly, CarPlay puts iPhone apps into a driver-friendly format, offering features like Apple Maps, Apple Music, and Siri for voice commands.
These integrations are designed to keep you connected and entertained while driving, without the need to handle your phone, so so you can focus on operating the car—at least the parts that involve your hands; your mind can still get wrapped up in apps.
Integrating ChatGPT and generative ‘AI’ chatbots into vehicles opens up a wide range of potential features beyond the basics. It could provide real-time alerts and suggestions based on driving conditions, such as weather updates or traffic warnings. The system could monitor vehicle health and remind you of upcoming maintenance needs, like oil changes (maybe not on EVs) or tire rotations. Music, podcasts, or audiobooks could be recommended based on your preferences, and in-car entertainment systems could be voice-controlled. Likewise, you could control smart home devices from your car, such as adjusting your thermostat or turning on lights before you arrive home. Based on your driving habits and preferences (or perhaps based on who paid the most to put ads in front of you), it could suggest nearby restaurants, gas stations, or points of interest. Over time, the system could ‘learn’ your routines and preferences, offering more personalized assistance and improving its responses.
Here’s a look at the state of automaker developments in this direction.
GM developed the integration of ChatGPT into their vehicles as part of their collaboration with Microsoft, as we previously reported. The automaker says users can interact with equipped cars in a natural and personalized way to input destinations and get help understanding and use vehicle features—similar to an interactive owner’s manual.
VW‘ssystem, as we’ve described,helps to make communication with the car more natural-like and to allow VW’s voice assistance software to help with more questions beyond the operation of vehicle systems.
Drivers in equipped Stellantis vehicles (see previous coverage) can access the navigation system and other functions via voice commands. ChatGPT is also controlled via voice commands, activated by the phrase « OK Iris » or a button on the steering wheel. On request, ChatGPT can provide tips on places of interest or works of art, or, especially useful for long car journeys with children, tell stories for young and old passengers.
Select Mercedes models (covered in DVN Interior) have ChatGPT as well, for intuitive speech recognition; the question does not have to be formulated in a cumbersome or prescribed manner.
China’s Nio, as we’ve previously described, offers their ‘Nomi’ personal assistant. It can be used via voice control, and can complex queries, responding in a natural language to deliver vehicle-specific information.
Recent Chatbot and ‘AI’-based announcements:
Intellias and Zeekr
Intellias has been nominated for three projects by Zeekr Technology Europe. This European subsidiary of China’s Geely is to receive real-time navigation, a location recommendation system, and a GPT-supported voice assistant from the development service provider. A new development center is to be opened in Krakow for the project scope.
The new location in Poland is intended to expand the capacities of the team in Gothenburg, according to ZTE CEO Giovanni Lanfranchi. Both companies want to present a prototype of a navigation app at CES in Las Vegas at the beginning of January.
Mercedes Intelligent Notes
MBUX Notes is Mercedes‑Benz’s new speech-enabled note-taking app. Developed in-house, it enables hands-free, multilingual voice notes—grocery lists, meeting agendas, capturing and organising ideas for a blog or renovation project, whatever might come to mind while driving.
It is said to be one of the first in-car notetaking apps using generative ‘AI’, and Mercedes says it gives customers a safer and more comfortable way to use their drive time more productively. With end-to-end encryption between the car and backend, the anonymized voice notes are sent to the Mercedes‑Benz Intelligent Cloud, where they’re summarized and structured using OpenAI’s GPT-4o generative ‘AI’.
Once a user has finished a note, they can ask MBUX Notes to read back the summary to them before sharing the notes via an expanding selection of email clients, so the notes are easily accessed outside the vehicle. MBUX Notes can also be connected to a growing list of other popular notetaking and productivity apps.
Ford In-Car Advertisement System
Ford has filed a patent for an in-vehicle advertisement system. Ford’s system, described in patent
US2024/0289844A1, would present advertisements visually on infotainment screens and audibly through car speakers. It would tailor the ads to individual drivers by analyzing factors including the vehicle’s destination and route.
In May 2021, Ford filed a patent for an in-vehicle advertisement system that would use a vehicle’s cameras to read roadside billboards and subsequently display targeted ads related to those billboards’ content on the infotainment screen. Then, in April 2023, the automaker published another, similar patent for a system that could also display ads and images on a vehicle’s windows. Now, this newly published patent gives idea for an in-vehicle advertisement system that may be used in future Ford vehicles, too.
These days, consumers are bombarded by advertisements—many targeted, based on their search history and interests or demographics—and automakers like Ford are exploring even more ways to push ads into our lives. If this patent comes to fruition, our cars will no longer offer any respite from targeted ads. Things like a vehicle’s destination, location, speed, drive mode, or surrounding traffic will determine how many and what kinds of ads to display.
Such a system would optimize ads for companies paying for them, but Ford notes in the patent that the number of ads could be tailored to a user’s preference, as well as the history of how they interact with certain ads.
Other Applications
Car infotainment systems have seen a variety of unusual and sometimes quirky apps. Here are a few examples:
- Pet monitoring: some systems allow you to monitor your pets in the back seat using in-car cameras.
- Meditation and relaxation: these apps provide guided meditation sessions or relaxing sounds to help reduce stress while driving.
- Gaming: While not safe for use while driving, some systems include simple games that passengers can play.
- Weather radar: beyond basic weather forecasts, these apps provide detailed radar images and severe weather alerts.
- Economical-driving coach: apps analyze your driving and offer tips to improve fuel/electricity efficiency.
Further integration of all these ‘AI’-based tools and apps will bring the ‘smartphonization’ of cars.
