A new system developed by the Fraunhofer Institute for Optronics, System Technologies and Image Exploitation IOSB is thought to be the world’s first to be able to use image data to draw conclusions about the driver’s activity and analyse how quickly they would be able to take control of the vehicle.

Researchers at Fraunhofer IOSB are focusing on the interaction between the driver and the vehicle — and working to bridge the gaps. “Our technology not only enables us to identify facial features but also the current poses of the driver and the passengers,” says Michael Voit, Group Manager at Fraunhofer IOSB. “We can then determine from these poses what the driver and passengers are currently doing.”

AI interprets driver activity

At the core of the development lie machine learning algorithms and processes, such as Artificial Intelligence (AI). Algorithms analyse the camera data in real time to find out whether the driver is on the phone, playing with children or looking at a passenger’s cellphone. IOSB technology therefore goes beyond image recognition and interprets the activities in context. The researchers first trained the system by manually annotating numerous camera shots: Where are people’s hands, feet and shoulders? Where are objects such as smartphones, books and other items located? They then evaluated the algorithms using new images and corrected or verified their results.

Video, infrared and 3D cameras can be supported

The system abstracts images of the driver or passengers to form a digital skeleton — a type of stick figure that replicates the person’s body poses. It consequently deduces the activity using the skeletal movement and supplementary object recognition. “The algorithms can thereby tell whether someone is sleeping or looking at the street, how distracted the person is and how long it will take them to focus back on the road,” explains Voit. The system supports both traditional video cameras and infrared cameras that can see in the dark, as well as 3D cameras that measure the distance between objects and the camera. The system even gives interior designers freedom in terms of camera placement.

The researchers have been dealing with issues related to activity detection in vehicle interiors in numerous research projects, collaborating with renowned car manufacturers such as Audi and Volkswagen, as well as with suppliers such as Bosch and Continental. The projects have been supported by the German Federal Ministry of Education and Research (BMBF), the German Federal Ministry for Economic Affairs and Energy (BMWi) or the Federal Ministry of Transport and Digital Infrastructure (BMVI). “We can not only detect the activities of the driver but those of all passengers, too — both in the front and back of the vehicle,” confirms Voit. “The technology is ready for pilot production. We are already in contact with companies who want to use our technology.” Linking the developments is the institute’s own driving simulator, which industry customers can also benefit from in the context of individual R&D projects. Simulating traffic situations, it forms the basis for collecting relevant driving and behavioural data. Furthermore, its extensive sensor technology means that the behaviour of all passengers can be studied.

Data protection is a priority

As far as data protection and security aspects are concerned, the researchers made this a priority from the start. “The camera data are analysed in real time, not saved and do not leave the vehicle at any point. Personalised models are also not needed for this, so no personal data is collected,” says Pascal Birnstill, Senior Scientist at Fraunhofer IOSB, who specialises in data security, data protection and transparency. The technology respects privacy from the outset and thus complies with the strict regulations and high level of data protection awareness in the EU.

Numerous applications 

A new EU regulation highlights how important activity detection is; driver monitoring is to become mandatory in automated cars, no matter their level of automation. Using Fraunhofer IOSB technology, vehicle manufacturers can not only meet this requirement but also make numerous visions in terms of autonomous driving a reality. For example, voice recognition for communication between people and cars is fast approaching its limits. The command “Park there” is not meaningful in itself. However, when used in conjunction with body pose identification and activity detection, the system can determine which parking space the user is pointing to at that very moment. The system can also help with safety aspects of driverless vehicles. While it is currently still the driver’s responsibility to ensure that all passengers comply with safety regulations and fasten their seatbelts, for example, this will soon be the job of the driverless vehicle — one example being autonomous taxis or buses. Reliable monitoring of the vehicle’s interior is also essential for this.