Figure 1. Computer vision approaches identify and separate vehicles in a street scene automatically for highway safety analysis.
Highway transportation researchers are collecting and analyzing an increasing amount of video data that provide new and enhanced visual and other sensors for conducting research in multiple areas such as system planning, operations, safety, and infrastructure condition assessment. While the research community is fortunate to be able to collect more and better data, the amount of data has the potential to overwhelm the capacity to assess the data using current methods. Automating data extraction from video files is expected to dramatically reduce the costs of using these data, making them accessible to the widest possible pool of researchers. Federal Highway Administration (FHWA’s) Exploratory Advanced Research (EAR) Program, working with the Office of Safety Research and Development (R&D) is sponsoring six research projects that will explore breakthroughs in machine learning to automate extraction of safety data from the naturalistic driving study (NDS) such as the Strategic Highway Research Program (SHRP2) project. More information about four of the projects can be found on the links at the end of this article.
Figure 2. Video data include cabin and roadway views. This view illustrates possible relationships between observed driver behavior and the roadway context outside the vehicle, which includes other vehicles, cyclists, pedestrians, and lane markings.
- Eye tracking technology now allows for more natural behavior and field applications. Examples of research using this technology include the EAR Program-sponsored research projects, “Seeing in the Dark Improving Understanding of Driver Visibility Requirements at Night,” [more] and “Development of Methodologies to Evaluate the Nighttime Safety Implications of the Roadway Visual Scene Under Varying Cognitive Task Loads.” [more]
Figure 3. Privacy can be a barrier for sharing video data of drivers among researchers. New methods for obscuring facial features while preserving critical features such as head pose, eye gaze direction, and facial expressions can allow for greater access.
- Increased use of traffic cameras and improved analysis software are providing data on vehicle classification, weight, speed, routing. Road-based and vehicle-based cameras also are providing improved recognition of pedestrians, e.g. the EAR Program-sponsored project, “Real-Time Pedestrian Detection Layered Object Recognition System for Pedestrian Collision Sensing.” [more]
- Use of LIDAR and improved analysis software are assisting with the cataloguing of road and roadside conditions including pavement surface conditions, roadside topography, and roadside hardware.
The SHRP2 safety area includes elements of the above types of data in the NDS). The amount of data can be orders of magnitude larger than what the highway researcher has worked with previously with research results expected to exceed one petabyte. The NDS involved collection of data on the daily travels of 3,150 volunteer drivers, whose vehicles were heavily instrumented for the study. Those drivers traveled 49.5 million miles during the study period, resulting in over 1.2 million hours of video and vehicle data. See the SHRP2 Data Access website for more information.
FHWA is also working with experts at Oak Ridge National Laboratory to test the research results and develop methods and test data sets for future researchers to use in comparing and validating the performance of automated data extraction algorithms.
Current methods for assessing the data include a mix of automated and manual, frame-by-frame coding that is not able to manage massive data stream and provides results that are not consistent or error free enough. Large data sets in their present form are too time-consuming and expensive to analyze using traditional data extraction methods, yet they are important for providing understanding of the context around rare events such as crashes.
Figure 4. This graphic shows a user interface for viewing data streams including in cabin video and roadway video.
More information on the projects is available at the links below:
- Machine Learning for Automated Analysis of Large Volumes of Highway Video.
- DCode: A Comprehensive Automatic Coding System for Driver Behavior Analysis.
- DMask: A Reliable Identity Masking System for Driver Safety Video Data.
- Driver Behavioral Situational Awareness System (DB-SAM).
- Video Deidentification in the Automobile Environment
- Quantifying Driver Distraction and Engagement Using Video Analytics