category: literaturenote citekey: pawarartificialintelligenceautomatedsolution2025 title: Artificial intelligence automated solution for hazard annotation and eye tracking in a simulated environment authors: "Pawar, Piyush; McManus, Benjamin; Anthony, Thomas; Yang, Jingzhen; Kerwin, Thomas; Stavrinos, Despina" year: 2025 date: 2025-00-00 2025 doi: 10.1016/j.aap.2025.108075 publication: "Accident Analysis & Prevention" url: "https://doi.org/10.1016/j.aap.2025.108075" zotero_key: HEE62XDW zotero_storage: 4GMRNXJX collections: transpordi_planeerimine folder: 001_artiklid firstAuthor: "Pawar, Piyush"

status: converted

Contents lists available at ScienceDirect

Accident Analysis and Prevention

journal homepage: www.elsevier.com/locate/aap

Artificial intelligence automated solution for hazard annotation and eye tracking in a simulated environment☆

Piyush Pawar a,* , Benjamin McManus ^a , Thomas Anthony ^b , Jingzhen Yang ^c , Thomas Kerwin ^d , Despina Stavrinos ^a

^a Institute for Social Science Research, University of Alabama, 306 Paul W. Bryant Drive East, Tuscaloosa, AL 35401, USA
^b Analytical AI, 1500, 1st Ave. N, Birmingham, AL 35022, USA
^c Department of Pediatrics, College of Medicine, The Ohio State University, Center for Injury Research and Policy, Abigail Wexner Research Institute at Nationwide Children's Hospital, 700 Children's Dr. RBIII-WB5403 Columbus, OH 43205, USA
^d Ohio State University, Driving Simulation Laboratory, The Ohio State University, 1305 Kinnear Road, Suite 194, Columbus OH 43212, USA

ARTICLE INFO

Keywords: Deep Learning Driving Hazard detection Translational Research Driving Simulation Gaze Tracking Object Detection

ABSTRACT

High-fidelity simulators and sensors are commonly used in research to create immersive environments for studying real-world problems. This setup records detailed data, generating large datasets. In driving research, a full-scale car model repurposed as a driving simulator allows human subjects to navigate realistic driving scenarios. Data from these experiments are collected in raw form, requiring extensive manual annotation of roadway elements such as hazards and distractions. This process is often time-consuming, labor-intensive, and repetitive, causing delays in research progress.

This paper proposes an AI-driven solution to automate these tasks, enabling researchers to focus on analysis and advance their studies efficiently. The solution builds on previous driving behavior research using a highfidelity full-cab simulator equipped with gaze-tracking cameras. It extends the capabilities of the earlier system described in Pawar's (2021) "Hazard Detection in Driving Simulation using Deep Learning", which performed only hazard detection. The enhanced system now integrates both hazard annotation and gaze-tracking data.

By combining vehicle handling parameters with drivers' visual attention data, the proposed method provides a unified, detailed view of participants' driving behavior across various simulated scenarios. This approach streamlines data analysis, accelerates research timelines, and enhances understanding of driving behavior.

1. Introduction

Use of driving simulators and sensors has expanded the data-driven approach in transportation research. Unique research-specific questions in a driving scenario can be designed involving a human subject participating through the simulation and collecting data on their performance and behavior. This has allowed researchers to understand and assess a driver's cognitive functions and visual behavior during driving to better understand the behavior surrounding potential hazards in a variety of driving situations. For such simulation-based research, the use of big data science and analytics is a major multi-disciplinary field, making use of the plethora of simulator data collected through these experiments. A research study design encapsulates the entire process from conducting the experiments to the dissemination of its final analysis and results. The researcher is often required to perform advanced and in-depth analysis that can require the data to be used in different forms and to need intricate pre-work done on the data before it can be used for analysis to answer the research questions (Pawar et al., 2020, 2021).

In driving behavior studies, a very common and important factor is visual behavior relative to the location of hazardous elements in the environment. Specifically, where and when do participants visually attend to driving environment stimuli, particularly roadway elements that may present a hazardous event, during these roadway simulations

^☆ This article is part of a special issue entitled: 'RSS 2024' published in Accident Analysis and Prevention.

^* Corresponding author.

E-mail addresses: ppawar@ua.edu (P. Pawar), bjmcmanus@ua.edu (B. McManus), thomas@analyticalai.com (T. Anthony), ginger.yang@nationwidechildrens.org (J. Yang), kerwin.6@osu.edu (T. Kerwin), dstavrinos@ua.edu (D. Stavrinos).

that emulate a real-world situation? This is done to understand the participant driver's behavioral and visual interaction with these elements. These could be roadside hazards such as a car parked by the side of a highway or a dangerous event such as a pedestrian crossing a city road (Stavrinos et al., 2021). Annotating these elements and combining them with additional cognitive data is an important task in these studies. However, to do this manually from frame to frame throughout the simulation duration for several experimental drives is a difficult process and could take the researcher months or years to complete these tasks, depending on the methodology and software available. With the use of Artificial Intelligence techniques and higher-performance computing, it is possible to employ machine learning and deep learning algorithms. These methods are computationally heavy mathematical calculations that can process large amount of data, both tabular data such as log files as well as non-tabular data such as images and video data, thus being able to perform the meticulous task of hazard annotations (Chai et al., 2021).This process can be automated for several data samples, and further combined with gaze tracking output, furthering the possibilities of the research studies.

The current work aims to develop a programmatic solution to examine simulated events of hazardous driving environment objects around a participant to quantify their visual reaction and attention to them. This requires using the recording from simulated driving data, detecting these specific objects in the simulation that are programmed as hazard events, and annotation of these objects on every frame of the simulation recording by drawing a bounding box around the object. Then, using the eye tracking data, the gaze projection of the participant on the screen is overlayed to these frames. The combined output of solution is to have annotations of hazards and gaze tracking overlayed on a frame-by-frame basis, enabling the researcher to conduct analysis and derive calculations relating to the participants' attentiveness and reaction.

In the field of computer vision and Artificial Intelligence, this problem can be defined as an object detection problem (Chai et al., 2021). The solution described in this paper employs the implementation of deep learning models; these are mathematical models where several computational units are interconnected in a multi-layered architecture like a network. These models can process images within this architecture to perform object detections tasks and further operations can be done to add annotation labels around the detected objects. The gaze tracking parameters from the eye tracking data files are transformed and then combined with these hazard annotated frames. This entire process is automated and can be performed by programmatically executing the solution that provides a higher level of efficiency and accuracy in these research tasks as compared to doing it manually and alleviating the workload on the researcher.

2. Methodology

2.1. Driving simulator and eye tracking

The development of this solution includes several steps to prepare and process various data files acquired from the simulator. These include tabular data as well as image and video data acquired from the simulation drives. The development environment requires several programming packages and libraries to be set up and compiled correctly which are used for the calculations performed by the deep learning models to process the image data. The results of these models are modified to fit the application of our study and combined with additional data for further calculations.

Data utilized are from an ongoing National Institutes of Healthfunded longitudinal study, Longitudinal Assessment of Driving After Mild TBI in Young Drivers (Yang et al., 2024). Each research site of the study utilizes a high-fidelity, immersive, full-cab driving simulator hardware and software developed by Realtime Technologies Inc. Data from one site are utilized for the current work discussed herein. The driving simulator has a fully functioning interior typical of real-world vehicles (brakes, accelerator, steering wheel, instrument cluster, etc.). The cab is on a motion base to provide realistic pitch cues with braking and acceleration. Three 80″ screens provide a 180◦ field of view to the driver. An additional screen behind the cab, as well as LCD screen in the side mirrors, display the rear view to the participant's car with respect to their position in the simulated world. SimCreator and SimCreatorDX software created graphical models of various objects and terrains into a continuous graphically rendered world and are synched across different servers that handle the projection and execution of this simulation across the various screens and the main simulator cab. It runs at 60 hz frequency with realistic environmental sounds of traffic, weather, and vehicles. Smart Eye Pro eye tracking cameras and software are integrated with the driving simulator software stream and run at 60 hz. The eye tracking software detects and monitors the participant's eye patterns and movements during a simulated drive using three camera sensors placed across the dashboard of the car that records various eye parameters such as pupil movement, gaze movement, etc., by creating a virtual three-dimensional coordinate system in the simulator space and projecting the participant's gaze on the simulator screen. All the systems and sub-systems of the simulator are time synched. These systems are connected on the internal local network of the lab and time sychronization between them is acheived using network time protocol (NTP). A seperate server is used to acquire real life time that acts as the NTP server role and the simulator systems act as NTP client role that communicate with this server to sync their system time across the internal network. This allows all the simulator systems to be synched on the same system time. Before each experiment, the simulator systems are caliberated to make sure these work in tandem with each other. The data from each experimental drive are saved on the local host system and transferred to data servers for storage and further analysis.

Participants drive a total of four simulated scenarios along with a standardized practice drive (Yang et al., 2024). Two of the four drives contain a series of hazardous events, and the data processes discussed herein utilize these drives. See "R2DRV: Study protocol for longitudinal assessment of driving after mild TBI in young drivers" (Yang et al., 2024) for full details related to all four simulator drives. Each driving scenario is approximately 8 min in length, with participants instructed to maintain 45 miles per hour. Participants begin the scenario with instructions to remain behind a lead vehicle that brakes at random intervals. Once the lead vehicle completes its final braking event, it pulls off the simulated roadway, and the participant driver continues driving forward without impedance. Participants perform general driving maneuvers (e. g., regulating speed, direction, lane positioning) but encounter three randomly presented events that require an immediate driving action (e. g. braking) to avoid colliding with the object presented in the event (e.g., car pulling into the roadway). The data files from an experiment drive of this study consists of several files that can be categorized as drive files (containing vehicle driving parameters) and eye files (containing eye movement parameters).

2.2. Development environment

Anaconda is a programming development and management platform software that allows one to create and host a virtual environment specific to the requirements of the application that is being developed. These virtual environments are customized to the required dependencies and packages needed for complex operations implemented in this solution. The platform uses an interactive coding document called Jupyter Notebook, which features modular development and code testing of the program with a graphical interface that allows for visualization of data and computational charts; it can also display image/video data, which is useful for testing the labeling and draw functions on the image (O'Hara et al., 2015).

Artificial Intelligence solutions use deep learning techniques that require complex mathematical calculations that require powerful

computing resources. The Nvidia-Cuda toolkit libraries are a set of instructions that make these computations possible with the use of a graphical processing unit (GPU). The Cuda libraries can make use of several cores of a GPU to perform multi-dimensional matrix computations that are required in deep learning operations. We use Python programming language for coding and building the functions for our A.I. solution; it is a general cross-platform language that has an extensive range of tools and libraries that focus on using Cuda instructions. Pytorch and Tensorflow are open-source packages that consist of deep learning libraries and provide a high-level Application Programming Interface (API) for the Python language that can be used for building deep learning models. Python also includes Pandas libraries that focus on big data science and analytics, allowing developers to deal with and operate on large datasets (Raschka et al., 2020).

This arrangement of programming tools makes for a powerful development environment for big data science and building A.I. models using structured and unstructured data.

2.3. Data pre-processing

The primary files needed for this solution that contain the experiment information obtained from a single drive test are the drive data file (.dat file) that contains the vehicle driving information, the eye log file (. log file) that contains the gaze movement information, and the video recording file (.mp4) of the experiment drive that records the simulation duration as experienced by the driver participant. These files are stored in their raw format as generated by the simulator and need to be transformed and converted into usable formats. The table structured data files (.dat file and.log file) are imported into the Python environment by using the Pandas libraries and reading them into tabular form called data frame objects using appropriate formatting arguments fitting for each file. These drive and eye data are time series datasets, with each row being recorded at consecutive timestamps throughout the duration of the simulation. This parameter recording is done at 60hz by the simulator; hence the data will have 60 readings at an interval of 16 ms for every single second. Now these data become readable and can be operated on further.

The drive data contain 40 columns, and the eye data contain 315 columns. These two data objects must be merged so that the aligned parameters of driving vehicle behavior and eye gaze movement can be obtained. This operation makes it possible to observe the driving reaction and attention of the participant with intricate detail. The sizes of these datasets present another obstacle as both are of different sizes with the drive data being larger and of longer duration than the eye data. This occurs because the recording of the driving parameters starts sooner by default, while the recording of the eye parameters is initiated by the researcher conducting the experiment when the actual driving test starts. This can be overcome by aligning the rows of eye dataset to its appropriate matching ones in the drive dataset by using the timestamp columns in each and performing a time synched merge of the two datasets. This merging is done to a millisecond detail, and the two datasets are reduced to one combined dataset where the drive and eye data are synched, and the time accuracy of the readings is maintained.

2.4. Deep learning

Human vision can look at an image and instantly recognize objects and their location in the field of view; for computers to perform similarly, we need algorithms that can process the input image with efficiency and accuracy. The implementation of such a solution for the application of object detection in our study is to automate the task of hazard annotations in the simulated environment.

2.4.1. Deep learning methods

Deep Learning is a field of artificial intelligence that uses neural network architectures to process data and perform complex

computations, which has allowed for the development of more complex methods to solve the problem of object detection and segmentation in computer vision. The availability of the high computing graphics processing unit (GPU) has made it possible to perform these complex computations and be able to implement deep learning models for processing image and video data for object detection applications in a practical manner. Our solution requires two major steps for object detection: classification and segmentation. An image can contain several objects belonging to different categories. The process is to detect these objects, recognize the instances of their occurrences across the image and assign them to a variety of pre-defined object classes, then localize their position in the given image. Advanced neural networks such as Convolutional Neural Networks (CNNs) are deep learning methods used to perform these operations and provide intricate detail and understanding of visual data. Use of these deep learning methods have been crucial in a variety of fields that require visual analysis by adopting CNN techniques to build models for the detection and segmentation of objects in a scene. Driving and traffic analysis is a primary field to study and understand the interaction of the driver and the hazardous elements (Sinha et al., 2018).

CNNs consist of several layers of deep neural architecture that can process multi-dimensional input images. It can extract pixel-level information to determine features and perform image segmentation. Further, region-based techniques such as Recurrent-CNN (RCNN) and Fast-RCNN use high-level convolutional neural networks for object detections by proposing potential regions for segmentation by localizing an object in the image. Its complex architecture divides the task into two steps. In the first step, the model detects the possible regions that contain the objects, then, in the second step, it runs a classifier on these proposed regions of interest. These algorithms can improve detection accuracy by refining the proposed bounding boxes around the objects (Wu et al., 2017; Ren et al., 2015).

2.4.2. You only look once − YOLO

The complex structure of Fast-RCNNs can be complicated for optimization and training. You Only Look Once (YOLO) is a method that can perform the tasks of object detection in a single pass through of its network. It looks at an image once and a single CNN is able to predict multiple bounding boxes for the objects in the scene and class predictions for those objects. Instead of previous methods that use sliding windows to section the image using a classifier and region-based proposal methods that are required to run several times over an image, YOLO uses full images for training and testing which allows it to learn contextual information (Redmon et al., 2016).

This model's neural network consists of several components that combine the different operations for object detection. The convolutional layers in the model's architecture are followed by fully connected layers that predict the bounding boxes and the object class probabilities. It extracts features from the entire image to generate its prediction bounding boxes around multiple objects, which allows it to detect objects of different classes at the same time. This allows the YOLO model to perform efficiently and with very fast processing times while maintaining detection accuracy. Several iterations of the YOLO model approach have been developed to improve on the basic model and provide a complete start-to-end detection runs on the image with improved processing times, making it usable for real-time applications (Redmon et al., 2016; Terven et al., 2023).

2.4.3. Implementation

Pre-trained models of these deep learning techniques allow for rapid prototyping and application development without the need of building and training these models from scratch and allow one to design solutions to tackle the need and requirement of the domain. For the purposes of the current solution, a pre-trained model of YOLOv7 is implemented. This is a high-speed object detector that can be used for real-time application to capture images directly from the camera footage and

process the detections at a high rate with accuracy, performing in the range of 5 to 160 frames per second. In our solution, we use pre-stored data acquired from the simulator, but we want to leverage the high speed and accuracy of this model and develop on top of this implementation to modify its functionality to suit for our purposes. The YOLOv7 is trained on the MS COCO dataset consisting of 2.5 million labelled instances of 91 objects, a total of 328,000 images (Terven et al., 2023; Lin et al., 2014). This pre-training of the model encompasses objects such as cars, pedestrians, traffic lights, etc., that are applicable to the driving and roadway safety domain and for the objective of our application. A pre-built implementation of YOLOv7 is available in Python that can be imported to the Anaconda development environment for testing. Its API provides many supporting functions that can be added as Python packages for use and further development (Terven et al., 2023).

This allows us to build a solution that can seamlessly read in the raw files from the simulator output of different formats and types and process them in the same programming environment. The functions of the data processing and gaze tracking can be combined with the object detection and annotation functions to obtain the results of our solution.

2.5. Hazard detection and gaze tracking

For our test case, we use a sample drive from the study for development of our processes and for testing the outcomes of the solution. In this sample drive, the participant drives through a seven-minute simulation drive. The participant begins the drive and a lead car appears from the left of the participant overtaking from the left lane and moving in front of the participant, who is then instructed to stay behind and follow this lead car. During this driving task, the lead car will apply sudden braking at random intervals and for the participant must maintain their position by appropriately responding. Then, as this task progresses, the lead car eventually pulls over to the side of the street while the participant continues forward.

The video output of the simulation is an.mp4 file and is the primary data file required for hazard annotations. The default video generated by the simulator is formatted into four quarters and contains four different footages: a recording of the main channel of the simulation, a primary camera angle of the driver, and two secondary camera footages. The recording of the simulation is the primary channel that is projected on the center screen and the primary one that the driver sees when driving through the scenario. To use the data file efficiently, the simulation footage is isolated and extracted from the raw file. This is done by using the FFmpeg package in Python programming for video processing and producing a cropped output video file that only contains the simulation main channel. This cropped video file can now be directly used as input to the YOLO model functions, where the video is read frame by frame and object detection is performed on each frame. These functions are to fit the outcome of the current solution. Instead of a general detection of every object in the simulated environment, in this sample test drive we are only concerned with the car object as hazard; the same approach is applicable for any other object of interest or object of significance based on the design of the scenario and the driving task. The model performs object detection only for car objects. In this sample test drive, one car object of similar model appears in the view of the participant at any given time, so the outcome of detections is specific to this type of car as it appears in the view of the participant. A four-point bounding box made of a standard rectangle shape is used for annotating around the object after it has been detected as specified in the simulation frame. The car object continues to be detected as long as it is in the frame view of the participant. Even as the lead car slows down and comes to a halt on the side of the street, it continues being detected by the algorithm from the side profile till it is out of the view of the video frames. The car hazard detection is performed and annotations are added to each frame for as long as the hazard is in view of the driver, as shown in Fig. 1. In addition to overlaying the bounding boxes to annotate the object in the frames of

Fig. 1. Hazard annotation bounding box.

the simulation video, the object detection process also saves the object type, confidence values for detection, processing time and the bounding box coordinates of each frame respectively into a dataset.

The Smart Eye system can perform gaze tracking from the participants' eye movements by creating a virtual coordinate system in the simulator space with the participant driver at its origin. This coordinate system is a three-dimensional space; however, since the simulation broadcast is done on a flat screen, the gaze is projected using two coordinates, the projection of both the left and right eye is combined into a single convergence, and the gaze is projected as a single point (x, y) on the screen. This information is extracted from the eye dataset, which contains a recording of the gaze point throughout the simulation duration at every time instance. The gaze points in their raw form are available in the range of − 1 to 1 for both x and y axes. These values need to be scaled to an appropriate range to match the dimensions of the frames extracted from the video of the center screen, which is of dimensions 500 x 940. These scaled values for gaze projection are then aligned with their respective frames extracted from the input video and combined with the hazard detections. Further functions are built to apply the gaze points on the frame image using draw methods along with the hazard annotations. Fig. 2 shows the output of combined hazard annotation and gaze tracking. The gaze is represented by a single gaze point on the frame, then a circle is drawn with the gaze point as the center to highlight the gaze area.

The hazard annotation functions and gaze tracking functions are modular and can be operated independently to get separate or combined results for testing and analysis.

2.6. Gaze metrics

The gaze metrics are measures of time and instances that can be used to describe the gaze behavior of the participant as they are driving through the simulation and in response to the object during the scenario.

Fig. 2. Hazard annotation and gaze tracking.

3. Analysis and results

With combined output of hazard annotations and gaze tracking, the intersection of the participant's gaze and hazard objects throughout the simulation can be observed. The gaze area circle can be used to estimate partial or complete overlap of the gaze and hazard bounding box or no overlap in case of the gaze straying away from the hazard. The gaze area circle is used to account for the periphery vision of the participant's gaze projection. The size of this circle is determined by several tests of different sizes, as larger size circle would tend to give more false positives, and a smaller size circle would tend to give more false negatives. So, the size of the gaze circle was reached by considering an optimum trade off. Using the overall merged and aligned datasets, we can obtain parameters such as frame time, object detection confidence, and intersection points and assign these to each respective frame for a frame-byframe analysis at each instance of the simulation. Figs. 3 and 4 show the outcome of two situations with no intersection of gaze and hazard and the case with intersection, respectively.

A more extensive analysis can also be done by doing frame analysis over a larger duration. Hazard events are periods in the scenario where a hazard object is activated and performs a distractive or potentially threatening maneuver to hinder the participant. Based on how this event is designed in the scenario, this event can last for a few seconds to a minute. Using the gaze and hazard intersections on the frames over the duration of such an event can allow us to analyze and understand the participant's reaction behavior. Fig. 5 shows a representation of a sample hazard event that we used for testing from its starting point to the endpoint. The simulation frames and the related data during this event are sectioned off separately and then can be operated further.

The section of hazard event is filtered out from the entire data to operate separately, which contains data from the start of the hazard event until the end of the hazard event. Using intersection instances throughout the event, this period is sectioned further into smaller segments described as looks and glances. These segments are moments when the participant looked at the hazard or kept their gaze on the hazard for a longer period. Timestamps are used to calculate the duration of these segments to get a millisecond level detailed time for these segments.

Table 1 contains the output of the gaze calculations performed for the sample test hazard, and these values can be stored as outputs relating to this event in a usable format for further analysis. A complete simulation could have several hazard events, and gaze metrics can be calculated for each of the events separately. In cases where the participant's gaze does not intersect with the hazard, the metrics are estimated with zero or null value as applicable, which are also valuable to understand the participant's distraction and attention behavior. These metrics are programmatically estimated in the same program by making separate functions for these calculations and can be modified according to the research protocols and guidelines of the study. These metrics are then stored in a tabular form that can be used by the researchers for further analysis and estimations.

Fig. 3. No gaze-hazard intersection.

Fig. 4. Gaze-hazard intersection.

Fig. 5. Hazard duration frameset.

Table 1 Gaze metrics.

Gaze metric	Definition
Long Glance	An event when the participant's gaze intersects with and remain on the object for a duration of more than 2.0 s (Klauer et al., 2006)
Short Glance	An event when the participant's gaze intersects with and remains on the object for a duration of longer than 0.75 s but less than 2.0 s
Look	A look is an event where the participant's gaze intersects with the object and remains on the object for any duration shorter than 0.75 s.
Glance reaction	The duration from the occurrence of the object to the first
time	glance event
Look reaction	The duration from the occurrence of the object to the first look
time	event
Percent glance	Percentage of time a participant looked at the object during a
time	hazard window/object appearance
Average Glance	The average time a participant glanced at the object divided by
length	the total time the object was presented
Total Glance	The total amount of time a participant looked at the object
length	during the hazard event/object appearance window

4. Discussion

This solution is a program that can process several data files generated by the simulator systems and sub-systems. It performs several preliminary tasks of merging and alignment to make the data usable and transforms it from its raw form to a user-readable format. The program performs the primary tasks of hazard annotations and gaze tracking effectively and combines these on the visual frames of the simulation. Further gaze metrics are calculated to make estimations for the participant's driving and attention behavior (Table 2).

In comparison to the previous solution published as "Hazard Detection in Driving Simulation using Deep Learning" (Pawar et al., 2021), this solution has improved processing times, and unifies the primary tasks of hazard annotation, gaze tracking, and gaze calculations

Table 2 Sample test gaze metrics.

| Gaze parameter | Value | | |-----------------------------|-------|--| | No. of Total Looks | 20 | | | No. of Total Glances | 1 | | | No. of Long Glances | 0 | | | No. of Short Glances | 1 | | | Glance Reaction Time (sec) | 9.699 | | | Look Reaction Time (sec) | 3.767 | | | Percent Glance Time (%) | 7.159 | | | Average Glance Length (sec) | 1.001 | | | Total Glance Length (sec) | 1.001 | |

in a single program with better performance. The primary benefit of this solution is the integration of the hazard annotations with gaze tracking and gaze calculations, which efficiently expands the overall results and outcomes of the solution. Table 3 presents the comparisons of processing times using the values of the current solution and estimations from the previous solution (Pawar et al., 2021) for this test hazard event that consists of 457 frames.

This solution is built and operated within its constraints and harbors certain limits. The use of deep learning classification of objects to detect them correctly can lead to a few misidentifications of objects. The models used in the solutions are pre-trained on photo-realistic image data for a general pool of objects and are adapted to be used for graphically rendered simulation, which can take away certain key features of realistic object identification. During testing for the sample hazard event, the model correctly identified the hazard object 394 times and misidentified or dropped detections 63 times, leading to 86.214 % of correct detections in this test. The performance of detections can be improved by using annotated frames for training the model further to fit the model's detection features toward specific hazards of the simulation as defined in the study, such as a blue car or a grey building. Additional functions can be built to augment missing detection frames by interpolating the bounding box annotations between consecutive correctly detected frames.

This solution was built using sample data from one driving simulation study, but the operations performed in it are applicable to other driving simulation-related studies that use a cab simulator and gaze tracking. The functions developed in the program are modular and can be used to get results according to the requirements of the researcher's analytical needs. The primary functions in this solution can be extrapolated and modified to perform other similar tasks of object detections/ annotations and gaze tracking as defined by the study outline to fit the guidelines of the research study. On a large scale, the solution can be made more robust by training and testing over a large scale of data using several drive experiments over the course of the study.

5. Conclusions

Automated AI solutions can be a powerful tool for driving research. Deep learning applications can be developed to design operations that are essential to the study. Some of these tasks may not be the domain of expertise for the researcher and require several hours of repetitive manual labor, so these tasks can be automated. The solution discussed in this paper serves as a proof of concept for the kind of automation methods that improve and progress the research toward achieving the goals and outcomes of a study.

The solution developed in this paper can perform several tasks that are required to achieve the outcomes of the study. It can perform these tasks effectively and fast, furthering the progression of the research and making it possible to implement operations that would be very difficult or not possible to do manually. It significantly alleviates the labor on the researcher, who can focus their efforts on more creative and meaningful analyses. This also opens possibilities to answer research questions that would otherwise not be feasible in the scope of the study. This solution

Table 3 Solution Comparison.

Parameter	New Solution	Previous Solution	Manual Operation
Avg. Processing time for single frame	77.926 msec.	6 sec.	1.5 min.
Avg. Processing time for a	35.612 sec.	45.7 min.	11.425 hrs.
hazard event		(estimated, 6 *	(estimated,
		457)	1.5 * 457)
Gaze Tracking	Available	N/A	N/A
Gaze Calculations	Available	N/A	N/A
Gaze Computation time	369 msec.	N/A	N/A

accelerated the processing time of the data and provided analysis-ready datasets, reduced limited datasets for specific analysis and datasets with additionally derived data such as the gaze metrics. These datasets can be directly used by the researcher to make the domain and research specific analysis and imported into other analytical tools such as SAS or SPSS.

Funding Sources

Research reported in this publication was supported by the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health (R01HD098175), The Ohio State University Chronic Brain Injury: Discovery Theme Initiative Award, and the University of Alabama at Birmingham College of Arts and Sciences Interdisciplinary Team Award. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funders.

CRediT authorship contribution statement

Piyush Pawar: Conceptualization, Formal analysis, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing. Benjamin McManus: Formal analysis, Methodology, Visualization, Writing – original draft, Writing – review & editing. Thomas Anthony: Conceptualization, Formal analysis, Methodology, Visualization. Jingzhen Yang: Funding acquisition, Investigation, Project administration, Resources, Supervision. Thomas Kerwin: Conceptualization, Formal analysis, Methodology, Visualization. Despina Stavrinos: Funding acquisition, Investigation, Project administration, Resources, Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

Data from this study is unavailable to access as it contains confidential participant information.

References

Chai, J., Zeng, H., Li, A., Ngai, E.W.T., 2021. Deep learning in computer vision: a critical review of emerging techniques and application scenarios. Mach. Learn. Appl. Klauer, S.G., Dingus, T.A., Neale, V.L., Sudweeks, J.D., Ramsey, D.J., 2006. The Impact of Driver Inattention on Near-Crash/Crash Risk: An Analysis Using the 100-Car Naturalistic DRIVING STUDY Data (No. HS-810 594). National Highway Traffic Safety Administration, Washington, DC.

Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, ´ P., Zitnick, C. L., Microsoft COCO: Common objects in context. In Computer Vision–European Conference on Computer Vision, 2014. 13 p. 740-755.

O'Hara, K.J., Blank, D., Marshall, J. 2015. Computational notebooks for AI education. Pawar, P., Anthony, T., McManus, B., Stavrinos, D., 2020. Data reduction solution for driving simulator. SoutheastCon 1–7.

pawarartificialintelligenceautomatedsolution2025.md 47 KB Historik Rå