홈 탐색 도움말
로그인
ardo
/
Obsidian
1
0
포크 0
파일 이슈 0 풀 리퀘스트 0 위키
트리: a0e0ef701f
브랜치 태그
Obsidian
/
Zotero
/
001_artiklid
/
Road Safety Indicators
/
roydeepreinforcementlearningbased2022.md

roydeepreinforcementlearningbased2022.md 88 KB
히스토리 Raw

category: literaturenote citekey: roydeepreinforcementlearningbased2022 title: A deep reinforcement learning-based intelligent intervention framework for real-time proactive road safety management authors: "Roy, Ananya; Hossain, Moinul; Muromachi, Yasunori" year: 2022 date: 2022-00-00 2022 doi: 10.1016/j.aap.2021.106512 publication: "Accident Analysis & Prevention" url: "https://doi.org/10.1016/j.aap.2021.106512" zotero_key: FRI6HVTP zotero_storage: ZVEQZGZD collections: transpordi_planeerimine folder: 001_artiklid/Road Safety Indicators firstAuthor: "Roy, Ananya"

status: converted

Contents lists available at ScienceDirect

Accident Analysis and Prevention

journal homepage: www.elsevier.com/locate/aap

A deep reinforcement learning-based intelligent intervention framework for real-time proactive road safety management

Ananya Roy a,1 , Moinul Hossain b,*,2 , Yasunori Muromachi c

  • a ALMEC Corporation, Head Office Transportation Planning Division (Overseas Department), Kensei Shinjuku Building, 5-5-3 Shinjuku, Shinjuku-ku, Tokyo 160-0022, Japan
  • b Department of Civil and Environmental Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh
  • c Department of Civil and Environmental Engineering, School of Environment and Society, Tokyo Institute of Technology G3-5, 4259 Nagatsutacho Midori-ku, Yokohama 226-8503, Japan

ARTICLE INFO

Deep reinforcement learning

Keywords: Cell transmission model Dynamic Bayesian network Real-time crash prediction and intervention model

ABSTRACT

We propose a variable speed limit (VSL) system for improving the safety of urban expressways in real time. The system has two main functions: monitoring traffic data and then using the data to assess crash risk through a realtime crash prediction model (RTCPM). When the risk is high, the system triggers VSL control to restore traffic conditions to normal. The study addresses several weaknesses in existing VSL-based real-time safety interventions. Existing models are not widely applicable due to varying detector spacing among different freeways, and even within a study area. Therefore, with the existing detector spacing as an input, a cell transmission model (CTM) is used to simulate traffic states for the desired cell size. A dynamic Bayesian network (DBN) is used for modeling in the RTCPM. The proposed CTM model is then modified to allow VSL control. Whereas existing studies selected various VSL strategies from a predefined list, we employ a deep Q-network, which is a reinforcement learning-based machine learning algorithm, for the VSL control. Two busy segments of the Tokyo Metropolitan Expressway were used as the study area. After several iterations, our proposed real-time system reduced the crash risk by 19%.

1. Background

A real-time proactive road safety management system (RT-PRSMS) is capable of monitoring and evaluating the road safety condition by instantaneously sensing the traffic states, introducing appropriate intervention upon detecting signs of the formation of a hazardous traffic condition, varying its evaluations and/or interventions in response to changing situations and past experience, and updating itself through its inherent capability to adapt to new situations (Hossain and Muromachi, 2011). Such a system broadly comprises a real-time crash prediction model (RTCPM) and real-time interventions. RTCPMs calculate the probability of a crash occurrence within a short period in the near future and for a specific road section, mainly using instantaneous traffic state data and their descriptive statistics. Then, the model outcomes can assist the drivers by warning them and/or suggesting that they adopt an appropriate driving strategy. In short, the outcomes of an RT-PRSMS

will be reflected through a reduction in the number and severity of crashes. As a proven technology, variable speed limit (VSL) control is superior to other interventions, and its combination with an RTCPM as a proactive real-time road safety management system dominates the literature.

Being in its infancy, several aspects of real-time proactive road safety management need to be improved before practical use. First, the data used in the RTCPM is macroscopic. However, apart from Roy et al. (2018a, 2020), the existing interventions in the literature follow an indirect path, where microscopic models are calibrated to reproduce the macroscopic environment, using a microscopic traffic simulation tool or driving simulator. Second, most of the existing literature is limited to recognizing the pattern of a hazardous traffic state and then identifying the best solution from a predetermined set of VSL-based interventions to improve safety. Hence, it lacks 'intelligence,' i.e., the ability to learn by observing how traffic states change in response to a particular set of

E-mail addresses: moinul@iut-dhaka.edu (M. Hossain), ymuro@enveng.titech.ac.jp (Y. Muromachi). 1 ORCID: 0000-0003-4160-665. 2 ORCID: 0000-0002-1178-9516.

* Corresponding author.

interventions and consequently formulating new and improved interventions. This drawback was partially addressed by Li et al. (2020); they developed a basic Q-learning (QL)-based reinforcement learning (RL) algorithm to apply the variable speed limit (VSL) through a cell transmission model (CTM), which reduced the rear-end crash risk by 19.4% under lower driver compliance situations on Interstate 880 in California, USA. The study also employed a logistic regression-based RTCPM, which is not inherently a real-time modeling method. Third, the studies on RTCPM are characterized by an imbalance between crash and non-crash data, which are large in volume and exhibit heterodox characteristics rarely addressed in the existing literature (Formosa et al., 2020; Yu et al., 2020; Peng et al., 2020). Finally, the existing studies are by nature proofs of concept and cannot be deployed independently to improve real-time safety. In the existing solutions, the microsimulation, RTCPM, and the set of rules for applying VSL control are independent components. However, a real-time proactive system must be able to apply VSL control, evaluate its efficacy, and then seamlessly alter the strategy instantaneously.

The first issue can be addressed by introducing a cell transmission model (CTM) (Daganzo, 1995), as demonstrated by Li et al. (2016, 2017, 2020) and Roy et al. (2018a, 2020), i.e., replacing the microsimulationbased practice with a macroscopic traffic simulation approach. The use of a CTM will also increase the transferability as any detector spacing can be taken as input to produce a simulated traffic state dataset for the desired detector spacing. The second issue can be resolved by using reinforcement learning (RL) for the VSL-based intervention design. Abdulhai and Kattan (2003) presented a review of the application of RL as well as its future potential for use in intelligent transportation systems (ITS), such as controlling traffic signals, variable message signs (VMS), and dynamic route guidance systems (ATIS and ATMS). Deep RL, such as the deep Q-network (DQN), is increasingly being used in various technologies that involve variable operating conditions and real-time adaptive control strategies. For the VSL-based intervention design, the DQN is more advantageous than Q-learning (QL) because of its ability to take continuous traffic parameters and scalability. Learned policies and active sensing behaviors employing deep RL exceeded the capabilities of commonly used heuristic approaches in several categories, such as task completion time, goal success rate, and the ability to generalize the problem (Isele et al., 2018). The third issue can be resolved by employing the dynamic Bayesian network (DBN) as a modeling method for RTCPM, which can deal with large, complex, and imbalanced data (Xu et al., 2015). Yang et al. (2018b) proved that this method can increase the crash prediction accuracy by an average of 19.8% compared to other models even when the crash to non-crash ratio increased from 1:1 to 1:10. Moreover, since the DBN can combine prior information and new information, the parameters of the real-time crash risk models can be updated across time and space to improve spatial and temporal transferability. The final issue can be addressed by implementing VSL control into the CTM to instantaneously know the efficacy of the VSL strategies and identify the best strategy.

Addressing the aforementioned research gaps, this study commenced by simulating pre-crash traffic state data using a CTM. Next, a dynamic Bayesian network-based RTCPM was reproduced, following Roy et al. (2018a, 2020). Finally, the CTM algorithm was modified to enable the generation of traffic states for various VSL strategies. This was eventually connected with the RTCPMs to evaluate the real-time safety impact of the proposed VSL strategies and to choose a strategy that minimized the safety risk.

2. Literature review

2.1. Real-time crash prediction models

Although this technology is not yet ready for implementation, substantial progress has been made. Modern-day RTCPMs can use the data from various instruments, such as loop detectors (Yang et al., 2018a–c), microwave vehicle-detection systems (MVDS) (Wang et al., 2017a), remote traffic microwave stations (Wu and Abdel-Aty, 2018), automatic vehicle identification (AVI) systems (Basso et al., 2020), a combination of these, e.g., loop detectors and probe vehicles (Katrakazas et al., 2017), and MVDS and video detectors (Wang et al., 2017b). Among these, loop detector data is widely used due to its long history and low implementation cost (average spacing 0.5 miles) (Wu et al., 2019; Peng et al., 2020; Liu and Chen, 2017). Apart from classic statistical approaches, such as logistic regression (Wu et al., 2019; Basso et al., 2018; Basso et al., 2020), Poisson distribution, and negative binominal distribution (Zhang et al., 2020), cutting-edge modeling methods are also used, such as Bayesian analysis and extreme value theory (Essa et al., 2019; Zheng et al., 2019; Guo et al., 2020), various types of neural networks (Lui and Chen, 2017; Park et al., 2018; Wu et al., 2019), support vector machines (Katrakazas et al., 2017; Wu et al., 2019; Basso et al., 2018; Basso et al., 2020), Bayesian networks (Hossain and Muromachi, 2012), Bayesian meta-analysis (Xu et al., 2015), dynamic Bayesian network (Sun and Sun, 2015; Roy et al., 2018a), Bayesian dynamic logistic regression (Yang et al., 2018b), probabilistic neural network (PNN) (Liu and Chen, 2017), deep neural networks (Yang et al., 2018a; Formosa et al., 2020), region-based convolutional neural network (RCNN) (Formosa et al., 2020), convolutional neural network (CNN) (Cai et al., 2020; Yu et al., 2020), random sampling cost-sensitive multilayer perceptron (RCSMLP) model (Peng et al., 2020), adaptive neuro-fuzzy inference system (ANFIS) (Liu and Chen, 2017), etc. They are to some extent transferable (Park et al., 2018), and offer good accuracy in predicting crashes (Yang et al., 2018b – 94%; Formosa et al., 2020 – 78%) with a low incidence of false alarms (Yang et al., 2018b – 10%; Formosa et al., 2020 – 10%).

2.2. Real-time interventions to reduce crash risk

From the perspective of the road authorities, studies have been conducted by following a specific framework that uses either a traffic simulation program (Lee et al., 2004; Lee et al., 2006; Abdel-Aty et al., 2006a,b; Abdel-Aty et al., 2007a,b; Yu and Abdel-Aty, 2014) or a driving simulator (Park et al., 2018), since human error is the cause of almost 90% of road crashes, to reproduce pre-crash traffic conditions (Lee and Abdel-Aty, 2008; Formosa et al., 2020; Park et al., 2018). Various countermeasures, such as variable message signs (VMS) (Park et al., 2018), variable speed limits (VSLs) (Lee et al., 2004; Abdel-Aty et al., 2006a,b; Abdel-Aty et al., 2007a,b; Lee and Abdel-Aty, 2008; Yu and Abdel-Aty, 2014; Abdel-Aty and Wang, 2017; Song and Wang, 2019), and coordinated or uncoordinated ramp metering (Lee et al., 2006; Abdel-Aty et al., 2007a,b; Abdel-Aty and Gayah, 2010) were applied to successfully bring crash-prone traffic conditions back to normal. Among the early reports, Lee et al. (2004) applied a speed reduction whenever the RTCPMs predicted a high probability of a crash. In such situations, the system instructed drivers to remain within the speed limit if the average speed of the stream was found to be higher. Otherwise, it reduced the speed to 80, 70, 60, 55, 50, or 40 km/h, whichever was the nearest lower speed, compared to the speed of the stream at that time. It was found that the traffic could be stabilized within 5 to 10 min after the VSL was posted. Abdel-Aty et al. (2006b) were successful in gradually introducing speed limit changes every 10 min at a 5 mph rate, abruptly changing the speed limit in space, and reducing the speed limits upstream but increasing them downstream. Several studies (Abdel-Aty et al., 2007a,b) found that VSL control was effective for medium- to high-speed regimes, but had little impact in low-speed situations. Abdel-Aty et al. (2017) introduced a new dimension by suggesting a homogeneous speed zone, above which VSL control is implemented. Lee and Abdel-Aty (2008) applied the VMS and VSL in tandem and postulated that such interventions can reduce speed variations. The study by Abdel-Aty et al. (2006b) also found that although VSL control improved the safety in the implementation zone, the high crash potential was found to be relocated downstream. Acknowledging this, Yu and Abdel-Aty (2014) developed an optimization algorithm to minimize the overall crash risk, encompassing the total VSL implementation corridor. At the same time, they incorporated driving variations by modeling the responses of three different driver compliance levels. Abdel-Aty and Wang (2017) also successfully demonstrated the ability of VSL control to reduce the crash risk for congested weaving sections of the expressway. The VSL coupled with RTCPMs has the potential to reduce both the delay and crash costs (Song and Wang, 2019).

From the vehicle manufacturers' perspective, Park et al. (2018)

presented a system where the probability of a secondary crash was estimated in real time, and the drivers were given warnings to be more vigilant, consequently avoiding possible crashes. In the future, it may be possible for the two institutions, i.e., the road authorities and the vehicle manufacturers, to share data for symbiotic benefits, and coordinate the interventions between the instrumented infrastructure and the vehicular warning and driving strategies. Such a possibility is also expected to flourish upon the introduction of connected and autonomous vehicles

Fig. 1. Detailed map of the study area.

(CAVs) into the network in the near future. Hossain et al. (2019) stated that CAVs in the traffic stream may act as probe vehicles, feeding the RTCPMs with traffic state data, and at the same time, they can adopt driving strategies to influence people to drive in a specific manner that will increase the safety. Li et al. (2020) recently emulated connected vehicle data to estimate the real-time crash potential for arterials.

3. Study area

The Tokyo Metropolitan Expressway's Shibuya Route 3 (11.9 km long) and Shinjuku Route 4 (13.5 km long) are the two routes of this study, which sustain a substantial number of crashes throughout the year. Route 3 has two lanes in each direction with 43 and 41 effective detectors in the inbound and outbound directions, respectively. Route 4 also has two lanes in each direction with 50 and 44 effective detectors in the inbound and outbound directions, respectively (the distance between each detector is roughly 250 m). The fixed loop detectors collect the cumulative vehicle count (veh/min), average speed (km/h), and average occupancy (%) aggregated for every 8 ms, round the clock. However, they were aggregated for every 1 min for this study. The detector data and the matching crash data were collected from both routes during the period from March 2014 to August 2014. Being two of the most important routes of the Tokyo Metropolitan Expressway, any crashes are precisely detected. Fig. 1 provides a detailed map with the direction of the traffic flow, the location of the detectors, and the ramps for both routes.

Source: Supplied by Tokyo Metropolitan Expressway Company Limited

4. Methodology

4.1. Overall work flow

The methodology of this study can be broadly classified into four steps: (i) Development of a cell transmission model (CTM) for the study area; (ii) Development of a dynamic Bayesian network (DBN)-based real-time crash prediction model (RTCPM) using the outcomes of (i); (iii) Implementation of variable speed limit (VSL) control into the CTM model; and (iv) Formulation of a reinforcement learning (RL)-based VSL intervention with a deep Q-network (DQN) to reduce the crash risk. Finally, through the integration of steps (ii) – (iv), the VSL-based proactive road safety management system was developed. The overall approach is illustrated in Fig. 2.

In the first step, the CTM was used to replicate the traffic in the study area and transform the nonhomogeneous detector data input to homogeneous output to extract macroscopic traffic data at specific locations that are spaced at equal distances along the expressway. Then, in the second step, this simulated data was used as input to develop the RTCPM using the DBN. In these two steps, the use of the CTM eliminated the criticism of using nonuniform detector spacing data, which prevails in most of the existing studies, and the application of DBN addressed the issue of making inferences in the absence of data on some critical variables and future inclusion of new parameters without needing to rebuild or fully calibrate the model. Therefore, these steps increased the transferability of the RTCPM models. In the next step, VSL control was implemented into the newly developed CTM model. The methodology was inspired by Li et al. (2016, 2017, 2020). This ensures that if the RTCPM, taking input from the CTM, identifies a traffic situation as hazardous, then VSL control could be applied to the CTM model, which would eventually produce a changed traffic state for evaluation by the RTCPM. In the fourth step, the job of the DQN was to find the most appropriate VSL strategy to be applied to ensure that a hazardous traffic condition could be brought back to normal. The loop in Fig. 2 suggests that for a particular freeway section, whenever the RTCPM identifies the evolution of a hazardous traffic condition, it uses the DQN to select the most appropriate VSL strategy, which is eventually implemented through the CTM. The process continues until the crash risk at that section is reduced to an acceptable limit.

The remaining part of this methodology section briefly introduces the modeling methods employed, i.e., CTM, DBN, VSL implementation into CTM, and RL. Section 5 elaborates on the experimental setups used to extract the data for model implementation and validation.

4.2. Cell transmission model (CTM)

The CTM is a macroscopic approach to simulating the traffic on desired road sections described by nodes and directed arcs, where each arc possesses some physical data, including its length and the parameters defining a flow–density relationship (fundamental diagram; FD) for a steady-state traffic condition (Fig. 3 (a)). The cells are of three types:

Fig. 2. Workflow diagram of the study.

Fig. 3. Fundamental diagram for the CTM with VSL control.

(i) ordinary, (ii) diverging, and (iii) merging. Here, the relationship between the traffic flow (q) and the density (k) is expressed in a triangular form, and the equations for a single highway link are approximated by a set of equations where the current conditions (state of the system) are updated at every time interval (Daganzo, 1995; Lee et al., 2006):

$$q = \min{vk, q_{max}, w(k_j - k)} \text{ for } 0 \le k \le k_j$$

$$\tag{1}$$

where v is the free-flow speed, qmax is the maximum flow (or capacity), w is the backward wave or wave speed, and kj is the jam density.

The length of the homogeneous cells, i , is equal to the free-flow speed multiplied by a one-time interval. The primary parameters of

the CTM are defined by the left limb, right limb, and apex of the triangular FD, as shown in Fig. 3 (a). The left limb decides on the sending flow from an upstream cell to a downstream cell, and the right limb decides how much flow it can receive depending on its current capacity. Mathematically, the state of the system at instant t is then given by the number of vehicles contained in each cell, ni (t). For each cell, Ni(t) represents the maximum number of vehicles that can be present in cell i at time t , and Qi(t) defines the maximum number of vehicles that can flow into cell i when the time advances from t to t +1 . If the cells are numbered consecutively starting at the upstream end of the road, from i = 1 to I, the recursive relationship of the CTM can be expressed as follows:

$$n_i(t+1)(t+1) = n_i(t) + yi(t) - y{i+1}(t)$$ (2)

where yi (t) is the inflow to cell i in the time interval (t, t + 1), given by the following:

$$yi(t) = \min{n{i-1}(t)Q_i}, d[N_i(t) - n_i(t)]}$$ (3)

where d = w/v.

The CTM simulation runs two steps for each time interval: i) Determination of the flow on each link with the equivalent of Eq. (3); and ii) Updating of the cell occupancies by transferring the flows of step (i) from the beginning cell to the end cell of each link.

The FD was generated by the following steps: i) Extraction of the flow and density data for each sound detector on the freeway; ii) Consideration of the maximum value of the flow as the capacity, qmax ; iii) Determination of the free-flow speed v from the density–flow pair, following Dervisoglu et al. (2009) and Kurzhanskiy and Varaiya (2010); iv) Calculation of the critical density as kc = qmax v; and v) Application of a constrained least-squares method to determine the backward wave speed, w.

The CTM model was validated by calculating the mean percentage error (MPE) defined as follows:

$$MPE = \frac{1}{N} \sum{1}^{N} \left| \frac{Value{Original} - Value{CTM}}{Value{original}} \right|$$ (4)

4.3. Dynamic Bayesian network (DBN)-based real-time crash prediction model (RTCPM)

The Bayesian network (BN) is a probabilistic graphical modeling method where a system is represented by a graph and a joint probability function on the sequence of T hidden-state variables X = { *X0, …, XT*− 1 } and the sequence of T observable variables Y = { *Y*0 , …, YT− 1 }, where T is the time boundary for the given event under investigation. The DBN can be expressed as follows:

$$Pr(X, Y) = \prod_{i=1}^{T-1} Pr(xi | x{t-1}) \prod_{i=1}^{T-1} Pr(y_i | x_i) . Pr(x_0).$$ (5)

The complete specifications of a DBN must define three sets of parameters: the state transition probability distribution functions (PDFs) Pr(XT | XT− 1) that specify the time dependencies between the states; the observation PDFs Pr(YT | XT) that specify the dependencies of the observation nodes regarding other nodes at time t, and the initial state distribution Pr(*X*0) that determines the initial probability distribution at the beginning of the process.

In the present study, a DBN-based RTCPM was developed following the findings of Roy et al. (2018a) where 16 different models were developed keeping the crash likelihood as the hidden-state variable and the combination of traffic flow variables (i.e., the flow differences between upstream and downstream, and the density and speed downstream of the crash location) and their descriptive statistics as the observed variables from the segments of Route 3. The model considers a time boundary of three minutes (each time slice is separated by one minute), represented by three states of a single BN, as shown in Fig. 4 (b). The best performing model among those 16 models has been used in this study as the chosen RTCPM, which includes the downstream occupancy and speed and the difference in flow per minute between the downstream and the upstream as the observed variables. The detector locations and corresponding observed variables are illustrated by Fig. 4 (a). The study employed the data on 30 cases of pre-crash traffic conditions and the corresponding 748 cases of a normal traffic state. Among these, the data on 20 crashes and the corresponding 496 cases of a normal traffic state were used for model training, and the remaining data were used for evaluating the model. Regarding the data extraction process, if a crash occurred on Wednesday, March 3 at 15:00, then the traffic flow data on March 3 at the timestamps of 14:59, 14:58, and 14:57, representing the data for 1, 2, and 3 min before the crash, respectively, were counted as pre-crash data, and the traffic flow data from all other Wednesdays at the same timestamps were labeled as normal traffic data.

The probability distribution of the system can be represented by the following:

$$\Pr(K{down}, V{down}, \Delta Q, crash) = \prod{i-2}^{T-2} \Pr(K{down,i-1}, V{down,i-1}, \Delta Q{i-1} | K{down,i-2}, V{down,i-2}, \Delta Q_{i-2})$$

$$\Pr(K{down}, V{down}, \Delta Q, crash) = \prod{i-2}^{T-2} \Pr(K{down,i-1}, V{down,i-1}, \Delta Q{i-1} | K{down,i-2}, V{down,i-2}, \Delta Q_{i-2})$$ (6)

distribution compacted with the notion of conditional independence. Here, each variable is represented by a node, and the influence of one variable on the others is demonstrated with acyclic directed edges (which may or may not represent causality). The model can be used to both understand the dynamics within the system and predict the state of variables in the presence of evidence on any one or more variables. Dynamic Bayesian networks (DBNs) are special cases of singly connected BNs specifically arranged for time series modeling. All nodes, edges, and probabilities represent the static state of a system for which the interpretations are identical to those of a BN. The state of a DBNbased system at time t depends only on its immediate past, i.e., its state at the time (t − 1), and thus complies with the definition of a firstorder Markov property. A DBN consists of a probability distribution

The RTCPM performance was investigated based on the overall accuracy and prediction accuracy of the crash likelihood using Eqs. (7) and (8), where TP = crashes were predicted as crashes, FP = normal cases were predicted as crashes, FN = crashes were predicted as normal cases, and TN = normal cases were predicted as normal cases.

Overall accuracy (%) = $$\left(\frac{TP + TN}{TP + FN + FP + TN}\right)$$ (7)

Probability of crash likelihood(%) = $$\left(\frac{TP}{TP + FN}\right)$$ (8)

Fig. 4. DBN-based RTCPM.

4.4. Variable speed limit (VSL) implementation into cell transmission model (CTM)

In this study, the RTCPM takes the input from the CTM and evaluates the safety hazard. If the calculated risk is found to be higher than the predefined threshold, the RL-based intervention model applies the necessary VSL strategy. Next, the change in traffic flow conditions caused by the VSL must be simulated by the CTM, and the safety of the new traffic condition is further assessed through the RTCPM. Therefore, it is imperative to implement the VSL into the proposed CTM. Generally, the FD for a link or segment remains the same throughout the simulation. Here, the VSL control optimization and policymaking are conducted by a RL-based method, thus eliminating the need for the modified CTM model to solve the optimality problem for crash risk reduction. Therefore, the model must be programmed to ensure that the FD does not remain fixed throughout the simulation. This study followed the methodology proposed by Li et al. (2016), in which the FD does not remain fixed throughout the simulation (see Fig. 3 (b)). The primary parameters of a CTM are defined by the left limb, the right limb, and the apex of the triangle of the FD. The left limb decides the sending flow from an upstream cell to the downstream cell and the right limb decides how much flow it can receive depending on its current capacity. The left and right limbs, or the sending and receiving functions, are adjusted to accommodate the VSL values as follows:

$$si(t) = \min{V{SL}(t).ki(t), Q{VSL}}V_{SL}(t) \in [a_t]$$ (9)

$$r_i(t) = \min{wi \cdot \left(k{i,jam} - ki(t)\right), Q{VSL}}$$ (10)

where si(t) is the sending flow at time t, ri(t) is the receiving flow at time t, and at is the action set that will be decided by the reinforcement learning (RL) agent. For example, at = {±20, ±0} with the upper limit of VF means that the current speed could be reduced or increased by 20 km/h, or the free-flow speed for the segment will be chosen, depending on the optimization of the target-reducing crash risk at a target cell. VF is the free-flow speed, VSL(t) is the speed limit chosen at time t, ki(t) is the density, QVSL is the maximum flow under the current speed limit VSL(t) , wi is the backward wave speed, and ki,jam is the jam density.

The flow in a cell i can be defined by the following receiving function:

$$qi(t) = \min{s{i-1}(t), r_i(t)}.$$ (11)

The density evolution can be determined by the following:

$$k_i(t+1) = k_i(t) + \Delta T / Li(q{i-1}(t) - q_i(t))$$ (12)

Here, ΔT is the simulation time step, which is equal to the time in which a vehicle passes a cell at the free-flow speed. Li is the length of cell i . The speed within each cell can be determined according to the current density ki(t) and speed limit VSL :

$$v{i}(t) = \begin{cases} \min{V{SL}(t-1), V{F}} i f k{i}(t) \le k{VSL} \ \left(k{i,jam}(t) - k{i}(t) \cdot \frac{w{i}}{k{i}(t)}\right) i f k{i}(t) > k_{VSL} \end{cases}$$ (13)

where dVSL is the density associated with the flow QVSL (Fig. 3 (b)) under the speed limit, VSL .

To summarize, to build the modified CTM model, the basic parameters of an FD, which are free-flow speed, backward wave speed, jam density, and maximum flow, will be estimated from the traffic flow data from the loop detectors, and these parameters will remain fixed. Then, similar to the traditional CTM, the simulation time step will be decided to obtain the cell length according to the formula cell length = free-flow speed × time step. This will ensure that the vehicles do not cross multiple cells at a time step. So, after calculating the free-flow speed and deciding on a suitable cell length, the time step is calculated. Next, unlike the traditional FD, the modified FD (Fig. 3 (b)) whose left limb can be moved lower depending on the selected VSL value, will be constructed with a sending and receiving function, as shown in Eqs. (9) and (10). The sending function will choose either the maximum flow under the current VSL value, QVSL or the actual flow under the current VSL value (VSL(t).ki(t)), whichever is the minimum. At the same time, the receiving function will follow Eq. (10) and choose the minimum of the maximum flow under the current VSL value, QVSL and the backward wave speed calculated from the jam density and the current density ωi. ki,jam − ki(t) . According to the classic CTM theory, the sending flow of the upstream cell (1)i − is the receiving flow of the downstream cell i . Hence, the flow entering cell i will follow the minimum sending flow of the upstream cell (1)i − and the minimum receiving flow of cell i , according to Eq. (11). As for the speed evolution in a cell, it will depend on whether the current density falls to the left or right side of the density under the current VSL value. If it falls on the left side, the speed will be dominated by either the VF or VSL value; on the other hand, if it falls on the right side, the speed will be decided by the jam density and the backward wave speed.

The algorithm for the necessary calculations is as follows. Initialize CTM

*Define CTM parameters using FD: qmax,VF, kj,w, kc ,*ΔT, Li Generate traffic flow parameters in cells using Eqs. (6) and (7) At time = t

If crash risk > threshold, activate VSL

(continued on next column)

(continued )

[crash risk by RTCPM] 
       [VSL by RL] 
     Calculate cell flow, qi(t) :
     si− 1(t)
            { VSL.ki(t)if(VSL.ki(t) ) < QVSL
                     QVSL, otherwise 
     ri(t)
         { wi.
                ki,jam − ki(t)
                               if(VSL.ki(t)) < QVSL
                        QVSL, otherwise 
     Get flow qi(t) = min{si− 1(t), ri(t) }
     Update density, 
     ki(t +1) = ki(t) + ΔT/Li(qi− 1(t) − qi(t))
Repeat for t = t + 1.

This study tested the concept of the VSL-based integrated real-time proactive road safety management system using RL. Shinjuku Route 4 of the Tokyo Metropolitan Expressway was selected for the test. The VSL was implemented into the CTM for a long freeway segment (between detectors 48 and 39) on Shinjuku Route 4.

4.5. Reinforcement learning (RL) with deep Q-network (DQN)

Reinforcement learning has its roots in behavioral psychology. An agent is trained through rewards for correct behavior and punishment for incorrect behavior. In the context of deep RL, a network is shown input data and is given a positive or negative response based on whether it produces the correct output from that input, i.e., RL has sparse and time-delayed labels. Over several iterations, the agent receives sufficient information about its environment and learns to choose actions that will maximize the reward. In the present study, RL is used to learn behavioral strategies for VSL-based real-time intervention. Therefore, the RL agent adopts a policy to reduce the crash risk in real time by activating different speed limits. In that process, the agent observes the state st at time t from the traffic data (flow, speed, occupancy). The state st is sent as an input to the RTCPM. If the yielded crash risk exceeds the preset threshold value, then the RL algorithm is triggered to choose an action at , which is a set of speed limits. The VSL-integrated CTM then runs the simulation and the state st transitions to the next state, st+1 . The agent receives a reward or penalty depending on the quality of the decision to choose action at to reduce the crash risk. The RL algorithm creates a balance between exploitation and exploration. Exploration helps the agent find actions that might not bring a high reward immediately but could bring a high accumulated reward by taking the consequential actions. On the other hand, exploitation causes the agent to choose the action that brings a high reward immediately, without considering the future. This study uses the ε-greedy method to create a balance between exploitation and exploration. The objective maximizes the cumulative rewards by reducing the crash risk.

As established by Mnih et al. (2013) and Mnih et al. (2015), a deep Qnetwork (DQN) takes advantage of deep learning in learning policies. It is a combination of RL and a multilayered deep convolutional neural network (CNN) (Krizhevsky et al., 2012). A DQN consists of an agent dealing with the state–action pairs in an environment while receiving rewards to maximize the cumulative future rewards. It uses deep neural networks to approximate the Q-values:

$$Q^{*}(s,a) = \max{\pi} \mathbb{E} \left[ r{t} + \gamma r{t+1} + \gamma^{2} r{t+2} \dots + \gamma^{n} r{t+n} | s{t} = s, a_{t} = a, \pi \right]$$ (14)

where Q*(s, a) is the maximum sum of the rewards rt discounted by γ at each time step t , (s) is the state, (a) is the action, and π = p(a|s) denotes the behavioral policy.

Learning with a DQN is carried out in the following steps. The agent:

  • Collects experiences under the current policy and stores them in a buffer called experience replay;
  • Selects experiences randomly from experience replay;
  • Updates the Q-network using the sample experiences;
  • Repeats the above steps.

Hence, the two techniques that enable a DQN to overcome unstable learning are experience replay and the target network. By using experience replay, it removes the correlations in the state sequences and smooths the data distribution. Since it periodically updates the Q-values towards the target values, the DQN reduces the correlations with the target.

The value function of a DQN can be expressed as Q(s, a, θi) , in which *θi*represents the weight given to the neural network (NN) at iteration i . During random sampling (experience replay), an agent's experiences et (st , at , rt , st+1) are stored at each time step t in a set of data, Dt { *e*1 ,…, et }. For updating, the target is to minimize the mean squared error between the target Q-value (according to the Bellman equation) and the current Q-value:

$$\mathit{L}{\textit{I}}(\theta{\textit{i}}) = \mathbb{E}{(s{t}, a{t}, r{t}, s{t+1})\widetilde{\textit{U}}}\left[\left(R{t+1} + \gamma \underset{a^{'}}{\textit{max}} Q\big(s{t+1}, a{t+1}, \theta{i}^{-}\big) - Q(s{t}, a{t}, \theta{\textit{i}})\right)^{2}\right] \tag{15}$$

where *θi*are the parameters of the DQN at iteration i and θ− *i*are the network parameters, used to compute the target at iteration i. The target network parameters θ− *i*are only periodically updated with the DQN parameters (θi) and are held fixed between individual updates.

As mentioned earlier, the goal is to minimize the error, which means that the current policy's outputs will become more similar to the target Q-values. Therefore, a gradient step on the loss function is performed according to the following:

crashes occurred following the methodologies of Roy et al. (2018a,b). To generate traffic flow data for the crash prediction analysis, while generating the DBN model, CTMs were reproduced separately for each road segment corresponding to a crash for each day; thus, the FDs for separate detectors were different based on their location and the day of data collection. In detail, after the CTM was run, macroscopic traffic flow data were available for each cell. Next, the pre-crash and normal traffic data were collected for three minutes from the nearest upstream and downstream cells of the crash location. The downstream speed, density, and difference in upstream and downstream flow were chosen as the information variables for the DBN-based model as they were found to be the most influential variables for the RTCPM construction. The model was eventually constructed with traffic data on 71 crash conditions and 1190 normal conditions. The model was validated using data on 30 crash conditions and 542 normal conditions. We used all types of real crash data, most of which were rear-end crashes with a few side-swipe crashes. The DBN was constructed for three time steps, each with one minute of data (Fig. 4 (b)).

It should be noted here that in the case of the CTM, the traffic condition in a cell is assumed to be steady. Hence, it might seem that the model can only operate at steady-state traffic conditions. However, the CTM does not assume a steady traffic condition among the cells, i.e., the intercell traffic can still be unsteady. In this study, the RTCPMs were constructed with traffic flow parameters such as downstream flow, speed, and difference in upstream and downstream density. Hence, the difference in the traffic parameters between adjacent cells was used as

$$\nabla{\theta{i}}L{i}(\theta{i}) = \mathbb{E}{(s{i}, a{i}, r{i}, s{i+1})\widetilde{U}^{(D)}} \left[ \left( R{t+1} + \gamma \underset{i}{max} Q(s{t+1}, a{t+1}) - Q(s{t}, a{t}, \theta{i}) \right) \nabla{\theta{i}} Q(s{t}, a{t}, \theta{i}) \right]$$

$$(16)$$

With such properties as experience replay and the target network, the DQN creates more stability between the input and output for training the network (Mnih et al., 2013).

5. Experimental setup

The present study employed data from two different routes of the expressway for different activities undertaken in this research. Details are presented in the following subsections.

5.1. Cell transmission model (CTM)

The CTM was developed using data from the basic freeway segments of Route 4. FDs of the traffic were generated using flow–density graphs. Four basic freeway segments (detector 62 to 54, 48 to 39, 35 to 33, 27 to 26) were considered, and their FDs were generated separately for each segment by using the flow and density data of the detectors available within those segments. For example, the longest freeway segment was found to be between detectors 48 and 39 (2.52 km). Hence, this segment was used as the study segment for the VSL application. The FD for this segment was generated using the flow and density data of all the detectors within this segment.

5.2. Dynamic Bayesian network (DBN) for real-time crash prediction model (RTCPM)

RTCPMs require both pre-crash and normal condition traffic data. The one-minute pre-crash data were collected three minutes before the occurrence of the crash, and the corresponding normal (or no-crash) data were collected for the same time frame, but on the days when no the information variables of the RTCPMs, which can capture hazardous traffic conditions between adjacent cells depending on the threshold given. Consequently, a hazardous traffic situation can be captured from the intercell traffic difference in a CTM.

5.3. Variable speed limit (VSL) implementation into cell transmission model (CTM)

While implementing VSL control, the CTM left limb (free-flow side) can be changed depending on the crash risk threshold. The VSL-based CTM was generated following previous studies (Li et al., 2016; Li et al., 2017). If the reinforcement learning (RL) agent detects a high crash risk, it would take action in the form of VSL control, which would change the left limb of the FD from the free-flow speed to the reduced VSL value. The selected study segment was between detectors 48 and 39, and a target location (detectors 42 to 39) was selected for reducing the crash risk using VSL control. Additionally, the different values for the flow of the section QVSL , corresponding to the applied variable speed limit VVSL , were calculated using the FD for the corresponding section. In this study, we assumed the drivers' compliance with the VSL as 100% for simplicity.

5.4. Reinforcement learning (RL) with deep Q-network (DQN)

Three major parameters must be defined before applying the DQN: states, actions, and rewards. Details are provided below.

State, st : The states of the DQN are the traffic parameters collected from the simulation from segments of Route 4 consisting of cells 1 to 17 (detectors 48 to 39) (Fig. 5 (b)) during 13:00 to 18:00 (deducting the warm-up period). The target location consists of cells 12 to 17 (detectors 42 to 39). In this study, we assume that the proposed VSL on the upstream segments affects the downstream segments where the crash risk

Fig. 5. Fundamental diagram (FD) and CTM simulated detector arrangement alongside original detector layout.

is expected to be reduced by the VSL.

Rewards, rt : The target location for calculating the crash risk consists of cells 12 to 17 (detectors 42 to 39). The crash risk start-up threshold for receiving rewards is set to 10. This 10% threshold value suggests that the traffic state in consideration is found to have a 10% probability of eventually leading to a crash. The rationale for selecting 10 as the threshold value is explained in Section 6.2. If the crash risk at the target location is lower than 10, a reward of rt = + 1 is provided, otherwise − 1 is given.

Action, at : In the case of DQN, the action set is used when VSL control is activated at the current state, st , and a possible crash risk is observed (i.e., a state leading to a crash risk ≥ 10); the set of actions to be taken is at+1 = { ± 20, ±0} with the upper limit of Vfree− flow . For example, if the speed observed at the current state, st , is 70 km/h, then the choice of actions would be at+1 = {90, 70, 50, Vfree− flow.} km/h and the agent will receive a positive or negative response (rt = + 1, − 1) based on how well the agent performed by taking that particular action at time t, which then changed the current state, st , to st+1 . The simulation is run from 13:00 to 18:00. The duration of each simulation time step is 1 min (total simulation time = 5 h × 60 min × 36 iterations = 10,800 min). To evaluate the policy established by the agent, the Q-

value is calculated for each action taken. For the target section consisting of cells 12 to 17, the input, hidden, and output neurons were (12, 36, 36, 4).

The learning rate (α) is a hyperparameter that controls how much to change the model in response to the estimated error each time the model weights are updated. This means how much it will remember the old learning and to what extent it will replace it with the new experience. The learning rate was initially set to about 0.0001. The discount factor represents the importance given to the predicted possible future rewards compared to the current reward. The value of the discount factor (γ) was kept at around 0.9. The final parameter (ε) explains the exploration and exploitation ratios of the agent during training. The value is set to 1 so that the model can explore more and gradually decrease it with every iteration.

6. Analysis and results

6.1. Cell transmission model (CTM)

The values of the FD parameters are set as follows, as illustrated by Fig. 5 (a): capacity (qmax) 3780 vehicles/h; free-flow speed (VF) 135 km/

Table 1 Overall accuracy and prediction accuracy of the crash likelihood of RTCPM.

Thresholds 5 10 15 25
Overall accuracy 49 60 64 77
Crash likelihood 50 50 40 30

h; critical density (kc) 28 vehicles/km; jam density (kj) 120 vehicles/km; and backward wave (w) 41 km/h. The cell length was kept at 150 m, which is the average interdetector spacing for this study route, to generate uniformly distributed detectors.

As shown in Fig. 5 (b), the cells generated through the CTM are validated by comparing the traffic data from the nearest original detector, for example, comparing the simulated data generated for cells 1 and 2 using CTM with the real data from detector 48. The data for the segment between detectors 48 and 39 were validated through a comparison with the traffic data between 12:00 and 18:00 on April 8, 2014. The mean percentage error (MPE) was calculated to validate the data using Eq. (4), where N = total number of data for six hours = the number of detectors × time (min) × variables = 18 × 360 × 3 = 19440. After extracting the real detector data, 18,348 data points were found to have no errors. The MPE was calculated to be 0.0825, 0.1667, and 0.0449 for the flow, speed, and occupancy, respectively, and an average MPE was calculated to be 10%, suggesting high accuracy achieved in the newly developed CTM.

6.2. Dynamic Bayesian network (DBN) for real-time crash prediction model (RTCPM)

An investigation of the RTCPM performance found that at threshold values of 5 and 10, i.e., at 5% and 10% probabilities of eventually leading to a crash, the model detected 50% of the crashes (Table 1). The overall accuracy includes the correct prediction of crashes and normal cases. The overall accuracy was 60% at a threshold of 10. Hence, for this study, a threshold of 10 was chosen for predicting the crash risk.

6.3. Variable speed limit (VSL) implementation into cell transmission model (CTM)

For the VSL application, the traffic data for the inbound direction on April 8, 2014 was chosen for data collection and simulation as no crashes occurred and the traffic was steady on that day. The FD was generated using traffic data collected during the time from 12:00–12:01 to 17:59–18:00 from the segment between detectors 48 and 39, because the traffic was steady with the presence of a backward wave during that time. Both the free-flow speed and backward wave are necessary to construct the FD to generate the left and right sides of the triangle. In Fig. 6, which is the space–time diagram of the speed data (in km/h) where the x-axis represents the time and the y-axis represents the detector location, the speed is color-coded from green (higher speed) to red (lower speed). The figure shows that the traffic was more or less steady in the study segment and the speed never dropped below 30 km/h. The QVSL values calculated for different VSL values using the FD for the corresponding section are: VSL,QVSL = {(10,750), (20,1350), (30,1830), (40,2220), (50,2520), (60,2880), (70,3060), (80,3240), (90,3420), (100,3600), (110,3720)}.

6.4. Reinforcement learning (RL) with deep Q-network (DQN)

The outcomes of the DQN-based VSL control strategy are presented in Table 2, which indicates the change in the crash risk before and after the application of VSL control using the DQN when the crash risk threshold was set to ≥ 10. The number of cases with decreasing and increasing crash risk was aggregated for each cell and hour during 13:00 to 18:00 along the selected study segments between cells 1 and 17 (detectors 48 to 39). The first hour, 12:00–13:00, is the warm-up time. The 'better' and 'worse' columns represent the decrease and increase in the number of cases due to the application of VSL control (with a risk ≥ 10), respectively. Hence, 'better' is the desired outcome of VSL control and vice-versa. Additionally, the 'VSL on/off' lines indicate whether or not the VSL was triggered, and also for how long. For example, from 13:00 to 14:00, the values 14 and 46 indicate that the VSL was 'off' for

Fig. 6. Space–time diagram of the speed on the selected study segments (detectors 48 to 39) during 12:00 to 18:00.

Table 2 Change in crash risk before and after the application of VSL control using DQN (cells 1 to 17 (detectors 48 to 39)).

Cells Detectors 12:00–13:00 13:00–14:00 14:00–15:00
No Change
(Zero)		 Better
(Decrease)		 Worse
(Increase)		 No Change
(Zero)		 Better
(Decrease)		 Worse
(Increase)		 No Change
(Zero)		 Better
(Decrease)		 Worse
(Increase)
1 48–47 60 0 0 60 0 0 60 0 0
2 48–47 60 0 0 60 0 0 60 0 0
3 47–46 60 0 0 60 0 0 60 0 0
4 47–46 60 0 0 60 0 0 60 0 0
5 46–45 60 0 0 60 0 0 60 0 0
6 46–45 60 0 0 60 0 0 60 0 0
7 45–44 60 0 0 60 0 0 60 0 0
8 45–44 60 0 0 60 0 0 60 0 0
9 44–43 60 0 0 60 0 0 60 0 0
10 43–42 60 0 0 59 1 0 60 0 0
11 43–42 60 0 0 59 1 0 60 0 0
12 42–41 60 0 0 60 0 0 60 0 0
13 42–41 60 0 0 60 0 0 60 0 0
14 41–40 60 0 0 56 4 0 54 4 2
15 41–40 60 0 0 56 4 0 54 4 2
16 40–39 60 0 0 50 6 4 55 3 2
17 40–39 60 0 0 50 6 4 55 3 2
VSL off (warm-up time) 14 13
VSL on (warm-up time) 46 47
Cells Detectors 15:00–16:00 16:00–17:00 17:00–18:00
No Change
(Zero)		 Better
(Decrease)		 Worse
(Increase)		 No Change
(Zero)		 Better
(Decrease)		 Worse
(Increase)		 No Change
(Zero)		 Better
(Decrease)		 Worse
(Increase)
1 48–47 60 0 0 60 0 0 59 1 0
2 48–47 60 0 0 60 0 0 59 1 0
3 47–46 60 0 0 60 0 0 59 1 0
4 47–46 60 0 0 60 0 0 59 1 0
5 46–45 60 0 0 60 0 0 59 0 1
6 46–45 60 0 0 60 0 0 59 0 1
7 45–44 60 0 0 60 0 0 59 1 0
8 45–44 60 0 0 60 0 0 59 1 0
9 44–43 60 0 0 60 0 0 57 2 1
10 43–42 60 0 0 58 1 1 54 5 1
11 43–42 60 0 0 58 1 1 54 5 1
12 42–41 59 1 0 56 3 1 54 5 1
13 42–41 59 1 0 56 3 1 54 5 1
14 41–40 53 2 5 49 7 4 49 7 4
15 41–40 53 2 5 49 7 4 49 7 4
16 40–39 51 4 5 45 9 6 52 5 3
17
VSL off		 40–39 51
13		 4 5 45
15		 9 6 52
17		 5 3

Table 3 Comparison of the number of events with a risk (NER) ≥ 10.

Cells Detectors No control Control Difference
Risk ≥ 10
1 48–47 83 83 0
2 83 83 0
3 47–46 77 77 0
4 77 77 0
5 46–45 58 58 0
6 58 58 0
7 45–44 50 50 0
8 50 50 0
9 44–43 53 53 0
10 43–42 45 43 2
11 45 43 2
12 42–41 58 50 8
13 59 50 9
14 41–40 69 54 15
15 69 55 14
16 40–39 64 50 14
17 64 50 14

14 min and 'on' for 46 min.

From Table 2, it can be seen that the crash risk improved at the target location (cells 12 to 17) in several hours and deteriorated in others. The risk mainly improved after the speed limit was lowered. In the case of DQN-based VSL control, improvement in the upstream cells was observed when the crash risk threshold was set to ≥ 10. There is a possibility that when the speed limit of the target location is set very low relative to the posted speed limit to reduce the risk, other high-risk segments may emerge upstream unless the change in the speed limit starts from the very upstream of the target location, gradually changing the desired speed limit. According to Table 2, the high-risk upstream segments (cells 1 to 11), where the risk became worse in the case of DQN-based VSL control, were not mainly observed in the present study. However, this issue requires further study to minimize the evolution of other high-risk segments upstream. It is also notable that the action strategy was implemented in such a way (±20*,*±0) with the upper limit of Vfree− flow that the consecutive VSL actions did not differ greatly and thus did not cause a sudden disruption in the traffic flow. Table 3 shows a comparison between no speed control and VSL control in terms of reducing the number of events with a risk (NER) ≥ 10 at the selected study segments (detectors 48 to 39 (cells 1 to 17)) and the target location (detectors 42 to 39 (cells 12 to 17)). It can be seen that the number of crash-prone events was reduced at the target location by 14%, 15%,

Fig. 7. Space–time diagram of the speed on the selected study segments (detectors 48 to 39 (cells 1 to 17)) during 12:00 to 18:00 after employing DQN-based VSL control (b) and without VSL control (a).

22%, 20%, 22%, and 22% (average 19%), and improvements were also observed upstream (cells 10 and 11).

Fig. 7 shows a space–time diagram of the speed on the selected study segments (detectors 48 to 39 (cells 1 to 17)) during 12:00 to 18:00 when VSL control was applied (Fig. 7 (b)) and there was no speed control (Fig. 7 (a)). While the goal is to improve crash safety, it is also important to consider the functionality of the route. The appearance of a capacity drop due to VSL control might cause congestion and increase the possibility of rear-end crashes. From Fig. 7, few changes are seen in the speed profile with the application of VSL control, particularly during 13:00 to 14:00 and 14:00 to 15:00. The DQN model performed well not only in reducing NER ≥ 10, but also in not generating any major speed fluctuations in the target location except for the slightly low speed that occurred during 16:00 to 17:00. The speed remained well distributed overall even after employing the VSL control.

7. Conclusion

This study proposed a DBN-based RTCPM that can blend prior information with new information to make predictions in real time, update model parameters across time and space to improve the spatial and temporal transferability, and handle the imbalance between crash and non-crash data during model construction. In addition, the concept of RT-RL-based VSL was introduced as an intervention for real-time crash risk reduction. The most significant contribution of this study is the proposed framework that seamlessly connects the macroscopic inputs and the RTCPMs with a VSL control powered by DQN-based RL intervention, which holds promise as a practical and proactive road safety management technology. The reason for choosing RL-based VSL control was to utilize the model-free properties of the RL algorithm. Previous studies have employed different VSL strategies to improve crash risks, by implementing policies such as applying a VSL control value and incorporating functions for gradually increasing or decreasing the VSL control speed, etc. There is no concrete solution for deciding upon a strategy for VSL control. With RL, it is not necessary to create a structured model to teach the agent with labeled input data; the agent learns without a model by playing a game and learning from experience to take the best possible actions to win rewards. In this study, the DBN-based RTCPM correctly identified 50% of the crash events, maintaining an overall accuracy rate of 60%. With a risk threshold of ≥ 10, the DQN-based RL intervention model reduced the number of crash-prone events for a target section consisting of five cells, each spanning over 150 m, by 14%, 15%, 22%, 20%, 22%, and 22% (average 19%) when applied for six continuous hours (12:00–18:00). Interestingly, crash risk improvements were also observed upstream (cells 10 and 11). However, this caused some disruption in the speed of the vehicles, leading to congestion during 15:30 to 17:00 in the target area.

Apart from these contributions, the study has some limitations. In particular, safety and operational efficiency are two functions that are not necessarily compatible and often compete with each other. It is imperative to investigate how the proposed VSL strategy could impact the operational efficiency of the expressway. In addition, although the proposed CTM-based method considered only 150 m as the cell length, the approach does not consider the oscillation of traffic flow variables within the cell. A future study should perform a significance test comparing various cell sizes to find an optimum size for the implementation of RT-PRSMS. Being DBN-based, the RTCPM introduced here allows for accommodating new variables, such as weather, lighting conditions, etc. in the future. However, such variables were not included here. Moreover, the transferability of the proposed RT-PRSMS needs to be evaluated with data from other parts of the world. Finally, one major limitation of the study is the assumption that 100% of the drivers will comply with the posted VSL without delay. It will be interesting to investigate how drivers react to the proposed VMS strategies by testing the framework in a driving simulator.

Real-time intervention can systematically reduce the probability of

road traffic crashes by altering the traffic states via the deployment of various traffic-influencing measures, which require institutional management. In the case of an RT-PRSMS, the institutional management functions can be handled either by the road authorities or the vehicle manufacturers. In the former case, the data sources can be various types of traffic sensors or probe vehicles that continuously extract real-time data on the traffic states. In the latter case, when the institutional management function is controlled by the vehicle manufacturer, information regarding the traffic stream can be obtained through intervehicular communication and then fed into in-vehicle RTCPMs. In the future, from the perspective of the road authorities, the presented framework can be expanded to the route/network level, thereby ensuring that safety improvement in one part of the network does not offset the risk to other parts, and that the efficacy of other proactive safety management instruments, such as variable message signs (VMS), mainline metering, and ramp metering, etc., independently as well as in combination, can be explored with the proposed framework. At the same time, with the advancement in autonomous vehicle technology and deeper instrumentation of highways, vehicles and infrastructures can share active safety features to further enhance traffic safety without compromising operational efficiency, leading to direct collaboration between road authorities and vehicle manufacturers.

CRediT authorship contribution statement

Ananya Roy: Conceptualization, Methodology, Data curation, Writing – original draft, Software, Visualization, Investigation. Moinul Hossain: Conceptualization, Data curation, Writing – review & editing. Yasunori Muromachi: Conceptualization, Methodology, Supervision, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

We thank Tokyo Metropolitan Expressway Company Limited for providing necessary data to conduct this study. This study was also supported by JSPS KAKENHI Grant Number 25420535.

References

Abdel-Aty, M., Dilmore, J., Hsia, L., 2006a. Applying Variable Speed Limits and the Potential for Crash Mitigation. Transp. Res. Rec. 1953, 21–30.

Abdel-Aty, M., Dilmore, J., Dhindsa, A., 2006b. Evaluation of Variable Speed Limits for Real-time Freeway Safety Improvement. Accid. Anal. Prev. 38 (2), 335–345.

Abdel-Aty, M., Dhindsa, A., Gayah, V., 2007a. Considering various ALINEA ramp metering strategies for crash risk mitigation on freeways under congested regime.

Transp. Res. Part C: Emerg. Technol. 15 (2), 113–134. Abdel-Aty, M., Pande, A., Lee, C., Gayah, V., Santos, C.D., 2007b. Crash risk assessment using intelligent transportation systems data and real-time intervention strategies to improve safety on freeways. J. Intell. Transp. Syst. 11 (3), 107–120.

Abdel-Aty, M., Gayah, V., 2010. Real-time crash risk reduction on freeways using coordinated and uncoordinated ramp metering approaches. J. Transp. Eng. 136 (5), 410–423.

Abdel-Aty, M., Wang, L., 2017. Implementation of variable speed limits to improve safety of congested expressway weaving segments in microsimulation. Transp. Res. Proc. 27, 577–584.

Abdel-Aty, M., Cunningham, R.J., Gayah, V.V., Hsia, L., 2017. Dynamic variable speed limit strategies for real-time crash risk reduction on freeways. Transp. Res. Rec. 2078 (1), 108–116.

Abdulhai, B., Kattan, L., 2003. Reinforcement Learning: Introduction to Theory and Potential for Transport Applications. Can. J. Civ. Eng. 30 (6), 981–991.

Basso, F., Basso, L.J., Bravo, F., Pezoa, R., 2018. Real-time crash prediction in an urban expressway using disaggregated data. Transp. Res. Part C: Emerg. Technol. 86, 202–219.

Basso, F., Basso, L.J., Pezoa, R., 2020. The importance of flow composition in real-time crash prediction. Accid. Anal. Prev. 137, 105436. https://doi.org/10.1016/j. aap.2020.105436.