category: literaturenote citekey: xuevaluationpredictabilityrealtime2016 title: Evaluation of the predictability of real-time crash risk models authors: "Xu, Chengcheng; Liu, Pan; Wang, Wei" year: 2016 date: 2016-00-00 2016 doi: 10.1016/j.aap.2016.06.004 publication: "Accident Analysis & Prevention" url: "https://doi.org/10.1016/j.aap.2016.06.004" zotero_key: JCP5XF8T zotero_storage: S4IMW5PT collections: transpordi_planeerimine folder: 001_artiklid/Road Safety Indicators firstAuthor: "Xu, Chengcheng"

status: converted

Contents lists available at ScienceDirect

Accident Analysis and Prevention

j ournal homepage: www.elsevier.com/locate/aap

Evaluation of the predictability of real-time crash risk models

Chengcheng Xu^a,b, Pan Liu^a,b,∗, Wei Wang^a,^b

^a Jiangsu Key Laboratory of Urban ITS, Southeast University, Si Pai Lou #2, Nanjing, 210096, China
^b Jiangsu Province Collaborative Innovation Center of Modern Urban Traffic Technologies, Si Pai Lou #2, Nanjing, 210096, China

a r t i c l e i n f o

Article history: Received 24 September 2015 Received in revised form 7 May 2016 Accepted 7 June 2016 Available online 19 June 2016

Keywords: Crash risk Predictability Real-time crash risk modeling Predictive performance Freeways

a b s t r a c t

The primary objective of the present study was to investigate the predictability of crash risk models that were developed using high-resolution real-time traffic data. More specifically the present study sought answers to the following questions: (a) how to evaluate the predictability of a real-time crash risk model; and (b) how to improve the predictability of a real-time crash risk model. The predictability is defined as the crash probability given the crash precursor identified by the crash risk model. An equation was derived based on the Bayes' theorem for estimating approximately the predictability of crash risk models. The estimated predictability was then used to quantitatively evaluate the effects of the threshold of crash precursors, the matched and unmatched case-control design, and the control-to-case ratio on the predictability of crash risk models. It was found that: (a) the predictability of a crash risk model can be measured as the product of prior crash probability and the ratio between sensitivity and false alarm rate; (b) there is a trade-off between the predictability and sensitivity of a real-time crash risk model; (c) for a given level of sensitivity, the predictability of the crash risk model that is developed using the unmatched case-controlled sample is always better than that of the model developed using the matched case-controlled sample; and (d) when the control-to-case ratio is beyond 4:1, the increase in control-to-case ratio does not lead to clear improvements in predictability.

1. Introduction

Crashes are rare and random events whose occurrences are influenced by a set offactors that are partly deterministic and partly stochastic. The circumstances that lead to a crash in one event will not necessarily lead to a crash in a similar event. The randomness of crashes introduces natural fluctuation in crash counts over time; and the average crash frequency in the short term may be significantly differentfrom the expected crash frequency in the long term. The natural variability in crash counts makes it difficult to identify whether the changes in the observed crashes are due to the natural variability or to the changes in site conditions. Without being accounted for properly, the randomness of crashes will eventually introduce regression-to-the-mean bias, resulting in a biased estimate of safety.

To account for the natural variability in crash data, traditional crash modeling framework uses the Poisson family distributions, such as the Poisson distribution, Poisson-gamma distribution and Poisson-lognormal distribution for estimating the expected number of crashes in the long term. Crash frequency models are developed on the basis of the data that are aggregated over time and space. The process of aggregation may result in the loss of useful information with regard to the causal factors to crashes. In essence, the major task of traditional crash frequency models is not to predict crashes. Instead, they aim at estimating the longtermaverage crash frequency under a given set of geometric design, traffic control, and traffic conditions.

During the past two decades, with the widespread use of freeway traffic surveillance systems, increased attention has been given to identifying the traffic flow conditions prior to crash occurrences using high-resolution traffic data. Real-time crash risk models have been developed to link crash risks to dynamic traffic flow parameters (Oh et al., 2001; Lee et al., 2003; Abdel-Aty et al., 2004, 2005; Abdel-Aty and Pemmanaboina, 2005, 2006; Abdel-Aty and Pande, 2006; Xu et al., 2012). Unlike crash frequency models,the real-time crash risk models treat each individual crash as the unit of analysis. The central idea is to estimate the relative risks of crashes for a relatively small time interval (usually 5 min) on the basis of the hazardous traffic conditions that commonly occur before crashes (crash precursors) combined with other influence factors such as geometric design characteristics and weather information. With

^∗ Corresponding author at: Jiangsu Key Laboratory of Urban ITS, Southeast University, Si Pai Lou #2, Nanjing, 210096, China.

E-mail addresses: xuchengcheng@seu.edu.cn (C. Xu), pan liu@hotmail.com (P. Liu), wangwei@seu.edu.cn (W. Wang).

crashriskmodels, variousproactive safetymanagementtechniques can be applied to make interventions before the occurrences of crashes (Lee et al., 2006a, 2006b; Khoury and Hobeika, 2007; Allaby et al., 2007).

The case-control study design forms the foundation of crash risk modeling. In a case-controlled dataset, the traffic data prior to crashes are taken as cases while those under crash free conditions are taken as controls. The logistic regression technique can then be used to distinguish between crash precursors and normal traffic conditions, andto establisha relationshipbetweencrashprecursors and crash risks. In the case-control study design, a typical case-to-control ratio of 1:4 has been widely accepted. That is, for each crash case, researchers may select four non-crash cases. By doing so rare events can be investigated in a relatively quick and cheap manner by greatly reducing the total number of controls.

In the real world, however, the number of non-crash cases is actually enormous compared with that of the crash cases. For example, considering a freeway section with 70 loop detector stations, the total number of non-crash cases in three years is close to 70 × 365 × 3 × 24 × 60 = 1.1E + 08, if the data are aggregated based on one-minute time intervals. Thus, the case-controlled samples are biased towards over-representing the crash cases. Assuming that a typical case-to-control ratio of 1:4 is used, a crash is expected to occur in one of everyfive records. However,the actual probability of a crash in a 5-min time interval is extremely small considering the large number of non-crash cases on the entire population. As a result, existing crash risk models provide only a measure of the relative risks of crash occurrences given the composition of the samples.

The probabilistic nature of both crash frequency and crash risk models raises fundamental questions: what is the role of the randomness in crash occurrences, and to what extend are crashes predictable? The primary objective of the present study was to investigate the predictability of real-time crash risk models. More specifically the present study sought answers to the following questions:(a) how to evaluate the predictability of a real-time crash risk model; and (b) how to improve the predictability of a real-time crash risk model.

2. Literature review

In some early studies, only the data prior to crashes were considered when modeling crash risks (Golob and Recker, 2004; Golob et al., 2004a, 2004b; Lee et al., 2006a, 2006b). Golob et al. (2004a) classified freeway traffic flow before crash occurrences into different states using clustering analysis, and then conducted nonlinear canonical correlation analysis to relate the characteristics of crashes to different traffic states. A procedure was also developed to predict the type of crashes that were most likely to occur for the traffic states being monitored (Golob et al., 2004b). Lee et al.(2006a, 2006b) compared the traffic flow conditions prior to sideswipe and rear-end crashes.Alogistic regression model was developed to estimate the relative risks of sideswipe crashes compared to rear-end crashes given the fact that a crash has occurred. This type of models mainly focuses on the relative risk of a particularly type of crashes given the fact that a crash has occurred. Accordingly, they cannot be used to predict the probability of crash occurrences.

More recent studies took into consideration the traffic flow data both prior to crashes and in crash-free conditions when developing crash risk models. Most of these studies followed a case-control study design in which the traffic conditions that were associated with crash occurrences and normal traffic conditions were investigated to identify crash precursors and their effects on crash risks. The use of the case-controlled dataset allows for the measurement of the relationship between traffic conditions and crash risks. The matched and unmatched case-control designs are two dominant approaches for modeling crash risks. The major difference between these two approaches lies in the methods of selecting non-crash cases. In the matched case-control design, the non-crash cases are matched with crash cases according to some confounding factors such as the time and the locations of crashes, while in the unmatched case-control design the control samples are randomly selected.

Both matched and unmatched case-control designs control for the impacts of confounding variables (Bruce et al., 2008). The major difference is that the matched case-control design accounts for the impacts of confounding factors at the stage of selecting controls; while the unmatched case-control design takes into account the impacts of confounding factors at the stage of data analysis (Bruce et al., 2008). The case-control study design reduces the non-crash cases greatly and accounts for the selection bias. However, both matched and unmatched case-controlled samples are biased samples in which the crash cases are over-represented.

With the case-controlled dataset, the logistic regression models can be developed to link crash risks to crash precursors. The conditional logistic regression technique is usually used in the matched case-control design to account for the selection bias (Abdel-Aty et al., 2004, 2005; Abdel-Abdel-Aty and Pemmanaboina, 2006). In the conditional logistic regressionmodel,the conditional likelihood function is used to make comparisons within each matched pair. Accordingly, it can deal with the confounding factors that lead to selection biases. In the unmatched case-control design, the conventional logit model is usually used (Xu et al., 2013; Ahmed et al., 2011).

The sensitivity and specificity have been widely accepted as the performance measures of the predictability of crash risk models. The sensitivity measures the proportion of the crash cases that are correctly identified. The specificity measures the proportion of the non-crash cases that are correctly identified, and (1-specificity) is usually called the false alarm rate, which represents the proportion of the non-crash cases that are mistakenly identified as crash cases. The use of specificity and sensitivity did not truly reveal the predictability of a crash risk model, and the reasons are twofold. First, most studies evaluate the sensitivity and false alarm rate on the basis of a case-controlled dataset. They are not indicative of the real prediction accuracy on the entire population. Second, because the crash risk models generate only the likelihood of crashes, a threshold needs to be selected to help identify crash precursors. The selectedthresholdheavily affects the sensitivity andfalse alarm rate that are associated with crash risk models. For example, one can deliberately increase the sensitivity, or in other words the prediction accuracy of crash cases, of crash risk models by setting up a very low threshold. In this condition, the false alarm rate will also get increased. The current practice is to select a threshold such that a balance can be reached between sensitivity and specificity. However, this cutoff is arbitrary in nature.

3. Predictability of crash risk models

The predictability of a crash risk model measures to what extend crashes are predictable given crash precursors. If the focus is on the prediction accuracy on the entire population, the predictability of crash risk models can be measured as the conditional probability of a crash given crash precursors. Based on the Bayes' theorem, the conditional probability of a crash given a crash precursor can be calculated as:

$$P(A|B) = \frac{P(B|A)P(A)}{P(B)} = \frac{P(B|A)}{P(B)}P(A)$$ (1)

where A represents the occurrence of a crash; B represents the crash precursor; P(A|B) represents the risk of a crash when a crash precursor is observed; P(B|A) represents the probability of observing a crash precursor before the occurrence of a crash; P(A) represents the prior probability or marginal probability of a crash without taking into account any information about the crash precursor; and P(B) represents the marginal probability of a crash precursor. According to the Bayes' theorem, P(A|B) is also called the posterior crash probability as it is derived from or depends upon the crash precursor. The ratio P(B|A)/P(B) is sometimes called standardised likelihood. With the above terminology, the Eq. (1) can be paraphrased as:

Posterior crash probability = Standardised Likelihood

× Prior Crash Probability (2)

The posterior crash probability represents the probability of crash occurrences given the crash precursor identified by the crash risk model. The prior crash probability P(A) measures the probability of having a crash during a specified time period which can be estimated using traditional crash frequency models. P(A) can be directly estimated given reported crash data. When crash data are not available, the conventional crash frequency model can be used for estimating P(A). Note that the expected crash frequency during a short time interval (5–15 min) is extremely small. Accordingly, assuming that the frequency of crashes during a short time interval can be modeled using zero inflated Poisson regression, the probability of observing y crashes during a specific time period can be estimated as:

$$p(y|\lambda) = \begin{cases} z + (1-z)\exp(-\lambda), & y_i = 0\ \frac{(1-z)\exp(-\lambda)\lambda^y}{y!} & y_i = 1 \end{cases}$$ (3)

where z represents the probability ofthe zero state in a zero inflated Poisson model; and represents the expected crash frequency of the Poisson distribution in the non-zero state.

Based on Eqs.(1) and (2),the following conclusions can be made:

(1) the posterior crash probability measures the predictability of a crash risk model. If crashes are fully predictable given the crash precursor identified by crash risk models, P(A|B) is equal to 1. On the contrary, if the crash occurrence is completely independent of the crash precursor, P(A|B) should be equal to the prior crash probability. In this condition, the crash risk model is not useful at all and the crash precursor is not helpful to improve the prediction accuracy of crashes;
(2) the reciprocal of P(A|B) measures the number of predictions that need to be made before a crash can be successfully predicted. In practical engineering applications, it can be translated to the number of interventions that are needed to prevent the occurrence of a crash in proactive safety management systems;
(3) the standardised likelihood P(B|A)/P(B) is determined by the crash risk model. It measures to what extent the use of crash precursors and crash risk models can increase the predictability of crashes as compared with traditional crash frequency models. Therefore, only when P(B|A)/P(B) is greater than 1, the crash risk model can be considered useful; and
(4) in the crash risk model, the crash precursor is determined by thepre-determinedthreshold. The thresholdshouldbe selected carefully, because it determines the standardised likelihood P(B|A)/P(B), and accordingly, the predictability of a crash risk model.

As mentioned before, the standardised likelihood P(B|A)/P(B) determines the predictability of the crash risk models relative to traditional crash frequency models. It is a critical parameter that can be used for comparing the predictive performance of different types of crash risk models. The standardised likelihood can further be calculated as follows:

$$\frac{P(B|A)}{P(B)} = \frac{P(R > r|y = 1)}{P(R > r, y = 1 \text{ or } 0)} = \frac{P(R > r|y = 1)}{P(R > r|y = 1)P(y = 1) + P(R > r|y = 0)P(y = 0)}$$ (4)

where r represents the pre-specified threshold for crash precursor; y represents whether crash occurs (=1 if a crash occurred, and 0 otherwise); P(R > r|y = 1) denotes the proportion of the crashes that are correctly identified,i.e.,the sensitivity; andP(R > r|y = 0)denotes the proportion of the non-crash cases that are misclassified as crashes, i.e., the false alarm rate.

In Eq. (4), P(y = 1) and P(y = 0) represents the probability of crash or non-crash cases in a given time interval. In this study, the realtime crash risk models were developed using traffic aggregated on 5-min time interval. The time interval for P(y = 1) and P(y = 0) is 5 min. Because crashes are extremely rare events, the crash probability in a 5-min time interval is much smaller than the probability of not having any crashes. Thus, the value of P(R > r|y = 1) × P(y = 1) is negligibly small compared with P(R > r|y = 0) × P(y = 0). The product term P(R > r|y = 1) × P(y = 1) can be left out in Eq. (4). The Eq. (4) is then simplified as follows:

$$\frac{P(B|A)}{P(B)} \approx \frac{P(R > r|y = 1)}{P(R > r|y = 0)P(y = 0)}$$ (5)

As the probability of non-crash cases in a 5-min interval is very close to 1, the term P(y = 0) in the denominator in Eq. (5) can also be left out. The Eq. (5) can be further simplified as:

$$\frac{P(B|A)}{P(B)} \approx \frac{P(R > r|y = 1)}{P(R > r|y = 0)} = \frac{P\text{roportion of identified crash cases}}{1-\text{Proportion of identified noncrash cases}}$$

$$= \frac{\text{Sensitivity}}{1-\text{Specificity}}$$ (6)

Thus, the predictability of a crash risk model can be approximately calculated as follows:

$$P(A|B) = P(y = 1|R > r) = \frac{\text{Sensitivity}}{1 - \text{Specificity}} \times P(A)$$

$$= \frac{\text{Sensitivity}}{1 - \text{Specificity}} \times P(y = 1)$$ (7)

With Eq. (7) it is concluded that the standardised likelihood can be approximately estimated as the ratio between the sensitivity and the false alarm rate associated with a crash risk model. The parameter can be directly used for measuring and comparing the performance of different crash risk models.

4. Test for the validity of the findings

A case study was conducted here to: (a) test for the validity of Eq. (7); and (b) illustrate how the findings mentioned above affect crash risk modeling. The data used for the case study were obtained from a 17-mile section on the I-880N freeway in the state of California, United States. The study period covers the entire year of 2009. Crash data were obtained from the Statewide Integrated Traffic Records System (SWITRS) which was maintained by the California Department of Transportation (Caltrans). A total of 508 crashes were identified and used. The geometry and real-time traffic data were obtained from the Highway Performance Measurement System (PeMS). There are 35 loop detector stations along the selected freeway section (see Fig. 1). The spacing between loop detector stations varies from 0.2 to 1.3 miles with an average of 0.5 miles.

For each crash, the authors collected 5-min traffic data ending at five minutes prior to crashes. For example, if a crash occurred

Fig. 1. Location of loop detector stations along the selected I-880N freeway section.

Table 1 Candidate variables considered for model development in this study.

	Candidate variables considered for model development in this study.
Symbol	Variables
AvgCntu	Average 30-s vehicle count at the upstream station (veh/30 s)
AvgOccu	Average 30-s detector occupancy at the upstream station (%)
AvgSpdu	Average 30-s speed at the upstream station (mile/h)
DevCntu	Std. dev. of 30-s vehicle counts at the upstream station (veh/30 s)
DevOccu	Std. dev. of 30-s detector occupancies at the upstream station (%)
DevSpdu	Std. dev. of 30-s mean speeds at the upstream station (mile/h)
AvgCntd	Average 30-s vehicle counts at the downstream station (veh/30s)
AvgOccd	Average 30-s detector occupancy at the downstream station (%)
AvgSpdd	Average 30-s speed at the downstream station (mile/h)
DevCntd	Std. dev. of 30-s vehicle counts at the downstream station (veh/30s)
DevOccd	Std. dev. of 30-s detector occupancies at the downstream station (%)
DevSpdd	Std. dev. of 30-s mean speeds at the downstream station (mile/h)
DifCntu-d	Absolute difference in count between upstream and downstream stations (veh/30s)
DifOccu-d	Absolute difference in occupancy between upstream and downstream stations (%)
DifSpdu-d	Absolute difference in speed between upstream and downstream stations (mile/h)
Length	Distance between upstream and downstream stations (mile)
Lanes	Number of lanes
Barrier	1 = barrier; 0 = no barrier
Rampon	1 = if there is an on-ramp between upstream and downstream stations; 0 = otherwise
Rampoff	1 = if there is an off-ramp between upstream and downstream stations; 0 = otherwise
Widths	Road surface width (ft)
Widtho	Outer Shoulder width (ft)
Widthi	Inner Shoulder width (ft)
Widthl	Lane width (ft)
Widthm	Inner median width (ft)

at 10:25 a.m., the traffic data were collected from 10:15 a.m. to 10:20 a.m. The purpose was to compensate for the possible errors in the reported crash occurrence time, and to identify hazardous traffic conditions ahead of the crash occurrence time to make preemptive measures possible. The traffic data were collected from the nearest upstream and downstream stations to each crash location. The raw 30-s traffic data were aggregated into 5-min intervals and converted into the fifteen traffic flow variables presented in Table 1.

Both matched and unmatched case-controlled samples were created for developing crash risk models. With regard to the unmatched case-controlled samples,ten non-crash cases were randomly selected from the crash-free days for each crash case. More specifically, the time and the location for each non-crash case were randomly generated. The time was randomly selected from 105, 120 time intervals, which is the number of 5-min periods in 2009 (60 min/5min × 24h × 365 days in 2009). Two neighboring detec-

Table 2 Estimation results of the Bayesian logistic regression model.

Parameters	Mean	2.50%	Median	97.50%
Constant	−0.714	−1.342	−0.741	−0.003
Lengtha	–	–	–	–
Widths	−0.019	−0.031	−0.019	−0.008
Rampoff	0.219	0.026	0.218	0.410
Rampon	0.221	0.054	0.220	0.392
AvgOccu	0.069	0.054	0.068	0.084
AvgSpdd	−0.035	−0.042	−0.035	−0.029
DevSpdd	0.026	0.001	0.026	0.051
DifSpdu-d	0.018	0.008	0.018	0.028

^a The variable length was modeled as an exposure variable.

tor stations (i.e., the upstream and downstream stations) for each non-crash were randomly selected from the 35 detector stations. Thus, the sampling universe contains 3.574E + 06 (=105, 120 × 34) potential time–space points. The non-crash cases were randomly selected from the sampling universe.

With regard to the matched case-controlled sample, the noncrash data were selected on the basis of three matching factors, including the time, the location, and season. For example, a crash occurred at post-mile 26.84 at 15:25 p.m. on July 20, 2010. Traffic data taken at the nearest upstream and downstream stations from 15:15 p.m.to 15:20 p.m. on July 20, 2010 were included in the crash cases as an observation. Then the paired crash free traffic data taken at the same two loop detector stations during the same period on ten randomly selected crash-free days in the same season were included in the non-crash cases. Two crash risk models were fitted. One model was developed using matched case-controlled samples and the other using unmatched case-controlled samples. For both models a control-to-case raito of 10:1 was used.

The estimation results of the Bayesian logistic regression model based on the unmatched case-controlled sample are given in Table 2. Four traffic flow variables affect crash risks significantly. The coefficients of the traffic flow variables AvgOccu, DevSpdd, and DifSpdu-d are postive, indicating that crash risk increases with an increase in the occupancy at upstream stations, the speed variance at downstream station, and the speed difference between upstream and downstream stations. The findings are reasonable because large speed variance in high-density traffic flow indicates turbulenttraffic conditions thatlead to high crash potential, and the vehicles from upstream stations have to decelerate abruptly when approaching relatively lower speed traffic at the downstream stations. The parameter ofthe traffic flow variableAvgSpdd is negative, indicating that a crash is more likely to occur in the traffic conditions with low average speed. The finding is consistent with that from previous studies (Ahmed et al., 2011; Ahmed and Abdel-Aty, 2012). A possible explanation is that the decreasing speed represents queue formations, resulting in increased risks of rear-end crashes.

In addition to the four traffic flow variables, three geometric design variables (Widths, Rampoff and Rampon) were also found to be statistically significant. Both variables Rampoff and Rampon have positive parameters, indicating that the presence of on- and off-ramps between the detector stations increases crash risks. The finding is intuitive because the complex traffic merging and diverg-

Fig. 2. The empirical relationship between P(y = 1|R > r) and the ratio of sensitivity and false alarm rate.

ing behaviors in the influence area of on- and off-ramps lead to increased crash risks. The negative coefficient of the variable Widths indicates that the probability of crash decreases as the road surface width increases. Generally speaking, the wide road surface width gives drivers more time and space for making crash avoidance maneuvers, which consequently reduces the probability of crashes.

For comparison, a Bayesian conditional logistic regression model was also developed on the basis of the matched casecontrolled sample (see Table 3). As expected, the traffic flow variables in the Bayesian conditional logistic regression model are the same as those in the Bayesian logistic regression, which was developed using the unmatched case-controlled sample. The parameters of these four traffic flow variables in two models are also quite similar. Note that the geometric design variables were

Fig. 3. The distributions of the non-crash cases over time and space.

Table 3 Estimation results of the Bayesian conditional logistic regression model.

Parameters	Mean	S.D.	MC Error	2.50%	Median	97.50%
AvgOccu	0.077	0.012	0.001	0.053	0.077	0.100
AvgSpdd	−0.028	0.005	0.001	−0.038	−0.028	−0.018
DevSpdd	0.033	0.022	0.002	0.007	0.032	0.061
DifSpdu-d	0.032	0.007	0.001	0.019	0.033	0.046

not included in the Bayesian conditional logistic regression model because they have been controlled for in the data selection procedure using the case-controlled method.

An evaluation dataset was also created to test for the validity of the findings. The evaluation dataset includes all the observations of the traffic data in 5-min time intervals on eight selected segments between two neighboring detector stations in 2009. Note that the selected eight segments have low miss data rates on the I-880N freeway in 2009. To test for the validity of Eq. (7), the curves in Fig. 2 were developed on the basis of the evaluation dataset. The following assumptions were investigated: (a) whether there exists a linear relationship between P(y = 1|R > r) and the ratio between the sensitivity and false alarm rate, and (b) whether the slope is equal to the prior crash probability P(A).

On the basis ofthe evaluation dataset,the crash probability conditional on crash precursor P(y = 1|R > r), and the ratio between the sensitivity and false alarm rate for different thresholds were calculated. For a particular threshold, the probability P(y = 1|R > r) was calculated as the number of the identified crashes divided by the number of the observations that were identified as crash precursors. The frequency of crash and total observations on the selected eight segments in 2009 is given in Table 4. Due to missing data, the frequency of crashes in the dataset is smaller than that recorded in the SWITRS database. Therefore, before calculating the probability P(y = 1|R > r), the identified crashes in the dataset was multiplied by an adjustment factor, which was calculated as the recorded crash frequency in the SWITRS database divided by the crash frequency in the dataset (see Table 4). Similarly, the number of the observations that were identified as crash precursors was also multiplied by an adjustment factor to account for the missing non-crash cases. The estimated values of P(y = 1|R > r) were then fitted against the ratio between the sensitivity and false alarm rate, and the fitted curves are plotted in Fig. 2.

Fig. 2a illustrates the relationship between the probability P(y = 1|R > r) and the ratio of sensitivity and false alarm rate, i.e., the standardised likelihood, on the segment between the loop detector station 400,041 and 400,685. There appears to have a perfect linear relationship between P(y = 1|R > r) and the ratio of sensitivity and false alarm rate. Moreover, the slope of the fitted line appears to be very close to the prior crash probability P(y = 1), which is estimated to be 0.0011 (see Fig. 2a). Note that the prior crash probability was estimated by the crash countin one year on the segment by the total number of 5-min intervals during one year. The intercept of the linear regression line is also very close to 0, indicating the consistence between predictability in Eq. (7) and field crashes. Similar results can also be obtained in Fig. 2b–h. The validation results confirm the validity of Eq. (7).

5. Implications of the findings

When developing crash risk models, the following issues need to be carefully considered: (a) the selection of the threshold for identifying crash precursors; (b) the use of a matched case-control or unmatched case control study design; and (c) the selection of the control-to-case ratio in the case-controlled dataset. Addition efforts are needed to better understand how these issues affect the predictability of crash risk models. Considering the fact that these issues often interact with each other, they need to be considered in a coherent framework when a crash risk model is to be developed.

5.1. Threshold of crash precursor

In practical engineering applications, the threshold of a crash precursor directly determines when interventions should be triggered in proactive safety management systems. The aforementioned discussions indicate that the standardised likelihood, which can be approximately estimated as the ratio between the sensitivity and the false alarm rate, can be used for measuring to what extent the threshold of crash precursor affects the predictability of a real-time crash risk model. To identify how the selected threshold affects predictability, the ratios between sensitivity and false alarm rate of the crash risk model developed by the Bayesian logistic regression were calculated for different threshold values (see Table 5).

Note that when estimating the standardised likelihood associated with the crash risk model, the sensitivity and false alarm rate were estimated using the entire population, not the casecontrolled samples. Because all the crash cases are included in the case-control design, the sensitivity of a crash risk model generally represents the sensitivity on the entire population. For estimating the specificity/false alarm rate the unmatched case-control design is followed. The reason is that with the unmatched case-control design the non-crash cases are selected randomly. With the random sampling procedure it can be guaranteed that the selected non-crash cases are evenly distributed over time and space (see Fig. 3). Accordingly, the randomly selected non-crash cases in the unmatched case-control design provide a better estimate of the entire population than those in the matched case-control design do.

The estimated false alarm rate was plotted againstthe estimated sensitivity value in Fig. 4 to illustrate how the selection of threshold affected the predictability of the crash risk model. The slope

Table 4 Frequency of crashes and total observations for each segment.

| Segment | | Frequency of crashobservations | Adjustment for crash | | Frequency of all observations | Adjustment for all observations | | |-----------|--------|--------------------------------|----------------------|--------|-------------------------------|---------------------------------|--| | | Sample | Population | | Sample | Population | | | | Segment 1 | 111 | 118 | 1.063 | 69393 | 105120 | 1.515 | | | Segment 2 | 28 | 33 | 1.179 | 64946 | 105120 | 1.619 | | | Segment 3 | 32 | 54 | 1.688 | 76842 | 105120 | 1.368 | | | Segment 4 | 10 | 25 | 2.500 | 70912 | 105120 | 1.482 | | | Segment 5 | 34 | 44 | 1.294 | 61093 | 105120 | 1.721 | | | Segment 6 | 19 | 21 | 1.105 | 66286 | 105120 | 1.586 | | | Segment 7 | 31 | 39 | 1.258 | 74588 | 105120 | 1.409 | | | Segment 8 | 27 | 32 | 1.185 | 62186 | 105120 | 1.690 | |

Table 5 The relationship between sensitivity and the ratio of sensitivity and false alarm rate.

Threshold	Sensitivity	False alarm rate	Sensitivity/false alarm rate	P(y = 1 px > T)	P(y = 1 px > T)−1
0.00009	91.1%	58.4%	1.6	0.00034	2968
0.00012	82.1%	38.9%	2.1	0.00046	2194
0.00019	68.1%	22.7%	3.0	0.00065	1543
0.00029	57.1%	12.8%	4.5	0.00096	1038
0.00040	49.0%	10.1%	4.9	0.00105	954
0.00078	37.2%	6.2%	6.0	0.00130	772
0.00117	25.2%	2.8%	9.0	0.00194	514
0.00140	20.7%	1.9%	10.9	0.00235	425
0.00221	15.0%	0.8%	18.8	0.00405	247
0.00289	12.0%	0.5%	24.0	0.00518	193
0.00370	9.1%	0.2%	45.5	0.00983	102

Fig. 4. The ROC curve and the ratio of sensitivity and false alarm rate.

of the dash line represents the ratio between sensitivity and false alarm rate. As shown in Fig. 4, the slope of the dashed lines are always greater than 1 no matter what threshold is selected, indicating that the predictability of a crash risk model is better than its corresponding crash frequency model (see Eq. (7)). The curves in Fig. 4 also suggest that there exists a trade-off between the ratio between sensitivity and the ratio between false alarm rate and sensitivity. More specifically, the sensitivity increases as the ratio between sensitivity and false alarm rate decreases. In other words, one can deliberately increase the prediction accuracy of crash cases at the cost of reducing the predictability of crash risk models, and vice versa.

The trade-off between the predictability and the sensitivity should be carefully considered when selecting the threshold of a crash precursor. As mentioned before, the reciprocal of the predictability of a crash riskmodelmeasures the number of predictions that need to be made before a crash can be successfully predicted, which directly affects the cost of a proactive safety management system; while sensitivity measures the proportion of crash cases that can be successfully predicted, which directly affects the benefits of a proactive safety management system. As a result, the selection of the threshold heavily affects the cost and benefit of a proactive safety management system. The sensitivity, the ratio between sensitivity and false alarm rate, and the reciprocal of predictability calculated using the developed crash risk model are summarized in Table 5.

The last column in Table 5 represents the number of predictions that need to be made before a crash can be successfully predicted. The data in Table 5 clearly illustrate to what extent a crash risk model can be used to predict crashes and whatis the costthat needs to be paid. For example, when the threshold of crash precursors is set to be 0.00117, the crash risk model can successfully predict 25.2% of the total crashes, and 514 predictions need to be made before a crash can be successfully predicted.

In previous studies, the threshold of a crash precursor was usually selected such that a balance can be reached between sensitivity and specificity. However, when this rule is followed, the number of predictions that need to be made to predict a crash is found to be around 2000 (see Table 5). In this circumstance, the cost of preventing a crash may exceed the benefit of preventing a crash. Theoretically, the threshold of crash precursors should be selected such that a balance can be reached between the cost devoted to preventing crashes using proactive safety management systems and the benefits achieved by preventing crashes (see Fig. 5). This can be easily achieved through an economical appraisal that takes into consideration the monetary value of crashes and the cost associated with various proactive safety management techniques. The procedure provided by the Highway Safety Manual can be considered here for estimating the cost-benefit ratio associated with the selected threshold and the proactive safety management system (AASHTO, 2010).

5.2. Matched and unmatched case-control design

The predictability of the crash risk models that were developed using the matched and unmatched case-controlled dataset was compared in Fig. 6. It is clear that for a given level of sensitivity, the predictability of the crash risk model that is developed using the unmatched case-controlled sample is always better than that of the model developed using the matched case-controlled sample.

Table 6 Comparison of predictability of models for matched and unmatched sample.

| Sensitivity | False alarm rate | | P(y = 1 R > r)−1 | Difference in P(y = 1 R > r)−1 | | | |-------------|------------------|-----------|------------------|--------------------------------|------|--| | | matched | unmatched | matched | unmatched | | | | 0.05 | 0.013 | 0.001 | 1204 | 93 | 1111 | | | 0.10 | 0.036 | 0.003 | 1667 | 139 | 1528 | | | 0.15 | 0.062 | 0.009 | 1914 | 278 | 1636 | | | 0.20 | 0.096 | 0.016 | 2222 | 370 | 1852 | | | 0.25 | 0.126 | 0.028 | 2333 | 519 | 1814 | | | 0.30 | 0.166 | 0.044 | 2562 | 679 | 1883 | | | 0.35 | 0.192 | 0.057 | 2540 | 754 | 1786 | | | 0.40 | 0.235 | 0.069 | 2720 | 799 | 1921 | | | 0.45 | 0.255 | 0.086 | 2623 | 885 | 1738 | | | 0.50 | 0.278 | 0.103 | 2574 | 954 | 1620 | |

Fig. 5. The selection of threshold based on the ratio of sensitivity and false alarm rate.

Fig. 6. ROC curves of the models based on matched and unmatched case-control design.

The reciprocals of the predictability of these two models for different levels of sensitivity are calculated and included in Table 6. Depending on the selected threshold of crash precursors, the difference in the reciprocal of crash predictability between these two models varies between 1111–1921 with an average of 1689. On average, the crash risk model developed using the matched case-controlled sample needs to make 1689 more predictions to successfully predict a crash than the crash risk model developed using the unmatched case-controlled sample does.

5.3. Effects of control-to-case ratio

Previous studies in epidemiology suggested that the statistical power is negligible by using a control-to-case ratio beyond 4:1 (Abdel-Aty et al., 2005; Ahmed and Abdel-Aty, 2012). The control-to-case ratio around 4:1 is used in most of the previous studies for real-time crash risk modeling. To investigate how the control-to-case ratio affects the predictability, crash risk models were developed using the unmatched case-controlled sample with different control-to-case ratios varying from 1:1 to 10:1. Table 7 gives the crash predictability for different control-to-case ratios. In general, the reciprocal of crash predictability increases with an increase in the control-to-case ratio. However, the increase in control-to-case ratio does not lead to a clear increase in the crash predictability when the control-to-case ratio is beyond 4:1. This confirms the finding that the control-to-case ratio around 4:1 is optimal for real-time crash risk modeling.

6. Conclusions

This study aims to investigate the predictability of real-time crash risk models. The predictability is defined as the crash probability given the crash precursor identified by the crash risk model. An equation was derived based on the Bayes' theorem for estimating approximately the predictability of crash risk models. The estimated predictability was then used to quantitatively evaluate the effects of the threshold of crash precursor, the matched and unmatched case-control design, and the control-to-case ratio on the predictability of crash risk models. The following conclusions are made:

(1) the predictability of a crash risk model can be measured as the product of the prior crash probability and the ratio between sensitivity and false alarm rate. The ratio between sensitivity and false alarm rate measures to what extent the use of crash precursors and a crash risk model increases the predictability of crashes as compared with traditional crash frequency models.
(2) there is a trade-off between the predictability and sensitivity of a real-time crash risk model. The sensitivity increases as the ratio between sensitivity and false alarm rate decreases.
(3) for a given level of sensitivity,the predictability ofthe crash risk model that is developed using the unmatched case-controlled sample is always better than that of the model developed using the matched case-controlled sample. The threshold of crash precursor should be selected such that a balance can be reached between the cost devoted to preventing crashes using proactive safety management systems and the benefits that can be achieved by preventing crashes.
(4) the predictability of a real-time crash risk model increases with an increase in the control-to-case ratio. When the control-tocase ratio is beyond 4:1, the increase in control-to-case ratio does not lead to clear improvements in predictability.

The findings obtained from this study can also be used to help determine if a crash risk model is good or not. As mentioned

Table 7 Predictability of crash risk model for different control-to-case ratios.

Sensitivity	P(y = 1 R > r)−1					P(y = 1 R > r)
	1:1	2:1	3:1	4:1	5:1	10:1	1:1	2:1	3:1	4:1	5:1	10:1
0.05	93	93	93	93	93	93	0.010800	0.010800	0.010800	0.010800	0.010800	0.010800
0.10	231	185	139	139	139	139	0.004320	0.005400	0.007200	0.007200	0.007200	0.007200
0.15	340	278	247	247	247	247	0.002945	0.003600	0.004050	0.004050	0.004050	0.004050
0.20	486	440	417	394	394	394	0.002057	0.002274	0.002400	0.002541	0.002541	0.002541
0.25	574	556	519	519	519	519	0.001742	0.001800	0.001929	0.001929	0.001929	0.001929
0.30	756	694	679	664	664	664	0.001322	0.001440	0.001473	0.001507	0.001507	0.001507
0.35	833	794	754	741	741	741	0.001200	0.001260	0.001326	0.001350	0.001350	0.001350
0.40	845	833	833	822	822	822	0.001184	0.001200	0.001200	0.001217	0.001217	0.001217
0.45	916	905	905	905	905	895	0.001092	0.001105	0.001105	0.001105	0.001105	0.001117
0.50	981	972	963	954	954	954	0.001019	0.001029	0.001038	0.001049	0.001049	0.001049

before, the reciprocal of predictability represents the number of predictions that need to be made before a crash can be successfully predicted. It measures to what extent a crash risk model can be used to predict crashes and what is the cost that needs to be paid. Accordingly, the performance of crash risk model should be evaluated mainly based on its predictability (P(A|B)), which can be approximately estimated as the product ofthe prior crash probability and the ratio between sensitivity and false alarm rate. The data in Table 5 suggests that the reciprocal of predictability can reach 514. The result is quite encouraging considering the fact that more advanced modeling techniques, such as the hidden Markov model, random parameters regression and deep learning algorithm, can be used to further improve the predictability of crash risk models. In past, crashriskmodelinghas focusedonimprovingpredictionaccuracy based on case-controlled samples. This procedure is flawed because prediction accuracy on a case-controlled dataset is not indicative of the real prediction performance on the entire population. Such procedure may lead to an imbalance between the cost of using crash risk models for crash prevention and the benefit of preventing a crash. The results of this study also suggest that crash risk modeling should focus on improving the predictability of crash risk models (P(A|B)), not the prediction accuracy.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Project #: 51322810 and 51508093). The authors would like to thank PeMS and Caltrans for providing the crash and traffic data that were used in this paper.