Obsidian/Zotero/001_artiklid/yeestimatingvariancebeforeafter2009.md

---

category: literaturenote citekey: yeestimatingvariancebeforeafter2009 title: Estimating the variance in before-after studies authors: "Ye, Zhirui; Lord, Dominique" year: 2009 date: 2009-00-00 2009 doi: 10.1016/j.jsr.2009.04.004 publication: Journal of Safety Research url: "https://doi.org/10.1016/j.jsr.2009.04.004" zotero_key: PF4MN2AB zotero_storage: QNI5K7C7 collections: transpordi_planeerimine folder: 001_artiklid firstAuthor: "Ye, Zhirui"

status: converted

Journal of Safety Research

j o u r n a l h om e p a g e : www. e l s ev i e r. c om / l o c a t e / j s r

Estimating the variance in before-after studies

Zhirui Ye a, ⁎, Dominique Lord ^b

• ^a Western Transportation Institute, Montana State University, P O Box 174250, Bozeman, MT 59717, USA
• ^b Zachry Department of Civil Engineering, Texas A&M University, 3136 TAMU, College Station, TX 77843, USA

article info abstract

Available online 11 June 2009

Keywords: Before-after study Variance estimation Bootstrap Resampling Non-parametric

Problem: To simplify the computation of the variance in before-after studies, it is generally assumed that the observed crash data for each entity (or observation) are Poisson distributed. Given the characteristics of this distribution, the observed value (xi) for each entity is implicitly made equal to its variance. However, the variance should be estimated using the conditional properties of this observed value (defined as a random variable), that is, f(xi|μi), since the mean of the observed value is in fact unknown. Method: Parametric and non-parametric bootstrap methods were investigated to evaluate the conditional assumption using simulated and observed data. Results: The results of this study show that observed data should not be used as a substitute for the variance, even if the entities are assumed to be Poisson distributed. Consequently, the estimated variance for the parameters under study in traditional before-after studies is likely to be underestimated. Conclusions: The proposed methods offer more accurate approaches for estimating the variance in before-after studies.

© 2009 National Safety Council and Elsevier Ltd. All rights reserved.

1. Introduction

The before-after study is a commonly used method for measuring the safety effects of a single treatment or a combination of treatments in highway safety (Hauer, 1997). Short of a controlled and full randomized study design, this type of study is deemed superior to cross-sectional studies since many attributes linked to the converted sites where the treatment (or change) was implemented remain unchanged. Although not perfect, the before-after study approach offers a better control for estimating the effects of a treatment. In fact, as the name suggests, it implies that a change actually occurred between the "before" and "after" conditions (Hauer, 2005a). Combined with the empirical Bayes (EB) technique, the before-after study can also minimize the bias caused by the regression-to-the-mean (RTM) commonly found in crash data analyses (Persaud, Retting, Garner, & Lord, 2001; Persaud, McGee, Lyon, & Lord, 2003). Despite their large popularity, it should be mentioned that not everyone agrees about their superiority over cross-sectional studies (Tarko, Eranky, & Sinha, 1998; Noland, 2003).

Before-after studies can be grouped into three types: the simple (naïve) before-after study; the before-after study with control groups; and the before-after study using the EB technique (also using a control group). The selection of the study type is usually governed by the availability of the data, such as crashes and traffic flow, and whether the transportation safety analyst has access to entities that are part of the reference group. The selection can also be influenced by the amount of available data (or sample size).

As described by Hauer (1997), the traditional before-after study (no matter which type is used) can be accomplished using two tasks. The first task consists of predicting the expected number (π̂) (in this paper, we will work with the estimated value; hence, π̂is an estimate of π) of target crashes for a specific entity (i.e., intersection, segment) or series of entities in the "after" period had the safety treatment not been implemented. The second task consists of estimating the number of target crashes (λ̂ ) for the specific entity in the "after" period. Here, the term "after" means the time period after the implementation of a treatment; correspondingly, the term "before" refers to the time before the implementation of this treatment. In most practical cases, either π̂or λ̂can be applied to a composite series of entities where a similar treatment was implemented at each entity.

Hauer (1997) proposed a four-step process for estimating the safety effects of a treatment. The process is described as follows:

• Step 1: For j= 1,2,…,n, estimate λ(j) and π(j). Then, compute the summation of the estimated and predicted values, such that λ̂ =Σλ(j) and π̂=Σπ(j).
• Step 2: For j= 1,2,…,n, estimate Var{λ̂ (j)} and Var{π̂(j)}. For each single entity, it is assumed that observed data (e.g., annual crash counts over a long timeframe) are Poisson distributed and λ̂ (j) can be approximated by the observed value in the before period. On the other hand, the calculation of Var{π̂(j)} will depend on the statistical methods adopted for the study (e.g., observed data in naïve studies, method of moments, regression models, EB technique). Assuming that crash data

^⁎ Corresponding author. Tel.: +1 406 994 7909. E-mail address: jared.ye@coe.montana.edu (Z. Ye).

in the before and after periods are mutually independent, then Var{λ̂ }=ΣVar{λ̂ (j)} and Var{π̂}=ΣVar{π̂(j)}.

Step 3: Estimate the parameters δ and θ, where δ̂ =π̂ - λ̂ (again, referring to estimated values) is defined as the reduction (or increase) in the number of target crashes between the predicted and estimated values, and θ ̂ =λ̂/π̂ is the ratio between these two values. The term θ has also been referred to in the literature as the index of effectiveness (Persaud et al., 2001). Hauer (1997) suggests that when less than 500 crashes are used in the before-after study, θ should be corrected to remove the bias caused by the small sample size using the following adjustment factor 1/[1+Var{π̂}/π̂

Step 4: Estimate the variances Var{δ̂ } and Var{θ ̂ }. These two variances are calculated using the following equations (note: Var{θ̂ } is also adjusted for the small sample size) below:

$$Var{\hat{\delta}} = Var{\hat{\lambda}} + Var{\hat{\pi}}$$ (1)

$$Var\left{\hat{\theta}\right} = \frac{\hat{\theta}^{2}\left[\left(Var\left{\hat{\lambda}\right}/\hat{\lambda}^{2}\right) + \left(Var\left{\hat{\pi}\right}/\hat{\pi}^{2}\right)\right]}{\left[1 + \left(Var\left{\hat{\pi}\right}/\hat{\pi}^{2}\right)\right]^{2}}$$ (2)

The four-step process provides a simple way for conducting beforeafter studies. Oneimportant assumptionwith this process is related to the computation of the variance Var(λ̂ ) (or Var{π}). As described above, ̂ observed crash data are assumed to be Poisson distributed for each entity and the observed data are directly used in the analysis. However, as noted by Hauer (1997), the varianceVar(λ̂ ) is in fact unknown. The properties of the Poisson distribution are in essence used to simplify the computation of the variance. In this case, the observed crash counts, used here as random variables, are used as a substitute for estimating the variance for each entity (i.e., the observation is assumed to be equal to its mean).1 Given the fact the mean of the observed value is unknown, the variance should be estimated using the conditional properties of the observed value (i.e., f(xi|μi)) (e.g., Cook & Wei, 2001; Diggle, Liang, & Zeger, 2002). Consequently, there is a need to evaluate how these conditional properties affect the estimation of the variance in the context of a before-after study.

The objectives of this paper are to evaluate whether or not the assumption that crash data should be used as a direct substitute to the variance is valid, even when one assumes the data are Poisson distributed for each entity, and if not, to examine how this may affect the estimation of the variance for calculating the inferences associated with the parameters used to estimate the safety effects in a beforeafter study. To accomplish the objectives of this study, parametric and non-parametric bootstrap resampling methods are investigated to evaluate this assumption. The bootstrap method is first applied to data simulated using a Poisson distribution and a Negative Binomial (or Poisson-gamma) distribution, and a mixture of these two distributions to evaluate its applicability in this research. Then, both methods are applied to two datasets of observed before and after crash data taken from the literature. The proposed methods are used to estimate λ̂, π̂, Var{λ̂ } and Var{π̂} and the output is compared with the traditional before-after method to compute these values.

The rest of this paper is divided into six sections. The first section presents the parametric method for estimating the variance of conditional random variables. The second section presents the characteristics of the bootstrap method used in this study. The third section covers the evaluation of the bootstrap method using simulated data. The fourth section presents the application of the methods to observe before and after data taken from the literature. The fifth section describes important discussion points associated with before-after studies and offers avenues for further work. The last section summarizes the key findings of this study.

2. Parametric Method

Since the mean of an entity is unknown, the analysis of the random variable must be carried out using the conditional properties of this variable with respect to the mean (i.e., f(x|η)), where η= {η(1), η(2),…, η(j)} (Cook & Wei, 2001; Agresti, 2002; Bolstad, 2004). Furthermore, the conditional property entails that the values can be approximated using any suitable distribution.

Researchers who have conducted before-after studies (non-randomized trials) using the same dataset in the before and after periods have analyzed the data using the conditional properties described above (note: in many cases, the mean of the observation is modeled as a random-effect variable). Examples of such observational studies where the mean of the Poisson distribution was modeled as a random-effect variable can be found in medicine (Cook & Wei, 2001), epidemiology (Laird & Ware, 1982; Diggle et al., 2002), and animal science (Schaik, Shoukri, Martin, Schukken, Nielen, 1999). Very recently, researchers in highway safety have also started using this conditional property for before-after studies (Persaud, Lan, Lyon, & Bhim, 2009; Park, Park, & Lomax, 2009). Depending upon the assumptions, the random-effect variable has been modeled using different marginal distributions. For example, Diggle et al. (2002) have proposed the Gaussian distribution for modeling the mean of the Poisson model. Because of the properties associated with the Poisson-gamma distribution (i.e., the closed form of the conjugate distribution), other researchers have proposed to model the mean using the gamma distribution (Cook & Wei, 2001; Diggle et al., 2002; Persaud et al., 2009; Park et al., 2009). As discussed by Lord, Washington, and Ivan (2005), it is important to point out that the Poisson-gamma distribution (as well as other mixed-Poisson distributions, such as the Poissonlognormal) is used to approximate the true characteristics of the motorvehicle crash process. It should be pointed out that by allowing the mean to follow a given distribution, the variance estimated will not be underestimated when a regression model is used in a before-after study.

Getting back to the primary objective of this analysis, if the means of the Poisson distributions for entities 1,2,…,j are assumed to follow a gamma distribution (η= {η(1),…η(j)} ~Gamma(ϕ, μ/ϕ)), it can be shown that the marginal distribution becomes the conjugate Poissongamma distribution, where ϕ is defined as the inverse dispersion parameter of the Poisson-gamma distribution. The mean and variance of η are μ and μ ² /ϕ, respectively.

Using the theorem proposed by Casella and Berger (1990), referred to as Conditional Variance Identity (CVI), it is possible to estimate the variance for a series of observed values, when each value is conditional upon the mean. This theorem states that "for any two random variables X and Y, Var{Y}=E[Var(Y|X)]+Var[E(Y|X)], provided that the expectations exist." For the curious reader, Agresti (2002) provides a very good discussion about the application of the CVI properties to Poisson random variables. His discussion in fact supports this work. This author states that the CVI needs to be used to estimate the variance of random variables because μ varies (i.e., unknown) due to unmeasured factors.

¹ To examine whether this assumption in before-after studies is reasonable, one can look into a single entity. If the actual expected number of crash counts (η(j)) for the jth entity in either the before or the after period, then it can be shown that Var{η̂|η̂(j)}= ΣVar{η̂ (j)}=Ση(j). In practice, η(j) is approximated using observed data x(j) (a random variable) and the true mean is therefore not known with certainty. By using the probability mass function (PMF) of the Poisson distribution, it is straightforward to compute the probability that Var{η(j)}=η(j)=x(j), assuming that x(j) represents the crash count over one year time period (or other very short time periods). It is obvious that the probability for the observed count (X) to equal the mean decreases as the mean η(j) increases. For example, the probability is about 40 percent when η(j)= 1, while it decreases to 10 percent when η(j)= 15. This entails that Var{η̂(j)}=x(j) may not be reasonable, since the count has a large probability not being equal to the "true" mean of an entity (if known). Thus, it is safe to assume that x(j) cannot be a good approximation of η(j).

Using X=η and Y=Ση(j), the variance (Var {Y}) can be derived as follows: Var{Y}=nμ+nμ ² /ϕ, where μ can be approximated by the mean value of entities μ̂ = ¹ P n ηð Þj , and the inverse dispersion parameter ϕ can be estimated from the sample of count data. Several estimators have been proposed to estimate the inverse dispersion parameter. They include the Method of Moments Estimate (MME; Anscombe, 1949), the Maximum Likelihood Estimate (MLE; Fisher,1941), the Maximum Quasi-Likelihood Estimate (MQLE; Clark & Perry, 1989), the multistage method (Willson, Folks, & Young, 1984), and the Bootstrapped Maximum Likelihood Estimate (BMLE; Zhang, Ye, & Lord, 2007).

3. Non-Parametric Bootstrap Resampling Method

In addition to the parametric method, a non-parametric method of bootstrap resampling was also used for estimating variance in this study (Efron & Tibshirani,1993). The bootstrapmethod is distribution-free and makes no assumption about the distribution of the observed data. Furthermore, the bootstrap method is usually not limited by the number of observations needed to get a good estimate of the mean and variance of the data, although it is suggested to use a sample size larger than five observations (Zoubir & Iskander, 2004). As discussed below, smaller sample size tends to underestimate the true variance of the distribution (Rich, 2001). This characteristic is unfortunately very common in highway safety (Lord, 2006). Nonetheless, it should be noted that the samples should have a good representation of the true distribution F (although we may not know its characteristics), since one cannot make good inferences based on unreliable data. Because of its useful properties, the non-parametric bootstrap resampling method has been used extensively in many fields of research, including geophysics (Tauxe, Kylstra, & Constable,1991; Kawano & Higuchi, 1995), biomedical engineering (Haynor & Woods,1989), and image processing (Hall,1989; Saradhi & Murty, 2001) among others.

Another advantage of the bootstrap method is related to the relative easiness for computing the values associated with the mean and variance. This computation can be accomplished using commercially available statistical software programs. For example, the software statistical program R (R Development Core Team, 2007), the one used in this study, provides a subroutine for performing bootstrap resampling. To extract a resample, one can simply use the following function in R.

sample xð Þ ;replace = T

Where x is the observed data, and T means TRUE for replacement.

4. Simulation Analysis for the Bootstrap Method

Simulation runs were performed in this part of the study to evaluate and validate the applicability of the non-parametric bootstrap method described above. Three series of simulations were analyzed. For the first two series, count data were generated using a Poisson or NB distribution for different sample sizes and distribution parameters, respectively. For the third series, count data were generated using a joint mixture of Poisson and NB distributions. The mean (λ) for the Poisson and NB distributions were equal, but the variance for each observation followed either one of the two distributions. Different proportions assigned to each distribution were evaluated. The three series were used to compare and validate the theoretical variances with the values obtained from the bootstrap output.

4.1. Simulation Framework

The data were simulated using a Monte Carlo simulation approach. All simulation runs were executed in R (R Development Core Team, 2007). The steps to simulate the data for the first two series were as follows:

• 1) Generate n i.i.d. variables from a Poisson distribution with mean value λ, or from a NB distribution with a mean value λ and an inverse dispersion parameter ϕ: χ= {X1, X2,…Xn} ~Poisson(λ) or χ= {X1, X2,…Xn} ~NB(λ,ϕ). Thus, n here refers to the sample size.
• 2) For the Poisson simulations, calculate the theoretical sum of variables S(χ)=nλ, and the theoretical variance of the sum S(χ) is equal to this sum. For the NB simulations, calculate the sum of variables S(χ)=nλ, then the variance of the sum S(χ) can be calculated using Var{S(χ)}=nλ+nλ² /ϕ.
• 3) Draw a sample from χ with replacement to obtain χ^j ⁎= {X¹ ⁎,…, Xn ⁎}, then calculate the sum of the resample S χ⁎ ^{= Pn} i= 1 X⁎ ⁱ ; repeat this resampling step B times to obtain S(χ¹ ⁎),…S(χ^B ⁎).
• 4) Calculate the variance of S(χ¹ ⁎),…,S(χ^B ⁎), that is, the bootstrap estimate of variance Var{S⁎}.
• 5) For the Poisson simulations, calculate the ratio r=Var{S⁎}/S(χ). For the NB simulations, calculate the ratio r=Var{S⁎}/Var{S(χ)}. The closer the ratio r to 1, the better the bootstrap estimate of the variance.
• 6) Repeat steps 1 to 5 for m times. Calculate the mean and variance of sample sum, Var{S⁎}, and r. Thus, the expected value and standard deviation of r are used for the evaluation of the bootstrap method.

The initial conditions, for the first two series of simulation runs, were the following:

The sample size (n): 20, 50, and 100;

The sample mean of the Poisson distribution (λ): 1, 5, 10, and 20; The sample mean and inverse dispersion parameter of the NB distribution (λ,ϕ): (1, 2), (5, 1), (5, 2), and (20, 2);

The number of resamples (B): 1000 times;

The number of iterations (m): 100 times.

For the third series, the same steps described above were used, but the proportion of count data generated by the Poisson and NB distributions were as follows (n= 100):

Scenario 1: Poisson= 50% (λ= 20), NB= 50% (λ= 20, ϕ= 2.0) Scenario 2: Poisson= 75% (λ= 20), NB= 25% (λ= 20,ϕ= 2.0) Scenario 3: Poisson= 25% (λ= 20), NB= 75% (λ= 20,ϕ= 2.0) As above, 100 iterations (m) were used.

4.2. Simulation Results

The simulation results for the first series (Poisson distribution) are shown in Table 1 for the sample sizes equal to 20, 50, and 100, respectively. From this table, the mean values of the variance estimated by the non-parametric bootstrap method are very similar to the theoretical values. Recall that the mean values for the sample sum were estimated using 100 iterations. Overall, the bootstrap method performed very well for estimating the variance. As expected, with a larger sample size, the bootstrap output provides better estimates. In fact, the variances of the ratio are decreasing with an increase in sample size. Similarly, with a larger sample size, the number of observations will provide a better estimate of the true distribution (see Lehmann, 1983, p. 114 for a discussion about the effects of sample size on the estimate of the Poisson distribution).

The simulation results for the second series (NB distribution) are summarized in Table 2. As described above, the NB simulations are used to characterize the over-dispersion found in count data or when the mean for each Poisson observation is gamma distributed. In fact, for some of these runs, the data are highly over-dispersed (i.e., ϕ= 1). Similar to the results shown for the Poisson distribution, the bootstrap method provides a good estimate of the theoretical variance in most cases, except for small sample sizes, where it slightly underestimates the true value. As expected, with the increase in sample size, the bootstrap estimations perform better.

Table 1 Results for the Poisson simulation.

Sample	Mean	Theoretical	Resampling	Ratio r a
Size	Value λ	Sum (or Variance)S(χ)	Variance Var(S⁎(χ))	mean	variance	MSEb
20	0.5	10	9.61	0.96	0.15	0.15
	1	20	18.05	0.90	0.10	0.10
	5	100	91.92	0.92	0.10	0.10
	10	200	187.86	0.94	0.09	0.09
	20	400	372.32	0.94	0.08	0.08
50	0.5	25	24.79	0.99	0.08	0.08
	1	50	48.8	0.98	0.06	0.06
	5	250	247.33	0.99	0.04	0.04
	10	500	496.55	0.99	0.04	0.04
	20	1000	980.34	0.98	0.03	0.03
100	0.5	50	50.10	1.00	0.04	0.04
	1	100	100.23	1.00	0.03	0.03
	5	500	502.35	1.00	0.03	0.03
	10	1000	1011.8	1.01	0.03	0.03
	20	2000	2017.4	1.00	0.02	0.02

Note:

The simulation results for the third series (a mixture of Poisson and NB distributions) are summarized in Table 3. This table shows that for every scenario evaluated, the bootstrap method properly estimated the variance as the predicted values are very close to the theoretical values. At this point, it is probably safe to state that the bootstrap method is a reliable approach for estimating the sample mean and sample variance of observed (and predicted) data.

In summary, the simulation results have shown that the nonparametric bootstrap resampling method is a simple and efficient approach for estimating the variance of random variables. This method offers reliable estimates even when the sample size is relatively small. In the next section, the bootstrap method is applied to observed data to examine whether the variance traditionally computed in before-after studies can be replicated.

5. Application of the Proposed Methods to Observed Data

This section describes the application of the proposed methods to observed before and after data. The methods were applied to two datasets. The first dataset contained crash data collected in North

Table 2 Results for the Negative binomial simulation.

Sample	Mean	IDPa	Theoretical	Theoretical	Resampling	Ratio r	c
Size	λ	ϕ	Sum S(χ)	Varianceb	Variance Var(S⁎ (χ))	mean	variance	MSEd
20	1	2	20	30	29.90	1.00	0.32	0.32
	5	1	100	600	575.49	0.96	0.29	0.29
	5	2	100	350	349.59	0.99	0.22	0.22
	20	2	400	4400	4165.20	0.95	0.20	0.20
50	1	2	50	75	76.48	1.02	0.12	0.12
	5	1	250	1500	1489.40	0.99	0.13	0.13
	5	2	250	875	838.45	0.96	0.08	0.08
	20	2	1000	11000	11192	1.02	0.10	0.10
100	1	2	100	150	153.51	1.02	0.05	0.05
	5	1	500	3000	2942.3	0.99	0.07	0.07
	5	2	500	1750	1748.3	1.00	0.06	0.06
	20	2	2000	22000	21616	0.98	0.04	0.04

Note:

• ^a IDP: Inverse dispersion parameter of a Poisson-gamma model (ϕ).
• ^b Var(Y)=λ+λ² /ϕ.

Table 3 Simulation results for the mixed-distribution simulation.

Proportion	Theoretical	Theoretical Variance	Mean Sample Suma	Resampling	Ratio r c
	Sum			Variance Var(S⁎(χ))	mean	variance
Scenario 1b
N1= 50	2000	12000	1987	11653	0.98	0.06
N2= 50
Scenario 2
N1= 25	2000	7000	1998	6924	0.99	0.12
N2= 75
Scenario 3
N1= 75	2000	17000	1990	16707	0.98	0.05
N2= 25

Note:

• ^a The sample sum reflects the use of equation λ̂ =Σλ̂ (j) or π̂=Σπ̂
• ^b In each scenario, the sample size is 100, which consists N1 data from the Negative Binomial distribution with mean λ= 20 and inversed dispersion parameter ϕ= 2.0, and N2 data from the Poisson distribution with mean λ= 20. 100 simulations are conducted for each scenario.
• ^c The ratio of bootstrap variance to theoretical variance (Var(S⁎(χ))/S(χ)). 100 r values are generated in each parameter setting (or each row) from the simulation. The mean and variance are calculated based on these 100 values.

Carolina for evaluating the safety effects of red-light cameras at signalized intersections. The second dataset included crash data collected for a study that examined the safety effects of converting signalized and unsignalized intersections to modern roundabouts. It is important to point out that this part of the analysis only focuses on the estimation of the variances and does not address nor discuss the conclusions reported in the original manuscript from which the data were extracted.

5.1. 1st Dataset – Red Light Running Crash Data

Red-light cameras have become an increasingly popular strategy for improving traffic safety at signalized intersections. This strategy aims at reducing the number of incidences associated with red-light running, with the hope that a reduction in the number of crashes occurring at signalized intersections would be observed. This topic has been studied extensively by a large number of researchers (Hooke, Knox, & Portas, 1996; Retting,Williams, Farmer, & Feldman,1999; Charlotte Department of Transportation [CDOT], 2000; Radalj, 2001; Lum & Wong, 2002; National Cooperative Highway Research Program [NCHRP], 2003; Flannery & Maccubbin, 2003). In most cases, they have reported a positive effect where a reduction in the number of motor-vehicle crashes caused by red-light running has been noted. However, it has recently been reported that the installation of cameras have also contributed to an increase in rear-end collisions, which according to some, reduces the overall effectiveness of this strategy (Council, Persaud, Eccles, Lyon, & Griffith, 2005; Washington & Shin, 2006).

For this analysis, data collected by the City of Charlotte Department of Transportation for estimating the safety effects of red-light cameras in Charlotte, NC were analyzed (CDOT, 2000). The data are summarized in Table 4. The data include information on the location where the cameras were installed as well as the crash frequencies (fatal injuries, injury types A, B and C, and property damage only) for the before and after periods. The CDOT installed cameras at 17 intersections and collected crash data three years before and after the installation. Table 4 shows that a reduction in motor-vehicle crashes (right-angle and rear-end) was not observed at every intersection where they were installed, but the overall number of crashes decreased in the after period.

For this dataset, a naïve before-after study was conducted.2 The variance-to-mean ratios were 8.38 and 7.02 for the before and after

^a The ratio of bootstrap variance to theoretical variance (Var(S⁎(χ))/S(χ)). 100 r values are generated in each parameter setting (or each row) from the simulation. The mean and variance are calculated based on these 100 values.

^b MSE: Mean Square Error. MSE= (Bias) ² +Variance.

^c The ratio of bootstrap variance to theoretical variance (Var(S⁎(χ))/S(χ)). 100 r values are generated in each parameter setting (or each row) from the simulation. The mean and variance are calculated based on these 100 values.

^d MSE: Mean Square Error. MSE= (Bias)2+Variance.

² The change in traffic flow was not applied here, since this information was not available. This does not affect the conclusion of the analysis or the premise of the paper.

Table 4 Crash data before and after the installation of red-light cameras in Charlotte, N.C. (CDOT, 2000).

Location	Crashes (3 years)		Location	Crashes (3 years)
	Beforea	Aftera		Beforea	After a
Beatties Ford Rd./ Hoskins Rd.	4	2	Westinghouse Blvd./S. Tryon	23	11
Morehead St./ College St.	29	10	Popla St./4th St.	24	20
Tyvola Rd./ Wedgewood Dr.	27	12	Albemarle Rd. @ Harris Bv.	61	34
Morehead St./ McDowell St.	18	10	Sharon Amity Rd. @ Central Av.	32	43
Brookshire Freeway/ Hovis Rd.	44	28	Eastway Dr. @ Kilborne Dr.	25	27
11th St./Brevard St.	26	16	Fairview Rd. @ Sharon Rd	27	28
Arrowood Rd./Nations Ford Rd.	9	14	Idlewild Rd. @ Independence Bv.	33	25
N.Tryon St./Harris Blvd.	43	46	Randolph Rd. @ Sharon Amity Rd.	18	12
South Blvd./ Archdale Dr.	25	29	Sum	468	367

Note:

periods, respectively. The crash data for both periods showed a significant amount of over-dispersion. The inverse dispersion parameter was estimated via the maximum likelihood estimation (MLE) techniques for both periods; note that the variance of the parameterization used in this research is the following Var(Y)=μ+μ² /ϕ. The inverse dispersion parameters were estimated to be 4.75 and 3.26 for the before and after periods, respectively.

For calculating the variances (Var{λ̂ } and Var{π}) (note: ̂ Var{π} here ̂ is estimated using observed crash counts), the traditional, the parametric, and the non-parametric bootstrap methods were applied to the data. The results are summarized in Table 5. From this table, it can first be observed that the mean values (λ̂and π) estimated by all ̂ methods are, as expected, exactly the same. On the other hand, the variances (Var{λ̂ } and Var{π}) estimated by the traditional method differ ̂ significantly from those estimated by the parametric and nonparametric methods. For instance, the variance Var{λ⁎̂ }=2800 calculated by the parametric method is much higher than the value obtained by the traditional method (Var{λ̂ }=S(χ)=367). This method underestimated both variances (Var{λ̂ } and Var{π}). The inferences associated ̂ with index of effectiveness θ̂have shown similar results, in which the variances of θ̂calculated by the proposed method are higher than the traditional method.

The results in Table 5 also show that Var{δ} and ̂ Var{θ} are also ̂ affected by the selection of the method. In both cases, the variances are much larger for the proposed methods than with the current method.

5.2. 2nd dataset-roundabout conversion

Many studies have shown that converting conventional intersections to modern roundabouts could significantly reduce motor-vehicle

Table 5 Comparisons of results for the red-light camera data.

Statistics	Traditional Method	Parametric Method	Bootstrap Method
π̂	468	468	468
ˆ λ	367	367	367
δ̂	101	101	101
Variance{π̂}	468	3174	2939
ˆ} Variance{λ	367	2800	2400
Variance{δ̂ }	835	5974	5339
θ̂	0.783	0.773	0.774
Variance{θ̂ }	0.003	0.020	0.018

Table 6 Crash data before and after the conversion of modern roundabouts (Persaud et al., 2001).

| Jurisdiction | Annual Crash Counts | | | | |-----------------------|--------------------------------|-------------------------------|--|--| | | Predicted After (crashes/year) | Observed After (crashes/year) | | | | Bradenton Beach, FL | 1.7 | 0.2 | | | | Fort Walton Beach, FL | 8 | 2.0 | | | | Gorham, ME | 6 | 3.2 | | | | Hilton Head, SC | 16 | 2.3 | | | | Manchester, VT | 0.4 | 0.4 | | | | Manhattan, KS | 3 | 0.0 | | | | Montpelier, VT | 1.2 | 0.3 | | | | West Boca Raton, FL | 1.5 | 1.7 | | | | Sum | 37.8 | 10.1 | | |

Note: the values in this table are already adjusted for the differences in time periods and traffic flows.

crashes and injuries (Schoon & Minnen, 1994; Troutbeck, 1993; Elvik, Mysen, & Vaa, 1997; Flannery & Elefteriadou, 1999; Persaud et al., 2001). Persaud et al. (2001), from which the data used in this study were extracted, analyzed the safety effects of converting stopcontrolled and signalized intersections to modern roundabouts. They collected data in seven U.S. states and performed a before-after study using the EB technique. In this study, the data collected for single lane, urban stop controlled intersections were used and a naïve before-after study was performed on the data. The characteristics of the data are summarized in Table 6. A total of eight intersections were converted to a modern roundabout. The data include the total number of crashes per year.

Results from the before-after study and the proposed method are shown in Table 7. This table shows that the variances calculated by both proposed methods are still larger than the sum of crashes over the entire group of entities for the before period. The estimated variances for all the methods are very similar for the after period. However, the bootstrap output shows that the variance is actually slightly lower than the mean (e.g., under-dispersion) for λ̂, which is explained by the very small sample size and low sample mean values. This is not uncommon for this type of dataset, as documented by Oh, Washington, and Nam (2006) and Lord, Geedipally, and Guikema (2009). The inverse dispersion parameter was estimated using the method of moments (MM); the MLE did not converge for this dataset (see Lord, 2006 about this issue). The inverse dispersion parameters were equal to 0.98 and 9.04 for the before and after periods, respectively. For the latter, it is most likely over-estimated, again caused by the low sample mean problem (Lord, 2006). Finally, Table 7 shows that the values θ̂ for the proposed methods are very comparable, but somewhat lower than the traditional method (0.260). Thus, it is possible that the traditional method could have underestimated the effectiveness of roundabout conversions (only for the naïve before-after study).

6. Discussion

The results of this study indicate that important limitations exist with the current approach for conducting traditional before-after

Table 7 Comparisons of results for roundabout conversions.

Statistics	Traditional Method	Parametric Method	Bootstrap Method
π̂	37.8	37.8	37.8
λ̂	10.1	10.1	10.1
δ̂	27.7	27.7	27.7
Variance{π̂}	37.8	220.6	189.7
Variance{λ̂ }	10.1	11.5	9.3
Variance{δ̂ }	47.9	232.1	199.0
θ̂	0.260	0.231	0.236
Variance{θ̂ }	0.008	0.011	0.010

^a Crash frequencies only include rear-end and right-angle crashes located on approaches where the camera is located.

studies. First, the results show that a large discrepancy exists between Var{λ̂ } and Var{π̂} estimated with the traditional method and the values calculated by the parametric and non-parametric bootstrap resampling methods. The traditional method uses observed data to approximate the expected value of a Poisson distribution and treat these values as known quantities for estimating the variance. However, the true variance is actually unknown, as noted by Hauer (1997). Thus, using observed counts as a substitute for the variance is not valid and will underestimate the true variance for each entity.

Second, as shown in the simulation results and the empirical analyses, the variances estimated using the traditional method is likely to be underestimated. This underestimation could have an important effect on the safety effectiveness of interventions or treatments. For instance, some interventions that would have been identified as being statistically significant (e.g., the 95%-percentile confidence interval does not include 1) using the traditional method could in reality be non-significant. This problem is even more important in the light of the methodology proposed by Hauer (2005b), which will be used to include or exclude accident modification factors (AMFs) in the upcoming Highway Safety Manual (HSM). This methodology uses information about the confidence intervals as well as the stability of the estimated factor of effectiveness (θ). Thus, a biased estimate of the variance will affect whether an AMF should be included or not in the HSM.

Third, the assumption that crash data are Poisson distributed, even for a single entity, needs to be re-examined in the context of the conditional properties of a random variable described in this manuscript. As discussed by Lord et al. (2005), crash data are in fact the results of Bernoulli trials with unequal probability of events. The Poisson distribution and mixed distributions, such as the Poissongamma or Poisson-lognormal, are in effect used as approximate distribution for analyzing motor-vehicle crashes. It is suggested to expand the work of Nicholson and Wong (1993) and Lord et al. (2005) for the analysis of crash counts at single entities.

7. Summary and Conclusions

This paper has documented the application of parametric and nonparametric bootstrap resampling methods for estimating the variance in before-after studies. This paper was motivated on the grounds that to estimate the variance, crash data on each entity that is part of the analysis are used as a substitute for estimating the variance. Although the actual variance is unknown, as noted by Hauer (1997), this assumption is used to simplify the computation of the variance. The literature on this topic has shown that the variance should be estimated using the conditional properties of random variables.

The objectives of this paper were to evaluate whether or not the assumption that crash data should be used as a direct substitute to estimate the variance is valid, and if not, to examine how this may affect the estimation of the variance for calculating the inferences of parameters used in traditional before-after studies. To accomplish the objectives of this study, a parametric method that incorporated the conditional properties of a random variable, and a non-parametric bootstrap method were investigated to evaluate this assumption. The bootstrap method was initially applied to data simulated using a Poisson distribution, a Negative Binomial distribution, and a mixture of these two distributions. Then, the traditional and the proposed methods were applied to two datasets containing observed before and after crash data taken from the literature.

The results of this study show that observed data should not be used as a substitute for the variance, even if the entities are assumed to follow a Poisson distribution. Consequently, the estimated variances are likely to be underestimated when the traditional method is used. The proposed methods can overcome this problem and provide a more accurate estimation of the variance. It is hoped that the results of this study will foster researchers to continue further on work conducting before-after studies, the characteristics of the Poisson distribution, and conditional probabilities applied in highway safety research.

Acknowledgements

The authors greatly acknowledge the comments and suggestions provided by Dr. Michael Longnecker from the Department of Statistics at Texas A&M University on the conditional properties of random variables, as described in this study. The authors also thank Dr. Alan Nicholson from the University of Canterbury for providing input relevant for this work.

References

Anscombe, F. J. (1949). The Statistical Analysis of Insect Counts Based on the Negative Binomial Distributions. Biometrics, 5, 165−173.

Agresti, A. (2002). Categorical Data Analysis. Hoboken, NJ: John Wiley & Sons.

Bolstad, B. M. (2004). Introduction to Bayesian Statistics. Hoboken, NJ: Wiley-Interscience. Casella, G., & Berger, R. L. (1990). Statistical Inference. Pacific Grove, CA: Duxbury Press. Charlotte Department Of Transportation [CDOT]. (2000). SafeLight Charlotte: annual report. August 1999 – July 2000.

Clark, S. J., & Perry, J. N. (1989). Estimation of the Negative Binomial Parameter k by Maximum Quasi-Likelihood. Biometrics, 45, 309−316.

Cook, R. J., & Wei, W. (2001). Selection effects in randomized trials with count data. Statistics in Medicine, 21, 515−531.

Council, F., Persaud, B., Eccles, K., Lyon, C., & Griffith, M. (2005). Safety evaluation of redlight cameras: executive summary. Report no. FHWA HRT-05-049. Washington, DC: Federal Highway Administration.

Diggle, P. J., Liang, K. Y., & Zeger, S. L. (2002). Analysis of Longitudinal Data. New York: Oxford Univ. Press Inc.

Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York: Chapman and Hall.

Elvik, R., Mysen, A. B., & Vaa, T. (1997). Traffic Safety Handbook (Norwegian). Oslo, Norway: Institute of Transport Economics (TØI).

Fisher, R. A. (1941). The Negative Binomial Distribution. Annals of Eugenics, 11, 182−187. Flannery, A., & Elefteriadou, L. (1999). A review of roundabout safety performance in the United States. Proceedings of the 69th Annual Meeting of the Institute of Transportation Engineers (CD-ROM). Washington, DC: Institute of Transportation Engineers.

Flannery, A., & Maccubbin, R. (2003). Using meta analysis techniques to assess the safety effect of red light running cameras. TRB 2003 Annual meeting paper Washington, DC: Transportation Research Board.

Hauer, E. (1997). Observational Before-After Studies in Road Safety: Estimating the Effect of Highway and Traffic Engineering Measures on Road Safety. Oxford, England: Pergamon Press, Elsevier Science Ltd.

Hauer, E. (2005). Cause and effect in observational cross-section studies on road safety. Draft report HSIS. Washington DC: Federal Highway Administration.

Hauer, E. (2005). A decision Based Approach to Include or Exclude AMFs. Draft Research Paper. Toronto, Ont.

Hall, P. (1989). Bootstrap methods for constructing confidence regions for hands. Communications in Statistics. Stochastic Models, 5(4), 555−562.

Haynor, D. R., & Woods, S. D. (1989). Resampling estimates of precision in emission

tomography. IEEE Transactions on Medical Imaging, 8, 337−343. Hooke, A., Knox, J., & Portas, D. (1996). Cost benefit analysis of traffic light & speed cameras. Police Research Series Paper, vol. 20. (pp. 57).

Kawano, H., & Higuchi, T. (1995). The bootstrap method in space physics: error estimation for the minimum variance analysis. Geophysical Research Letters, 22(3), 307−310.

Lehmann, E. L. (1983). Theory of point estimation. New York: Wiley.

Laird, N. M., & Ware, J. H. (1982). Random-effects models for longitudinal data. Biometrics, 38(4), 963−974.

Lord, D. (2006). Modeling motor vehicle crashes using Poisson-gamma models: Examining the effects of low sample mean values and small sample size on the estimation of the fixed dispersion parameter. Accident Analysis & Prevention, 38(4), 751−766.

Lord, D., Geedipally, S. R., & Guikema, S. (2009). Extension of the Application of Conway-Maxwell-Poisson Models: Analyzing Traffic Crash Data Exhibiting Under-Dispersion. Working Paper. College Station, TX: Zachry Department of Civil Engineering, Texas A&M University.

Lord, D., Washington, S. P., & Ivan, J. N. (2005). Poisson, Poisson-Gamma and zero inflated regression models of motor vehicle crashes: Balancing statistical fit and theory. Accident Analysis & Prevention, 37(1), 35−46.

Lum, K., & Wong, Y. (2002). Effects of red light camera installation on driver behaviour at a signalised cross-junction in Singapore. Road and Transport Research, 11(3).

National Cooperative Highway Research Program [NCHRP]. (2003). Impact of red light camera enforcement on crash experience. National Cooperative Highway Research Program, vol. 57.

Nicholson, A., & Wong, Y. (1993). Are accidents Poisson distributed? A statistical test. Accident Analysis and Prevention, 25(1), 91−97.

Noland, R. B. (2003). Traffic fatalities and injuries: the effect of changes in infrastructure and other trends. Accident Analysis and Prevention, 35, 599−611.

Oh, J., Washington, S. P., & Nam, D. (2006). Accident Prediction Model for Railway-Highway Interfaces. Accident Analysis and Prevention, 38(2), 346−356.

• Park, E. S., Park, J., & Lomax, T. (2009). Fully Bayesian multivariate approach to beforeand-after safety evaluation. TRB 2009 Annual meeting paper. Washington DC: Transportation Research Board.
• Persaud, B. N., Retting, R., Garder, P., & Lord, D. (2001). Observational before-after study of U.S. roundabout conversions using the empirical Bayes method. Transportation Research Record, 1751, 1−8.
• Persaud, B., Lan, B., Lyon, C., & Bhim, R. (2009). Comparison of empirical Bayes and full Bayes approaches for before-and-after road safety evaluations. TRB 2009 Annual meeting paper. Washington DC: Transportation Research Board.
• Persaud, B. N., McGee, H., Lyon, C., & Lord, D. (2003). Development of a Procedure for Estimating the Safety Effects for a Contemplated Traffic Signal Installation. Transportation Research Record, 1840, 96−103.
• R Development Core Team (2007). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria ISBN 3-900051- 07-0, Retrieved Jun 18, 2007, from http://www.R-project.org
• Radalj, T. (2001). Evaluation of effectiveness of red light camera programme in Perth. Road Safety Research, Policing and Education Conference, Melbourne, Australia.
• Retting, R., Williams, R., Farmer, C., & Feldman, A. (1999). Evaluation of red light camera enforcement in Oxnard, California. Accident Analysis and Prevention, 31, 69−74.
• Rich, H. (2001). Using the Bootstrap with small data sets: the smoothed Bootstrap. RSS Matters, Benchmarks Online, University of North Texas Retrieved May 10, 2007, from http://www.unt.edu/benchmarks/archives/2001/october01/rss.htm
• Saradhi, V. V., & Murty,M. N. (2001). Bootstrapping for efficient handwritten digit recognition. Pattern Recognition, 34(5), 1047−1056.
• Schaik, G. V., Shoukri, M., Martin, S., Schukken, Y. H., Nielen, M., Hage, J. J., et al. (1999). Modeling the Effect of an Outbreak of Bovine Herpesvirus Type 1 on Herd-Level Milk Production of Dutch Dairy Farms. Journal of Dairy Science, 82, 994-952.
• Schoon, C., & Minnen, M. J. (1994). The safety of roundabouts in the Netherlands. Traffic Engineering and Control, 142−148.
• Tarko, A. P., Eranky, S., & Sinha, K. S. (1998). Methodological considerations in the development and use of crash reduction factors. Preprint CD. 77th Annual Meeting of the Transportation Research Board.

• Tauxe, L., Kylstra, N., & Constable, C. (1991). Bootstrap statistics for Palaeomagnetic data. Journal of Geophysical Research, 96, 11723−11740.

• Troutbeck, R. J. (1993). Capacity and design of traffic circles in Australia. Transportation Research Record 1398, TRB, 68-74. Washington, DC: National Research Council.

• Washington, S. P., & Shin, K. (2006). Impact of red light cameras on safety in Arizona. Proceedings of the 85th Annual Meeting of the Institute of Transportation Engineers (CD-ROM) Washington, D.C.: Institute of Transportation Engineers.

• Willson, L. J., Folks, J. L., & Young, J. H. (1984). Multistage Compared with Fixed-Sample-Size Estimation of the Negative Binomial Parameter k. Biometrics, 40, 109−117.

• Zhang, Y., Ye, Z., & Lord, D. (2007). Estimating the Dispersion Parameter of the Negative Binomial Distribution for Analyzing Crash Data Using a Bootstrapped Maximum Likelihood Method. Transportation Research Record, 2019, 15−21.

• Zoubir, A. M., & Iskander, D. R. (2004). Bootstrap techniques for signal processing. Boston: Cambridge University Press.

Zhirui Ye, Ph.D., is a research scientist at Western Transportation Institute, Montana State University. Dr. Ye holds his Ph.D. in Transportation Engineering from Texas A&M University, and B.S. and M.S in Civil Engineering from Southeast University (China). Dr. Ye's current research interests include traffic safety, traffic operations, Intelligent Transportation Systems, and surface transportation weather.

Dr. Lord is an Assistant Professor in the Zachry Department of Civil Engineering. Prior to joining Texas A&M University in September 2004, he was an Associate Research Scientist with the Center for Transportation Safety at the Texas Transportation Institute (TTI), and a research assistant at Ryerson University. Dr. Lord holds a B.Eng. (1992) in Civil Engineering from McGill University, and an M.A.Sc. (1994) and Ph.D. (2000) from the University of Toronto. Dr. Lord's primary interests are conducting fundamental research on accident analysis methodology, new and innovative statistical methods for modeling motor vehicle collisions (including Bayesian statistics), and before/after evaluation techniques. His other research interests include problems associated with the crash data collection process, safety audits, human factors related to older drivers and pedestrians, and traffic flow theory. He has had numerous papers published in peer-reviewed journals and international conferences.