category: literaturenote citekey: zhouquantilebasedscenariogeneration2025 title: Quantile-based scenario generation for automated vehicle safety evaluation authors: "Zhou, Hang; Ma, Chengyuan; Ma, Ke; Li, Xiaopeng" year: 2025 date: 2025-00-00 2025 doi: 10.1016/j.aap.2025.108043 publication: "Accident Analysis & Prevention" url: "https://doi.org/10.1016/j.aap.2025.108043" zotero_key: 3R2XHXXN zotero_storage: GWXGBKWJ collections: transpordi_planeerimine folder: 001_artiklid firstAuthor: "Zhou, Hang"

status: converted

Contents lists available at ScienceDirect

Accident Analysis and Prevention

journal homepage: www.elsevier.com/locate/aap

Quantile-based scenario generation for automated vehicle safety evaluation

Hang Zhou, Chengyuan Ma ∗ , Ke Ma, Xiaopeng Li ∗

Department of Civil and Environmental Engineering, University of Wisconsin-Madison, United States of America

A R T I C L E I N F O

Dataset link: https://github.com/michigan-traf fic-lab/Dense-Deep-Reinforcement-Learning, ht tps://github.com/CATS-Lab/Filed-Experiment-Data-ULTra-AV

Keywords: Automated vehicle Safety evaluation Scenario generation Car-following model

A B S T R A C T

As automated vehicles (AVs) are increasingly deployed, ensuring their safety and reliability is crucial before widespread adoption. Existing safety evaluation methods typically focus on generating a testing scenario library with a large number of safety-critical scenarios; however, this approach presents two key limitations. First, the safety testing of the testing scenario library is time-consuming, making it impractical to apply to the production qualification test for every individual production AV. Second, most methods aim to maximize the risks of the scenario, often overlooking that some highly hazardous situations are unavoidable. Considering these research gaps, this study introduces a quantile-based scenario generation method for AV safety evaluation. The proposed method generates scenarios with varying levels of risk, determined by a specified quantile of the risk index, enabling a comprehensive and efficient assessment of AV safety. With the knowledge of the quantile of the scenario library, safety evaluation can rapidly identify the safety performance of each individual AV with a theoretical bound using a limited number of tests. To address the challenge posed by the rarity of safety-critical events, an adaptive variance reduction framework based on importance sampling theory, combined with Particle Swarm Optimization, is employed to minimize estimation variance and optimize scenario distribution. Experiments validate the method's ability to reduce estimation variance in the multi-lane scenario and demonstrate how it compares the safety performance of commercialized AVs.

1. Introduction

The rapid evolution of automated vehicle (AV) technology has brought various types of AVs onto public roads, with continual iterations and advancements. For example, Baidu's Apollo Go currently operates over 300 driverless vehicles in Wuhan, China, offering more than 6000 rides daily (Wang et al., 2024), while Tesla is advancing its Full Self-Driving technology toward robotaxi deployment (Business, 2024). Each model and software update introduces distinct safety challenges, raising critical questions about AV reliability at scale. Addressing these concerns requires rapid, effective safety evaluation to ensure AVs can safely integrate into complex real-world environments.

Currently, AV testing and evaluation are conducted primarily through simulation tests and field experiments. The key to both methods lies in the generation of critical testing scenarios. Ding et al. (2023) reviewed the scenarios generation methods in AV testing, and separated them into three types: data-driven generation, adverbial generation, and knowledge-based generation. Data-driven generation (Li et al., 2019; Scanlon et al., 2021; Tan et al., 2021; Wang et al., 2022; Zhang et al., 2023; Yan et al., 2023) always has two steps. The first step directly samples from the dataset to reproduce the real-world log, which usually suffers from the problem of rareness. The second step is using density estimation models to learn the distribution of scenarios,

However, these traditional scenario generation methods face a challenge caused by the rarity of safety-critical scenarios in naturalistic driving environments (NDE), a phenomenon known as the ''curse of rarity'' (Liu and Feng, 2024). As noted in Feng et al. (2023), the probability of a crash occurring is approximately 1*.*9× 10−6 per mile for average American drivers. This rarity makes it difficult for simulations to capture all critical events needed for robust AV safety evaluation. To address this problem, Feng et al. (2023) and the previous studies by the authors Feng et al. (2021), Feng et al. (2020c,a ,b) proposed a testing scenario generation method based on Importance Sampling (IS), which

E-mail addresses: cma97@wisc.edu (C. Ma), xli2485@wisc.edu (X. Li).

https://doi.org/10.1016/j.aap.2025.108043

which enables the generation of unseen scenarios. Adversarial generation methods (Chen et al., 2021; Shi et al., 2024; Jia et al., 2024) consist of two components, one is the generator, and the other is the victim model, i.e., the AV. Risky scenarios will be created by attacking the AV system. The idea is similar to the famous generative adversarial network. Knowledge-based generation methods (Klischat et al., 2020; Wang et al., 2021; Song et al., 2022; Zhu et al., 2024) are introduced because the above two methods are always against traffic rules and physical laws. These methods will first explore rule-based methods that artificially design the structure and parameters of scenarios. Then, turn to the learning-based methods that use explicit knowledge to guide the generation.

^∗ Corresponding authors.

Fig. 1. Illustration of the quantile-based scenario.

is a variant of Monte Carlo simulation that can reduce the variance of the estimation. By applying IS with Deep Reinforcement Learning (DRL), they accurately estimate AV's crash rate. A similar idea of using IS in scenario generation is also applied in Zhao et al. (2016, 2017). Both papers utilize the University of Michigan Safety Pilot Model Deployment Program's data to accelerate the safety evaluation of AVs in specific driving scenarios. Zhao et al. (2017) emphasizes car-following maneuvers and employs modified human-controlled vehicle behaviors to intensify AV evaluations, significantly reducing test duration. In contrast, Zhao et al. (2016) targets lane-changing scenarios, especially unsafe cut-ins, using skewed statistics and the cross-entropy method to create high-risk testing scenarios, offering a potential reduction in AVs' development and validation time.

Although IS-based methods reduce the overall time for scenario generation, they still have some limitations. First, scenario generation methods in the literature typically generate scenario libraries based on AVs' behavior models. For AVs with similar behavior models, such as each individual production AV of the same brand and model, or models with small-scale iterations in software and hardware, these methods still require a large number of tests across all scenarios in the library to estimate the safety performance. Second, most scenario generation methods in the literature aim to maximize the risk of target scenarios. However, some highly dangerous scenarios may be unavoidable for vehicles. Identifying such scenarios does not contribute to the evaluation of AVs. These shortcomings imply that we need to generate a scenario library containing scenarios with specific risk quantiles, especially for the production qualification test for a large number of production AVs. With the quantile information, only a limited number of field tests are needed to guarantee the safety performance of an AV with theoretical bounds. As illustrated in Fig. 1, if the tested AV can safely pass the -quantile scenario , it indicates that the AV can ensure safety in at least percent of all scenarios. On the other hand, if the AV fails to safely pass the -quantile scenario , it indicates that the AV is at risk in at least 1 − percent of all scenarios. With just two tests, the safety performance of the tested AV can be bounded within the range of − . Besides, the dangerous scenarios that the AV fails to safely navigate can also serve as benchmarks to improve the hardware or software of the tested AV. Due to the diversity and robustness of such a scenario library, small-scale updates to AV software and hardware will not require updating the scenario library, thereby saving significant time in testing and development.

This study introduces a method for generating risk scenarios whose risk indices fall within a specified quantile of all scenarios. Accurately estimating the quantile still presents the challenge of efficiently searching within a high-dimensional scenario space while ensuring unbiasedness. To address this challenge, the algorithm is based on an adaptive variance reduction framework for IS to make the quantile estimation converge to the real value. Particle Swarm Optimization (PSO) is applied during the variance reduction process to optimize the distribution used in the IS. Finally, we validate our method in a car-following scenario and compare the safety performance of several commercialized AVs from the Ultra-AV dataset (Zhou et al., 2024). Our approach can be extended to other performance metrics of AVs, such as energy efficiency and mobility.

We propose a quantile-based scenario generation approach in which the target scenarios' risk indices fall within a specified quantile of all scenarios. This method can generate scenarios with varying risk levels to test the safety performance of each AV produced.
An adaptive variance reduction framework for quantile-based scenario generation is proposed. The framework utilizes IS to reduce the estimation variance and PSO to optimize the sampling distribution.

The remainder of the paper is organized as follows. Section 2 models the driving scenario quantile. Section 3 introduces our solution method based on the IS and the PSO. Section 4 presents the experimental results. Section 5 concludes the paper.

2. Modeling the scenarios quantile

We define the driving scenario as a Markov Decision Process (MDP) (Feng et al., 2023) in consecutive discrete time steps. Denote = {1, 2, …, } as the set of time steps, where is the total number of time steps and is the time interval. Let the variables of the entire MDP be denoted as = ((0), (0), (1), …, ( )). Here, () represents the vehicles' state at time step ∈ , including the position and speed of both the AV and background vehicles (BVs). () denotes the actions of the BVs at time step ∈ , which include the longitudinal acceleration and the lateral action (i.e., left lane change, straight, and right lane change).

According to the Markov property, the probability of each test scenario can be calculated as

$$P(X) = P(S(0)) \prod_{t=0}^{T} P(U(t) \mid S(t)), \tag{1}$$

where represents the probability. Therefore, the probability of a specific driving scenario under a particular AV controller in the NDE can be determined by estimating the initial state probability ((0)) and the probability of BV actions (() ∣ ()) for each scenario.

Denote the distribution of scenarios in the NDE as , i.e., ∼ (). Denote and as the probability and expectation when ∼ . Let = () ∈ [0, 1] represent the risk index of scenario . Let () be the distribution function of , which satisfies:

$$F(y) = PG(f{risk}(X) \leq y) = EG[I(f{risk}(X) \leq y)] = \intx I(f{risk}(X) \leq y) dG(x)$$ (2)

where (⋅) denotes the indicator function, which takes the value 1 (resp., 0) when its argument is true (resp., false).

The -quantile of for 0 < < 1 is denoted as = −1() = inf{ ∶ () ≥ }. Given ∈ [0, 1] and an AV model, the objective is to obtain the scenario corresponding to the risk index −1().

Fig. 2 describes the data flow of the quantile-based scenario generation and safety evaluation framework. The input for the framework consists of the behavior distribution for BVs and the behavior model of the tested AV. Based on these inputs, simulation and sampling can be conducted using methods such as Monte Carlo simulation. For each scenario in the sampled scenario set, the occurrence probability and

Fig. 2. Data flow of the quantile-based scenario generation and safety evaluation.

risk value are computed based on the BV behavior distribution and the chosen risk index. According to Eq. (2), these values are used to derive the probability distribution function of the risk values, enabling the extraction of the -quantile scenario for safety evaluation. If the AV can safely pass the -quantile scenario, it implies that it can safely pass of all scenarios.

3. Quantile-based scenario generation

3.1. The basic idea of IS

Since it is challenging to derive the mathematical formulation of (), we cannot compute directly. The Monte Carlo method estimates by sampling a set of i.i.d. samples ¹ , ² , … , with ∼ , ∈ = {1, 2, …, }. Then, through the simulation, ( can be obtained to estimate

$$\hat{F}{MC}(y) = \hat{F}{MC}(f{risk}(X)) = 1 - \frac{1}{n} \sum{i=1}^{n} I(f_{risk}(Xi) > f{risk}(X)), \tag{3}$$

$$\hat{\alpha}{MC,p} = \inf{y : \hat{F}{MC}(y) \ge p},\tag{4}$$

where ̂() and ̂, are the estimations of and using Monte Carlo method with samples.

The first step in Eq. (3) using Monte Carlo simulation can obtain an unbiased estimation of . However, with limited testing time and samples, the variance of the estimations will be large. The IS is applied to reduce the variance of the estimations without influencing the unbiased property.

To apply importance sampling to this problem, we choose a distribution , which is continuous for the probability measure associated with . This implies the existence of a ''density'' () satisfied that

$$g(x) = L_H(x)h(x) \tag{5}$$

where () and ℎ() are the density function of and , respectively. Then, we can express () using the distribution . With (), we have

$$\begin{split} F(y) &= \intx I(f{risk}(X) \leq y) dG(x) = \intx I(f{risk}(X) \leq y) \frac{dG(x)}{dH(x)} dH(x) \ &= \intx I(f{risk}(X) \leq y) L_H(x) dH(x) \end{split}$$

$$= EH[I(f{risk}(X) \le y) L_H(X)] = 1 - EH[I(f{risk}(X) > y) L_H(X)] \quad (6)$$

Similar to the Monte Carlo method, can be estimated by generating i.i.d. samples ¹ , ² , …, with ∼ (Glynn, 1996).

$$\hat{F}_{LS}(y) = 1 - EH[I(f{risk}(X) > y)L_H(X)]$$

$$= 1 - \frac{1}{n} \sum{i=1}^{n} I(f{risk}(X_i) > y)L_H(X_i), \qquad (7)$$

$$\hat{\alpha}_{IS,p} = \inf{y : \hat{F}_n(y) \ge p},$$ (8)

where ̂ () is the estimation of using IS and ̂, is the corresponding estimation of the p-quantile.

3.2. Variances of Monte Carlo and IS estimations

This section analyzes the variance of the estimations obtained by Monte Carlo and IS in the last section, i.e., ̂, and ̂,. According to Dong and Nakayama (2020), ̂ satisfies a central limit theorem (CLT):

$$\sqrt{n}[\hat{\alpha}_p - \alpha] \Rightarrow N(0, \tau^2), \quad \text{as } n \to \infty,$$ (9)

where represents a normal distribution, and the asymptotic variance ² has the form

$$\tau^2 = \frac{\psi^2}{f^2(\alpha)},$$ (10)

where is the density function, and the numerator is the asymptotic variance in the CLT for the estimation of ̂():

$$\sqrt{n}[\hat{F}(\alpha) - p] \Rightarrow N(0, \psi^2), \text{ as } n \to \infty.$$ (11)

Thus the reduction of the variance of ̂(), i.e., , can reduce the variance of ̂ . Specifically, for the Monte Carlo and IS methods, the formulations of are:

$$\psi_{MC}^2 = Var_G[I(Y \le \alpha)] = p(1 - p).$$ (12)

$$\psi_{\text{IS}}^2 = VarH \left[ I(f{risk}(X) > \alpha) L_H(X) \right]$$

$$= F \left[ I(f_{\text{c}}(X) > \alpha) LH(X) \right] - \left[ F \left[ I(f{\text{c}}(X) > \alpha) L_H(X) \right]^2 \right]$$ (13)

$$= EH \left[ \left( I(f{risk}(X) > \alpha) L_H(X) \right)^2 \right] - \left( EH \left[ I(f{risk}(X) > \alpha) L_H(X) \right] \right)^2$$ (14)

(14)

Fig. 3. The framework of the Adaptive Variance Reduction algorithm.

$$= EH \left[ I(f{risk}(X) > \alpha) L_H(X)^2 \right] - EG \left[ I(f{risk}(X) > \alpha) \right]^2$$ (15)

$$= EH \left[ I(f{risk}(X) > \alpha) L_H(X)^2 \right] - (1 - p)^2, \tag{16}$$

3.3. Adaptive variance reduction framework

As we analyzed in Eq. (16), the variance of the IS estimation is determined by the distribution . To reduce IS, this section presents an adaptive variance reduction algorithm. The objective of the algorithm is to approximate the optimal distribution, which can be formulated as:

$$\min{H} \psi{IS}^{2} = \min{H} E{H} \left[ I(f{risk}(X) > \alpha{p}) L_{H}(X)^{2} \right]$$ (17)

$$\approx \min{H} \frac{1}{n} \sum{i=1}^{n} I(f_{risk}(X_i) > \alpha_p) L_H(X_i)^2$$ (18)

$$\approx \min{H} \frac{1}{n} \sum{i=1}^{n} I(f_{risk}(Xi) > \hat{\alpha}{IS,p}) L_H(X_i)^2$$ (19)

Since is unknown, we can only use the estimation ̂ to approximate the variance in Eq. (19). An iterative framework is thus developed to solve this optimization problem, which allows ̂, to converge to . As shown in Fig. 3, the algorithm starts with distribution to estimate ̂ 0 , using the Monte Carlo method. Denote ∈ {1, 2, …, } as the iteration number, where is the maximum number of iterations. In the th iteration, the PSO is applied to derive a new distribution with a smaller variance. Scenario set sampled from distribution is used to estimate ̂ ,. By iterating these two steps, ̂ , will converge to

The optimization of Eq. (19) requires a large number of samples. Besides, the estimation of ̂ , in each iteration also requires to generate sample set from the new distribution . To reduce the computational time, samples from are all stored in set and reused in the PSO. For scenarios sampled from any distribution ^∗ , the variance IS can be expressed as:

$$\psi_{\text{IS}}^2 = EH \left[ I(f{risk}(X) > \alpha) L_H(X)^2 \right] - (1 - p)^2$$ (20)

$$= E{H^*} \left[ I(f{risk}(X) > \alpha) \left( \frac{g(X)}{h(X)} \right)^2 \frac{h(X)}{h^*(X)} \right] - (1 - p)^2$$ (21)

$$= E{H^*} \left[ I(f{risk}(X) > \alpha) LH(X) L{H^*}(X) \right] - (1 - p)^2, \tag{22}$$

where ^∗ ( ) = ()∕ℎ ∗ ().

Thus the objective function can be rewritten as:

$$\min{H} \psi{\text{IS}}^2 \approx \min{H} \frac{1}{|\mathcal{X}|} \sum{i=1}^{|\mathcal{X}|} I(f_{risk}(X_i) > \hat{\alpha}) L_H(Xi) L{H_i}(X_i), \tag{23}$$

where is the distribution scenario where is sampled from.

In the iterative process, estimating ̂ , from distribution is a critical step, as ̂ , is the final output of the algorithm, and each ̂ ,, ⁰ ^≤ ^≤ , is an important term in the objective function of the PSO. Algorithm 1 provides the details of this step. It is worth noting that, in Step 5, we need to record the probability and risk value of each sample for use in the subsequent optimization process of PSO. Steps

Algorithm 1 The pseudo-code of the quantile estimation.

1: Input: Distribution
2: Initialize an empty scenario set = ∅.
3: Sample i.i.d. replications ¹ , ² , ..., with ∼ , ∈ .
4: Run the simulation for all , ∈ to obtain ( ) and (
5: Store tuple ( , ( ), ( ) , ∀ ∈ into
6: Sort all scenarios in in ascending order according to (
7: Calculate ^∑ = from the end of the sequence to find the max which satisfies ^∑ = ^≥ (1 − ), and denote it as , i.e., = { ∶ ∑ = ^≥ (1 − )}.
8: Obtain ̂, = (
9: Output: ̂ , and

6–8 adopt the quantile estimation method proposed in Glynn (1996) to accelerate the estimation process.

Finally, Algorithm 2 summarizes the detailed procedures of the algorithm with pseudo-code.

Algorithm 2 The pseudo-code of the adaptive variance reduction framework.

1: Initialize a scenario set = ∅ and distribution ⁰ = .
2: Run Algorithm 1 with distribution ⁰ to obtain ̂ , and
3: for = 0 to do
4: Solve problem (23) using PSO with ̂ , and scenario set to obtain the distribution +1;
5: Run Algorithm 1 with distribution +1 to obtain ̂ , and +1
6: Update with
7: end for
8: Output: ̂ ,.

3.4. PSO

PSO is a heuristic evolutionary algorithm that optimizes by iteratively trying to improve a population of candidate solutions. We apply PSO to solve the optimization problem in Eq. (23) for the following two reasons. Firstly, due to the gradient-free and global search properties of meta-heuristic algorithms such as PSO, they exhibit faster convergence when dealing with large-scale optimization problems compared to machine learning algorithms. This characteristic can reduce the algorithm's overall runtime. This is particularly important for quantile problems that require solving the optimization problem iteratively multiple times. Secondly, compared to other metaheuristic algorithms, PSO is a better approach to solving problems with continuous decisions. This is because the particle states in PSO are defined in continuous space, enabling smooth searches within the continuous solution space. Moreover, as a population-based evolutionary algorithm, PSO supports parallel computation, further reducing runtime. We demonstrate in the experimental section that PSO achieves good results for car-following and lane-changing scenarios. For higherdimensional problems, reinforcement learning algorithms may better address the ''curse of dimensionality'' (Feng et al., 2021, 2023). The

Fig. 4. Illustration of the testing scenarios.

framework proposed in Algorithm 2 can be easily integrated with reinforcement learning algorithms.

The principle of PSO is straightforward. As an evolutionary algorithm, it solves a problem by having a population of candidate solutions, here dubbed particles, and moving them around in the search space according to a simple mathematical formula over the particle's position and speed. Each particle's movement is influenced by its local best-known position. Still, it is also guided toward the best-known positions in the search space, which are updated as other particles find better positions. This is expected to move the swarm toward the best solutions.

In our problem, the decision variables are the discrete distribution . A property of IS is conducted to accelerate the process. According to Owen (2013), for any action , if its probability in the original distribution () = 0, the probability in the new distribution is also 0. Therefore, we scan the NDE data to delete actions with a probability less than 1 ∗ 10−10 and fix the corresponding variables to 0. For other variables, the range is set to [1 ∗ 10−10 , 1]. The distribution will be normalized after the update in each iteration to ensure their sum equals 1. The optimization will stop by 3 criteria: maximum iteration times, minimum iteration times without a change in the objective value, and minimum iteration times with a small change in the objective value.

4. Numerical experiments

4.1. Experiment setting

4.1.1. Testing scenario

Similar to Feng et al. (2021), the proposed method is validated using a single-lane scenario involving one lead vehicle (LV) and one following AV, and a multi-lane scenario with three lanes and four BVs, as illustrated in Fig. 4. We primarily focus on these scenarios because the current functions of most production AVs are primarily limited to car-following and lane-changing, and there are very few AV behavior models and field test data available in the literature for more complex scenarios. The length of each scenario is set to = 5 seconds, with a time interval = 0*.2 seconds. At each time step, both the AV and the BVs select a longitudinal action from 31 discrete acceleration levels, ranging from [−4,* 2] m/s² with an interval of 0.2 m/s² . In the multilane scenario, the AV also needs to make decisions on left and right lane changes.

In our experiments, we use the inverse Time-to-Collision (TTC) (Kiefer et al., 2005) as the risk index (). TTC is commonly used to quantify the risk or proximity of a vehicle to a potential collision. Specifically, the TTC at time represents the time remaining until a collision occurs between two vehicles, assuming the current trajectory and speed differential are maintained. A lower inverse TTC indicates a safer situation, while a higher inverse TTC suggests increased risk. For this study, we set the inverse TTC range between [0, 1], where an inverse TTC greater than 1 indicates a collision. Notice that our proposed framework is applicable to any risk index that increases as the scenario's risk level increases. Other risk indices, such as Time Exposed TTC and Time Integrated TTC (Archer, 2005), or Deceleration Rate to Avoid Crash (Minderhoud and Bovy, 2001), can also be used. For more risk indices, we refer readers to the review paper (Shi et al., 2018). Here, we use inverse TTC because TTC is widely used in both industry standards (Forkenbrock et al., 2011; NHTSA, 2021) and literature (Wang et al., 2022). Additionally, inverse TTC magnifies dangerous scenarios where TTC is less than or equal to 1 and ensures that the risk level increases as the value increases.

The NDE data used in this experiment is sourced from Feng et al. (2023), and is available at https://github.com/michigan-traffic-lab/ Dense-Deep-Reinforcement-Learning. The NDE data is collected from a highway scenario, with vehicle speeds ranging from 20 m/s to 40 m/s and accelerations between −4 m/s² and 2 m/s² . The discretization intervals for distance, speed, and acceleration are set to 1 m, 1 m/s, and 0.2 m/s² , respectively.

4.1.2. AV models

Since the AV always follows the LV in our testing scenarios, the AV model is specifically a car-following model, which predicts the following AV's longitudinal acceleration (Zhang et al., 2024):

$$\hat{a}_t = f(a_t, r_t, v_t^f, \Delta v_t^f) \tag{24}$$

where is the longitudinal acceleration of the following AV, denotes the spatial gap (i.e., bumper-to-bumper distance) between the LV and the following AV, is the velocity of the following AV, and ^f represents the velocity difference between the LV and the following AV.

In the training stage, similar to Feng et al. (2023), we apply the famous Intelligent Driver Model (IDM) (Treiber et al., 2000) and Minimizing Overall Braking Induced by Lane change (MOBIL) (Kesting et al., 2007) models with stochasticity as the benchmark AV carfollowing and lane-changing models. To validate the generated scenarios, we use these scenarios to test the safety performance of six commercially available AVs, which are randomly selected from the Ultra-AV dataset (Zhou et al., 2024) and denoted as ℎ to ℎ . Since there are currently very few commercial AVs capable of autonomous lanechanging, as well as limited corresponding trajectory datasets, we only test these AVs in single-lane scenarios. The car-following models of each AV are calibrated using a Multilayer Perceptron (MLP) neural network model. MLP is a feedforward neural network often used in machine learning for its ability to capture nonlinear relationships between input features. In our implementation, the MLP consists of five fully connected layers with 8, 16, 8, 4, and 1 neurons, respectively. Table 1 shows the calibration results, where the root mean square error (RMSE) and the coefficient of determination (² ) are listed to show the accuracy of the car-following model. All vehicles demonstrate high prediction accuracy, accurately reflecting the car-following behaviors of the corresponding AVs.

4.2. Experiment results

4.2.1. Comparison between the Monte Carlo method and IS method

To validate the effectiveness of the quantile estimation method based on the IS, we conducted 100 estimations using both the Monte Carlo method and our proposed method. Figs. 5–8 illustrates the mean

Fig. 5. Comparison of the mean of the inverse TTC among different quantiles obtained by the Monte Carlo and the IS methods in the single-lane scenario.

Table 1 Calibration results of the car-following models for ℎ- ℎ

| Vehicle ID | 𝑉 𝑒ℎ𝐴 | 𝑉 𝑒ℎ𝐵 | 𝑉 𝑒ℎ𝐶 | 𝑉 𝑒ℎ𝐷 | 𝑉 𝑒ℎ𝐸 | 𝑉 𝑒ℎ𝐹 | | | | |------------|--------|--------|--------|--------|--------|--------|--|--|--| | RMSE | 0.0856 | 0.3719 | 0.1663 | 0.1592 | 0.1113 | 0.2151 | | | | | 𝑅2 | 0.8268 | 0.9583 | 0.9221 | 0.8962 | 0.9550 | 0.8478 | | | |

and variance of the 100 estimations across different quantile ranges in both single-lane and multi-lane scenarios.

Figs. 5 and 6 show the mean distribution, where we observe that the two methods yield similar mean values of inverse TTC across most quantiles. However, as the quantile range focuses on rarer scenarios with higher risk, a noticeable deviation in the mean values between the two methods emerges. Furthermore, we can observe that the probability of accidents in multi-lane scenarios is higher than in single-lane scenarios. This observation aligns with real-world expectations, as lanechanging maneuvers and the presence of more BVs increase the overall risk level.

In contrast, the variance distribution in Figs. 7 and 8 clearly shows that the IS-based method significantly reduces the estimation variance compared to the Monte Carlo method, especially in the single-lane scenario. In the single-lane scenario, the variance of both methods reaches its peak around 0.00005. This is because as the quantile approaches 1, the inverse TTC values converge to 1. When the quantile reaches 0.00005, the inverse TTC approaches its maximum value of 1, indicating a collision. For quantiles greater than 0.00005, as the rarity of events increases, the variance also increases. In the multi-lane scenario, we can still observe that the variance of the IS-based method is generally lower than that of the Monte Carlo method. However, compared to the single-lane scenario, the gap between the two methods becomes smaller. This is because in the multi-lane scenario, the IS method also needs to optimize the BVs' lane-changing decisions, which increases the dimension of the optimization problem, and PSO may not have fully converged to the global optimum. This suggests that while PSO remains effective in some complex scenarios, in extremely highdimensional cases, such as those involving intersections or pedestrians, more advanced optimization methods may be needed. Similar to the single-lane scenario, the peak variance in the multi-lane scenario occurs at approximately a quantile of 0.0015. Overall, in the tested scenarios, the IS-based method outperforms the standard Monte Carlo method.

4.2.2. Safety evaluation of commercialized AVs

This section utilizes the quantile-based scenario generation method to test the performance of commercialized AVs. We select six scenarios with quantiles ranging from 0.00002 to 0.0004 and run simulations with the car-following models of six commercialized AVs, as described in Section 4.1.2. Table 2 presents the testing results, where bold numbers indicate vehicles with inverse TTC values greater than or equal to those of the benchmark model IDM.

From Table 2, we observe that most vehicles demonstrate good safety performance compared to IDM. Only two vehicles, ℎ and ℎ, exhibit inverse TTC values that exceed those of IDM in certain

Fig. 6. Comparison of the mean of the inverse TTC among different quantiles obtained by the Monte Carlo and the IS methods in the multi-lane scenario.

Fig. 7. Comparison of the variance of the inverse TTC among different quantiles obtained by the Monte Carlo and IS methods in the single-lane scenario.

scenarios. Specifically, ℎ shows worse safety performance than IDM across multiple scenarios, experiencing three collisions in the testing scenarios. Therefore, we can preliminarily conclude that ℎ demonstrates poor safety performance. Notably, we found that all AVs experienced collisions in Scenario 1. To investigate the cause, we plotted the collision trajectories of Scenario 1 using ℎ as the following vehicle, as shown in Fig. 9. In this scenario, the LV performs an emergency brake with maximum deceleration, while the AV reacts slowly to the LV's braking, creating a potential collision hazard. This indicates that the selected AVs are still unable to guarantee safety in high-risk scenarios involving sudden braking by the LV.

5. Conclusions

In this study, we introduced a quantile-based scenario generation method for evaluating the safety performance of AVs. By generating scenarios with specified risk quantiles, we can conduct a more comprehensive assessment of AV safety than traditional methods and provide an efficient way for the production quality test of production AVs. An IS-based method is applied to solve the quantile-based scenario generation problem, in which an adaptive variance reduction framework combined with the PSO can iteratively reduce the sampling variance. The experimental results demonstrated that most of the AVs performed

Fig. 8. Comparison of the variance of the inverse TTC among different quantiles obtained by the Monte Carlo and IS methods in the multi-lane scenario.

Table 2 Testing results of the six commercialized AVs using the generated scenarios.

| Scenario | Quantile | Inverse TTC | | | | | | | | | |----------|----------|-------------|-------|-------|-------|-------|-------|-------|--|--| | | | IDM | 𝑉 𝑒ℎ𝐴 | 𝑉 𝑒ℎ𝐵 | 𝑉 𝑒ℎ𝐶 | 𝑉 𝑒ℎ𝐷 | 𝑉 𝑒ℎ𝐸 | 𝑉 𝑒ℎ𝐹 | | | | 1 | 0.00002 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | | | | 2 | 0.00004 | 0.946 | 1.000 | 0.647 | 0.746 | 0.688 | 0.676 | 0.637 | | | | 3 | 0.00006 | 0.790 | 1.000 | 0.541 | 0.697 | 0.517 | 0.467 | 0.541 | | | | 4 | 0.00008 | 0.663 | 0.676 | 0.232 | 0.436 | 0.738 | 0.462 | 0.376 | | | | 5 | 0.0001 | 0.598 | 0.520 | 0.500 | 0.580 | 0.528 | 0.518 | 0.492 | | | | 6 | 0.0002 | 0.520 | 0.376 | 0.345 | 0.432 | 0.386 | 0.373 | 0.351 | | | | 7 | 0.0003 | 0.475 | 0.110 | 0.116 | 0.396 | 0.472 | 0.296 | 0.258 | | | | 8 | 0.0004 | 0.465 | 0.104 | 0.154 | 0.428 | 0.342 | 0.423 | 0.298 | | |

Fig. 9. A collision case in the Scenario 1 where the ℎ collided with the LV.

well compared to the stochastic IDM, with one AV exhibiting notable safety deficiencies. However, none of the AVs were able to make safe decisions in scenarios where the LV performed an emergency brake.

Future work could extend the scope of the experiments by incorporating more complex urban scenarios, such as signalized intersections with cross-traffic and turning maneuvers. With the increasing deployment of high-level automation AVs, such as Tesla FSD, more trajectory data and driving behavior models will become available, which will facilitate the development of algorithms and experimental designs for these complex scenarios. These extensions would enable a more comprehensive validation of the quantile-based scenario generation method across a wider range of driving conditions, further enhancing the robustness of AV safety evaluations. To support such evaluations, developing more efficient optimization algorithms for IS distribution becomes crucial. Specifically, combining the fast convergence properties of heuristic algorithms like PSO with the strengths of DRL methods in handling high-dimensional problems would be a promising direction.

In addition, it is valuable to explore the design of a quantile-based testing scenario library that is universally applicable to all AV models. However, this problem remains highly challenging, as different AV models exhibit varying sensitivities to dangerous scenarios. An AV's control law can be specifically tuned to handle certain hazardous scenarios exceptionally well, without necessarily reflecting its overall safety performance. Unless a large number of strict and impractical assumptions are imposed on the behavior models, it is difficult to establish a universal scenario ranking. Addressing this issue requires further effort, a possible way might be changing the existing modeling method for the driving scenario.

Finally, we will also explore alternative approaches to achieve scenario generation under different quantiles. Considering that the driving process can be modeled as a Markov process, one potential direction is to apply Markov Chain steady-state probability theory to calculate the probabilities of all scenarios and subsequently determine the corresponding quantiles. This study can serve as a benchmark for evaluating the effectiveness of other scenario generation methods.

CRediT authorship contribution statement

Hang Zhou: Writing – review & editing, Writing – original draft, Validation, Methodology. Chengyuan Ma: Writing – review & editing, Writing – original draft, Supervision, Methodology, Conceptualization. Ke Ma: Writing – review & editing, Investigation, Formal analysis, Conceptualization. Xiaopeng Li: Writing – review & editing, Supervision, Funding acquisition, Formal analysis, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This research was sponsored by the National Science Foundation, United States of America, CPS: Small: NSF-DST: Turning ''Tragedy of the Commons (ToC)'' into ''Emergent Cooperative Behavior (ECB)'' for Automated Vehicles at Intersections with Meta-Learning (No. 2343167).

Data availability

The data used in this manuscript can be accessed at https://github. com/michigan-traffic-lab/Dense-Deep-Reinforcement-Learning and htt ps://github.com/CATS-Lab/Filed-Experiment-Data-ULTra-AV.

Code will be made available upon request.

References

Archer, J., 2005. Indicators for traffic safety assessment and prediction and their application in micro-simulation modelling: A study of urban and suburban intersections (Ph.D. thesis). KTH.
Business, C., 2024. Elon musk unveils his 'cybercab' robotaxi. https://www.cnn.com/ 2024/10/10/business/elon-musk-tesla-robotaxis/index.html, (Accessed 24 October 2024).
Chen, B., Chen, X., Wu, Q., Li, L., 2021. Adversarial evaluation of autonomous vehicles in lane-change scenarios. IEEE Trans. Intell. Transp. Syst. 23 (8), 10333–10342.
Ding, W., Xu, C., Arief, M., Lin, H., Li, B., Zhao, D., 2023. A survey on safetycritical driving scenario generation—A methodological perspective. IEEE Trans. Intell. Transp. Syst..
Dong, H., Nakayama, M.K., 2020. A tutorial on quantile estimation via Monte Carlo. Monte Carlo Quasi- Monte Carlo Methods: MCQMC 2018, Rennes, Fr. July 1– 6 3–30.
Feng, S., Feng, Y., Sun, H., Bao, S., Zhang, Y., Liu, H.X., 2020a. Testing scenario library generation for connected and automated vehicles, part II: Case studies. IEEE Trans. Intell. Transp. Syst. 22 (9), 5635–5647.
Feng, S., Feng, Y., Sun, H., Zhang, Y., Liu, H.X., 2020b. Testing scenario library generation for connected and automated vehicles: An adaptive framework. IEEE Trans. Intell. Transp. Syst. 23 (2), 1213–1222.
Feng, S., Feng, Y., Yu, C., Zhang, Y., Liu, H.X., 2020c. Testing scenario library generation for connected and automated vehicles, part I: Methodology. IEEE Trans. Intell. Transp. Syst. 22 (3), 1573–1582.
Feng, S., Sun, H., Yan, X., Zhu, H., Zou, Z., Shen, S., Liu, H.X., 2023. Dense reinforcement learning for safety validation of autonomous vehicles. Nature 615
Feng, S., Yan, X., Sun, H., Feng, Y., Liu, H.X., 2021. Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment. Nat. Commun. 12 (1), 748.
Forkenbrock, G., Heitz, M., Hoover, R.L., O'Harra, B., Vasko, S., Smith, L., 2011. A Test Track Protocol for Assessing Forward Collision Warning Driver-Vehicle Interface Effectiveness. Technical Report.
Glynn, P.W., 1996. Importance sampling for Monte Carlo estimation of quantiles. In: Mathematical Methods in Stochastic Simulation and Experimental Design: Proceedings of the 2nd St. Petersburg Workshop on Simulation. Citeseer, pp. 180–185.
Jia, L., Yang, D., Ren, Y., Qian, C., Feng, Q., Sun, B., Wang, Z., 2024. A dynamic test scenario generation method for autonomous vehicles based on conditional generative adversarial imitation learning. Accid. Anal. Prev. 194, 107279.
Kesting, A., Treiber, M., Helbing, D., 2007. General lane-changing model MOBIL for car-following models. Transp. Res. Rec. 1999 (1), 86–94.
Kiefer, R.J., LeBlanc, D.J., Flannagan, C.A., 2005. Developing an inverse time-tocollision crash alert timing approach based on drivers' last-second braking and steering judgments. Accid. Anal. Prev. 37 (2), 295–303.
Klischat, M., Liu, E.I., Holtke, F., Althoff, M., 2020. Scenario factory: Creating safetycritical traffic scenarios for automated vehicles. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems. ITSC, IEEE, pp. 1–7.
Li, W., Pan, C., Zhang, R., Ren, J., Ma, Y., Fang, J., Yan, F., Geng, Q., Huang, X., Gong, H., et al., 2019. AADS: Augmented autonomous driving simulation using
data-driven algorithms. Sci. Robot. 4 (28), eaaw0863. Liu, H.X., Feng, S., 2024. Curse of rarity for autonomous vehicles. Nat. Commun. 15 (1), 4808.
Minderhoud, M.M., Bovy, P.H., 2001. Extended time-to-collision measures for road traffic safety assessment. Accid. Anal. Prev. 33 (1), 89–97.
NHTSA, 2021. Metrics and Models to Evaluate Driving Safety. Technical Report, National Highway Traffic Safety Administration, URL https://www.nhtsa.gov/ document/metrics-and-models-evaluate-driving-safety, (Accessed 27 January 2025).
Owen, A.B., 2013. Monte Carlo theory, methods and examples. https://artowen.su. domains/mc/.
Scanlon, J.M., Kusano, K.D., Daniel, T., Alderson, C., Ogle, A., Victor, T., 2021. Waymo simulated driving behavior in reconstructed fatal crashes within an autonomous vehicle operating domain. Accid. Anal. Prev. 163, 106454.
Shi, H., Dong, S., Wu, Y., Nie, Q., Zhou, Y., Ran, B., 2024. Generative adversarial network for car following trajectory generation and anomaly detection. J. Intell. Transp. Syst. 1–14.
Shi, X., Wong, Y.D., Li, M.Z.F., Chai, C., 2018. Key risk indicators for accident assessment conditioned on pre-crash vehicle trajectory. Accid. Anal. Prev. 117, 346–356.
Song, Y., Chitturi, M.V., Noyce, D.A., 2022. Intersection two-vehicle crash scenario specification for automated vehicle safety evaluation using sequence analysis and Bayesian networks. Accid. Anal. Prev. 176, 106814.
Tan, S., Wong, K., Wang, S., Manivasagam, S., Ren, M., Urtasun, R., 2021. Scenegen: Learning to generate realistic traffic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 892–901.
Treiber, M., Hennecke, A., Helbing, D., 2000. Congested traffic states in empirical observations and microscopic simulations. Phys. Rev. E 62 (2), 1805.
Wang, X., Krasowski, H., Althoff, M., 2021. CommonRoad-RL: A configurable reinforcement learning environment for motion planning of autonomous vehicles. In: 2021 IEEE International Intelligent Transportation Systems Conference. ITSC, IEEE, pp. 466–472.
Wang, X., Peng, Y., Xu, T., Xu, Q., Wu, X., Xiang, G., Yi, S., Wang, H., 2022. Autonomous driving testing scenario generation based on in-depth vehicle-to-powered two-wheeler crash data in China. Accid. Anal. Prev. 176, 106812.
Wang, S., Zhao, Z., Xie, Y., Ma, M., Chen, Z., Wang, Z., Su, B., Xu, W., Li, T., 2024. Recent surge in public interest in transportation: Sentiment analysis of Baidu Apollo Go using weibo data. arXiv preprint arXiv:2408.10088.
Yan, X., Zou, Z., Feng, S., Zhu, H., Sun, H., Liu, H.X., 2023. Learning naturalistic driving environment with statistical realism. Nat. Commun. 14 (1), 2037.
Zhang, Y., Ma, K., Xu, Z., Zhou, H., Ma, C., Li, X., 2024. A modeling methodology for car-following behaviors of automated vehicles: Trade-off between stability and mobility. Available At SSRN 4866270.
Zhang, P., Zhu, B., Zhao, J., Fan, T., Sun, Y., 2023. Safety evaluation method in multilogical scenarios for automated vehicles based on naturalistic driving trajectory. Accid. Anal. Prev. 180, 106926.
Zhao, D., Huang, X., Peng, H., Lam, H., LeBlanc, D.J., 2017. Accelerated evaluation of automated vehicles in car-following maneuvers. IEEE Trans. Intell. Transp. Syst. 19 (3), 733–744.
Zhao, D., Lam, H., Peng, H., Bao, S., LeBlanc, D.J., Nobukawa, K., Pan, C.S., 2016. Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques. IEEE Trans. Intell. Transp. Syst. 18 (3), 595–607.
Zhou, H., Ma, K., Liang, S., Li, X., Qu, X., 2024. A unified longitudinal trajectory dataset for automated vehicle. Sci. Data 11 (1), 1123.
Zhu, B., Huang, Y., Zhao, J., Zhao, W., Zhang, P., Song, D., 2024. Camera-in-the-loop based test scenario generation method for pedestrian collision avoidance system. Accid. Anal. Prev. 207, 107737.

zhouquantilebasedscenariogeneration2025.md 55 KB História Raw