--- category: literaturenote citekey: weiadbenchhierarchicalcot2025 title: "AD^2-Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions" authors: "Wei, Zhaoyang; Qiang, Chenhui; Jiang, Bowen; Han, Xumeng; Yu, Xuehui; Han, Zhenjun" year: 2025 date: 2025-00-00 2025 url: "https://arxiv.org/abs/2506.09557v1" zotero_key: WC33RDRU zotero_storage: PF3KCMXV collections: transpordi_planeerimine folder: 001_artiklid firstAuthor: "Wei, Zhaoyang" status: converted --- # AD2 -Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions Zhaoyang Wei1∗ , Chenhui Qiang1∗ , Bowen Jiang1∗ , Xumeng Han1 , Xuehui Yu2 , Zhenjun Han1† 1University of Chinese Academy of Sciences(UCAS), 2Tencent CDG # Abstract Chain-of-Thought (CoT) reasoning has emerged as a powerful approach to enhance the structured, multi-step decision-making capabilities of Multi-Modal Large Models (MLLMs), and is particularly crucial for autonomous driving under adverse weather conditions and complex traffic environments. However, existing benchmarks have largely overlooked the need for a rigorous evaluation of CoT processes in these specific and challenging scenarios. To address this critical gap, we introduce AD2 -Bench, the first Chain-of-Thought benchmark specifically designed for autonomous driving under adverse weather and complex scenes. AD2 - Bench is meticulously constructed to fulfill three key criteria: comprehensive data coverage across diverse adverse environments, fine-grained annotations that support multi-step reasoning, and a dedicated evaluation framework tailored to assess CoT performance. The core contribution of AD2 -Bench is its extensive collection of over 5.4k high-quality, manually annotated CoT instances. Each intermediate reasoning step in these annotations is treated as an atomic unit with explicit ground truth, enabling unprecedented fine-grained analysis of MLLMs' inferential processes under text-level, point-level, and region-level visual prompts. Our comprehensive evaluation of state-of-the-art MLLMs on AD2 -Bench reveals the accuracy below 60%, highlighting the benchmark's difficulty and the need to advance robust, interpretable end-to-end autonomous driving systems. AD2 -Bench thus provides a standardized evaluation platform, driving research forward by improving MLLMs' reasoning in autonomous driving, making it an invaluable resource. # 1 Introduction Visual perception is an indispensable capability for autonomous driving systems, which enables environmental comprehension and navigational decision-making. The sophisticated scene understanding afforded by such visual data directly underpins essential tasks like nuanced motion analysis [\[15\]](#page-9-0), optimal path determination, and precise spatial occupancy assessment [\[30,](#page-10-0) [9\]](#page-9-1). Modular architectures, frequently adopted in these systems, often result in inadequate inter-module synergy and compromised overall system performance, particularly in dynamic and complex real-world environments. The recent emergence of multimodal large language models (MLLMs) [\[35,](#page-11-0) [33,](#page-10-1) [28\]](#page-10-2), exhibiting humanlike understanding and reasoning capabilities, offers new avenues for tackling complex systemic issues. The end-to-end integration of MLLMs alleviates the fragmentation inherent in modular designs, paving the way for more robust and adaptable solutions to navigate the challenges arising from adverse conditions and unforeseeable situations encountered in autonomous driving [\[10,](#page-9-2) [17,](#page-10-3) [42,](#page-11-1) [47\]](#page-11-2). While ∗ Equal contribution. † Corresponding authors. (hanzhj@ucas.ac.cn) | Table 1: Comparison between AD2
-Bench and existing datasets. AD2
-Bench is the first large-scale | |---------------------------------------------------------------------------------------------------------| | multimodal dataset in adverse conditions for autonomous driving with hierarchical CoT benchmark. | | Dataset | Multimodal | Adverse Weather | Complex Scene | QA Pairs | Multi-prompt | Chain of Thought | Driving Suggestion | |----------------------|------------|-----------------|---------------|----------|--------------|------------------|--------------------| | CODA [29] | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | | StreetHazards [22] | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | | nuScenes-QA [40] | ✓ | ✗ | ✗ | 83k | ✗ | ✗ | ✗ | | BDD-X [26] | ✓ | ✗ | ✗ | - | ✗ | ✗ | ✗ | | DRAMA [38] | ✓ | ✗ | ✗ | 14k | ✗ | ✗ | ✓ | | DriveLM [42] | ✓ | ✗ | ✗ | 15k | ✗ | ✗ | ✓ | | CODA-LM [29] | ✓ | ✗ | ✓ | 24k | ✓ | ✗ | ✓ | | -Bench (ours)
AD2 | ✓ | ✓ | ✓ | 70k | ✓ | ✓ | ✓ | ![](_page_1_Figure_2.jpeg) Figure 1: Adverse conditions include adverse weather (up) and complex scenarios (down). the versatile capabilities of MLLMs have undergone extensive evaluation, their robust assessment for autonomous driving applications remains significantly hampered by prevailing and critical deficiencies within current benchmarks: 1) Data Coverage: Current benchmarks [\[40,](#page-11-3) [24\]](#page-10-7) predominantly feature benign driving conditions. The scarcity of real-world data for adverse weather and complex scenarios (*e.g*., rain, fog, nighttime, intersections) compels reliance on synthetic data that often lacks requisite realism and diversity [\[2,](#page-9-3) [16\]](#page-10-8). 2) Annotation Quality: Meticulous manual annotation for crucial high-quality ground truth is often prohibitively costly. This frequently leads to using model-generated labels [\[10\]](#page-9-2), whose questionable accuracy and consistency [\[52\]](#page-11-5) subsequently compromise the reliability and availability of the data. 3) Evaluation Paradigm: Direct reasoning evaluation of MLLMs often overlooks critical details, especially under adverse driving conditions. While Chain-of-Thought (CoT) prompting [\[49,](#page-11-6) [57\]](#page-12-0) can elicit a more nuanced reasoning process, benchmarks to assess the fidelity of the reasoning steps vital for complex autonomous driving scenarios are severely lacking. To address these deficiencies, this paper introduces AD2 -Bench, the first benchmark for evaluating CoT reasoning in MLLMs under demanding autonomous driving conditions. AD2 -Bench consists of approximately 10k real-world images, including four major dimensions and 33 subtasks, meticulously curated to specifically target *adverse weather and complex scenes*. As Fig. [1](#page-1-0) illustrates, these images depict a wide spectrum of demanding scenarios, including severe weather, challenging lighting conditions, and intricate traffic situations, many of which push the boundaries of even human driving proficiency. Benchmarks capable of robustly assessing CoT fidelity for MLLMs, particularly within the complex scenarios encountered in autonomous driving, remain scarce. To bridge this gap, AD2 -Bench provides 5.4k images meticulously and manually annotated CoT instances developed by domain experts. A key innovation is its unique CoT pipeline where each intermediate reasoning step is treated as an atomic question with already annotated ground truth by experts. This granular, step-by-step annotation, a novel contribution to autonomous driving datasets (see Tab. [1\)](#page-1-1), facilitates a profound evaluation of inferential integrity that extends beyond mere final answer accuracy. Leveraging this detailed annotation, we propose a multi-dimensional metric for the perception→understanding→reasoning CoT hierarchy, jointly accounting for semantic similarity, reasoning coherence, and decision consistency, thus offering interpretable and quantifiable assessment for MLLMs. Probing the fine details of MLLM reasoning during CoT is essential, but vision encoder weaknesses in adverse conditions present a hurdle. To overcome this and enable deeper reasoning assessment, AD2 -Bench uniquely incorporates diverse *visual prompts* (illustrated in Fig. [3\)](#page-3-0)—a feature rare in existing AD datasets (Tab. [1\)](#page-1-1). These include region-level, text-level, and point-level cues providing ![](_page_2_Figure_0.jpeg) Figure 2: (a) Task Categories. Our benchmark spans 4 key dementions and 33 subtasks highly related to real-world scenarios, including 10k high-resolution images and 70K annotations. (b) Model Performance. Average accuracies of advanced MLLMs are shown on the dataset. targeted perceptual assistance, facilitating focused evaluation of the LLM's reasoning on critical scene elements. We conclude the main advantages of AD2 -Bench as follows: - (1) Scale & Depth: The CoT benchmark AD2 -Bench is proposed for adverse-weather and complex-scene autonomous driving, containing 10K real-road images and 70K QA pairs, of which 5,406 samples include the full CoT chain. - (2) Annotation Quality: All VQA and CoT steps are manually annotated by domain experts; the integration of region, text, and point-level visual prompts both boosts verification efficiency and explicitly steers model attention during evaluation, leading to more reliable reasoning. - (3) Novel Metrics: We propose multi-dimensional metrics for the perception → understanding → reasoning CoT hierarchy, jointly accounting for semantic similarity, reasoning coherence, and decision consistency, thus offering interpretable and quantifiable assessment for MLLMs. - (4) Task Difficulty & Value: Under extreme weather and complex traffic, state-of-the-art models struggle to surpass 60% accuracy on AD2 -Bench, which not only reveals significant challenges for robust end-to-end autonomous driving but also necessitates a rigorous evaluation of general MLLM perception, understanding, and reasoning under adverse scenarios. # 2 Related Work #### 2.1 Multimodal Large Language Models in Autonomous Driving Multimodal large language models (MLLMs) fuse vision encoders with powerful language backbones, enabling unified visual–textual reasoning [\[5,](#page-9-4) [8,](#page-9-5) [47,](#page-11-2) [43,](#page-11-7) [23,](#page-10-9) [59,](#page-12-1) [19,](#page-10-10) [36,](#page-11-8) [32,](#page-10-11) [12,](#page-9-6) [54,](#page-12-2) [17\]](#page-10-3). Representative systems such as BLIP-2 [\[28\]](#page-10-2), LLaVA series [\[35,](#page-11-0) [33,](#page-10-1) [34,](#page-11-9) [1\]](#page-9-7), and Qwen-VL series [\[3,](#page-9-8) [48,](#page-11-10) [4\]](#page-9-9) achieve impressive results on image captioning, VQA, and cross-modal reasoning with minimal task-specific adaptation. Although architectures differ—*e.g*., BLIP-2's frozen encoders plus "Querying Transformer" versus LLaVA's CLIP-based projection layer—all employ large-scale image–text pre-training followed by instruction tuning, yielding rich semantic grounding that is attractive for safety-critical domains like autonomous driving. Building on these foundations, recent studies embed MLLMs directly into driving stacks[\[20,](#page-10-12) [50,](#page-11-11) [42,](#page-11-1) [47,](#page-11-2) [37,](#page-11-12) [56\]](#page-12-3). DriveLLM augments a conventional perception–planning pipeline with LLMbased commonsense reasoning, improving high-level maneuver selection and providing language explanations [\[17\]](#page-10-3). DriveVLM leverages a vision-language backbone for scene description and hierarchical planning, while its hybrid variant combines classical 3-D perception to mitigate spatialreasoning gaps [\[47\]](#page-11-2). DriveGPT4 goes further, training a GPT-4V-style model on driving-specific instructions so it can both output control signals and verbally justify its actions [\[56\]](#page-12-3). Despite these advances, challenges such as precise 3-D grounding and real-time efficiency persist: DriveVLM reports degraded performance on fine-grained spatial queries, and the DriveMLLM benchmark ![](_page_3_Figure_0.jpeg) Figure 3: To mitigate the challenges of vision encoder in adverse scenarios, we evaluate MLLM's ability for understanding and reseaoning under diverse vision prompts. Region, point, and text level vision prompts respectively target understanding and reasoning tasks for multi-objects, occluded objects, and referred entities. These prompts enable models to progressively comprehend intricate scene details for informed decision-making. Our empirical findings also indicate that MLLMs struggle with textual inputs of spatial coordinates (*e.g*., bounding boxes or points). Thus, we render corresponding visual cues directly onto the input images. highlights systematic errors in relative-distance estimation [\[21,](#page-10-13) [46,](#page-11-13) [24\]](#page-10-7). These limitations motivate our hierarchical chain-of-thought benchmark, which probes perception, reasoning, and decisionmaking under diverse, adverse scenarios. #### 2.2 MLLM Benchmarks for Autonomous Driving Language-augmented driving corpora extend canonical perception datasets—BDD100K [\[61\]](#page-12-4) and nuScenes [\[6\]](#page-9-10)—with textual or dialog annotations to test vision–language models. BDD-X [\[26\]](#page-10-6) provides behaviour rationales, Talk2Car [\[18\]](#page-10-14) grounds referring expressions in 3-D space, and GOHD [\[55\]](#page-12-5) offers hazard-focused explanations. NuScenes-QA [\[40\]](#page-11-3) and NuScenes-MQA [\[24\]](#page-10-7) scale QA to hundreds of thousands of pairs, whereas DriveLM [\[42\]](#page-11-1) supplies graph-structured dialogues spanning perception, prediction, planning, and control. Complementary efforts tackle commonsense (LingoQA [\[39\]](#page-11-14)), trust and robustness (AutoTrust [\[53\]](#page-12-6)), and rare hazards via synthetic StreetHazards [\[22\]](#page-10-5), real-world CODA [\[29\]](#page-10-4) and its multimodal extension CODA-LM [\[10\]](#page-9-2). DRAMA [\[38\]](#page-11-4) targets criticalobject advice, and DriveRef [\[51\]](#page-11-15) revisits referential grounding. Despite this breadth, existing benchmarks rarely isolate the fine-grained spatial reasoning essential for safe manoeuvring—an evaluative gap our work addresses. #### 3 AD2 -Bench We present AD2 -Bench, the first bilingual MLLM benchmark focusing on adverse weather and complex autonomous driving scenarios. Critically, it emphasizes detailed scene descriptions over concise VQA-style answers to elicit realistic driving suggestions. #### 3.1 Data Collection and Vision Prompt Existing benchmarks often lack sufficient real-world data for autonomous driving under adverse conditions. To address this, we curated images covering genuine rain, snow, fog, sandstorms, and nighttime scenarios from datasets like CODA [\[29\]](#page-10-4), ACDC [\[41\]](#page-11-16), DAWN [\[25\]](#page-10-15), and diverse web resources, avoiding reliance on synthetic data. We employed a rigorous filtering process involving selection based on existing labels where available, automated pre-filtering using advanced MLLMs (Gemini2.5-Pro), and final manual verification by experts to ensure relevance and quality. For more detailed definitions and examples, please refer to Appendix [A.2.](#page-13-0) Vision Prompt Strategy. To mitigate the visual challenges posed by adverse weather, we introduce three granular prompt levels in Fig. [3:](#page-3-0) *image-level* (no extra cues) for holistic reasoning, *region-level* bounding boxes for multi-object or region-specific queries, and an innovative *point-level* prompt that ![](_page_4_Figure_0.jpeg) Figure 4: Hierarchical Chain-of-Thought (CoT) framework in AD2 -Bench guides MLLMs step-bystep by decomposing reasoning into distinct, adaptable stages, typically including perception (basic & advanced), various analyses like key vehicle/pedestrian (relation understanding & reseaoning), and a final suggestion (decision). The vision prompt in Step 2 can be optioned, and and are the same image displayed in different steps. This structured decomposition serves two key purposes: it enhances the MLLM's ability to capture fine-grained scene details progressively, and critically, it facilitates reliable, atomic-level annotation and verification of each reasoning step on our end, ensuring high accuracy for crucial scene elements annotation. avoids box overlap in heavy occlusion. When finer reference is required, we additionally employ *text-level* prompts. Experiments show that point prompts deliver the best trade-off between annotation cost and accuracy under severe occlusion while preserving the lowest labour overhead. More details applied on reasoning task in Appendix [A.11](#page-19-0) and Fig. [9.](#page-25-0) Atomic Annotation. To enhance Chain-of-Thought (CoT) annotation accuracy, we introduce the Atomic annotation. This begins with an "atomic decomposition" of the VQA task into distinct subtasks in Fig. [4,](#page-4-0) each assigned to specialized experts for improved efficiency and focused expertise. For quality assurance, these initial atomic annotations are rigorously validated: *Gemini 2.5-Pro*, *GPT-4o*, and *Qwen 2.5-Max* score them against the visual input and provide detailed evaluations. Human experts then synthesize this multi-faceted feedback with their domain knowledge to refine it into high-fidelity *Intelligence Atoms*. While individual *Wisdom Atoms* ensure accuracy for intermediate CoT steps, expert specialization can challenge holistic VQA coherence. The Atomic annotation protocol mitigates this using an *Atomic Flow* strategy within its multi-round process. For each VQA instance, experts from diverse atomic specializations are dynamically grouped. This cross-pollination of expertise ensures logical consistency and narrative coherence across the sequence of *Wisdom Atoms* that constitute the full CoT. Format of *standard VQA* and *multiple-choice* in [A.1](#page-13-1) and [A.10.](#page-19-1) #### 3.2 Analysis of evaluation dimensions Basic Perception. Accurate perception is the cornerstone of the perception–understanding–reasoning pipeline in autonomous driving. AD2 -Bench therefore isolates a *basic-perception* tier that evaluates object presence, category recognition, localization/detection of vulnerable road users (pedestrians, cyclists, etc), and text-prompted *reference detection* for category-aware object retrieval. For every image in the benchmark, we annotate a corresponding basic-perception QA pair, and require cross-validation by at least three experts to mitigate the subjective noise introduced by adverse weather. Advanced Perception. The *advanced-perception* in Step 1 of Fig. [4](#page-4-0) measures fine-grained capabilities such as attribute recognition, dense captioning, OCR, lane-line detection, and vehicle motion recognition. Problems are crafted with real nuisance factors in mind: strong/weak illumination in *daytime* and *dawn*; heavy occlusion in snow, fog, or sand scenes; and optical scattering/reflection in rain and at night. Twenty experts independently supply answers, then peer-review one another. Initial manual VQA responses are post-processed with Gemini 2.5-Pro to improve fluency and generate multiple-choice counterparts. All annotations undergo a rigorous two-stage verification, including a cross-check and a final expert review, to guarantee quality and consistency for annotations. Relation Understanding. Beyond ego–object interactions, autonomous driving must grasp inter-object relations in Step 2, 3, 4 of Fig. [4.](#page-4-0) AD-Bench provides four relation types: *(1) Spatial*: location of a prompt-specified target relative to the ego car or a reference object; *(2) Social*: identity links such as parent–child or co-worker, enriched with fine-grained action cues (hand-holding, tool-passing, *etc*.); *(3) Causal*: how scene changes drive target/ego behaviour; *(4) Occlusion*: whether a prompt target is occluding or being occluded, and why. Each relation type is independently annotated by three experts and mutually cross-checked. Gemini 2.5-Pro then aggregates atomic annotations into VQA entries, which are audited by experts from other relation categories. This multi-round protocol sustains annotation fidelity, even under severe weather. Reasoning and Decision. Leveraging AD2 -Bench's real-world images from adverse conditions (*e.g*., rain, snow, fog), we designed challenging reasoning tasks integrated into complete perceptionto-reasoning *CoT* evaluation chains. From these, we curated a dedicated 5,046-image CoT test suite by refining existing perception and relation annotations into complex reasoning challenges covering diverse scenarios. These reasoning challenges, reflected in each step of Fig. [4](#page-4-0) and integrated with step-by-step perception and understanding, further enhance the capability for fine-grained scene reasoning. The questions cover: *(1) Weather reasoning*: how rainy or snowy slipperiness, and low visibility in dust storms or fog, affect the ego car; *(2) Night reasoning*: risks caused by darkness, glare, and multi-source light scattering; *(3) Region-level reasoning*: As illustrated in Fig. [3,](#page-3-0) this task requires the model to reason about areas enclosed by a region-level prompt—for example, zones depicting traffic accidents or construction sites; *(4) Occlusion reasoning*: employing point prompts to infer attributes, trajectories, and collision risk of partially visible objects; *(5) Cross-role reasoning*: inferring a specified target's intention and proposing counter-measures from that target's perspective; *(6) Holistic reasoning and decision*: synthesizing all scene elements from the ego car's viewpoint to provide global driving advice in Step 5 of Fig[.4.](#page-4-0) #### 3.3 Metrics for Assessing Chain-of-Thought Reasoning Quality Evaluating Chain-of-Thought (CoT) intermediate reasoning steps in autonomous driving requires new metrics, as current ones fall short. We propose novel metrics specifically for CoT sequence quality, employing an LLM evaluator (*e.g*., GPT-4o) and predefined criteria to score (1-10 scale) the logic, consistency, and semantic accuracy of each reasoning step, moving beyond traditional black-box assessments. Let the sequence of N input questions be Q = {Qi} N i=1, the corresponding model-generated CoT answer steps be A = {Ai} N i=1, and the ground truth answer steps be GT = {GTi} N i=1. Step-wise Accuracy & Completeness Score (SACS). Measures the factual accuracy and informational completeness of each individual reasoning step (Ai) compared to its corresponding ground truth (GTi). This assesses the model's ability to produce correct and relevant information at each stage. The score for each step is obtained via an LLM evaluation function, Evalacc(Ai , GTi), which returns a score from 1 to 10. The overall SACS is the average score across all steps: $$SACS = \frac{1}{N} \sum_{i=1}^{N} \text{Eval}_{acc}(A_i, GT_i).$$ (1) The MLLM assesses correctness against GTi , coverage of key information points present in GTi , and relevance to the question Qi Step-wise Logical Progression Score (SLPS). Evaluates the logical coherence and flow between *adjacent* steps (Ai and Ai+1) within the model's *own* generated sequence. This metric focuses on whether each step naturally and logically follows from the previous one, indicating sound reasoning progression. The score for each transition is Evalprog(Ai , Ai+1). The overall SLPS averages these transition scores: $$SLPS = \frac{1}{N-1} \sum_{i=1}^{N-1} \text{Eval}_{prog}(A_i, A_{i+1}).$$ (2) ![](_page_6_Figure_0.jpeg) visible signs of rain, snow, or fog... MiniCPM-o: The curr e n t natural conditions appear to be clear and sunny, as indicated by the bright lighting and Figure 5: Bad case: (a) In light fog conditions, the ambiguity caused by the non-dense fog leads many MLLMs to make erroneous judgments. (b) When key vehicles are distant and occluded by rain, most MLLMs fail to identify them accurately, sometimes even exhibiting object hallucinations. condition is foggy weather. Foggy weather has low visibility... The MLLM assesses if Ai+1 logically builds upon Ai , maintains context, uses information from Ai appropriately, and avoids abrupt, unexplained shifts in reasoning. > Overall Reasoning Coherence Score (ORCS). Assesses the holistic consistency, focus, and logical integrity of the *entire* reasoning chain (A = {A1, . . . , AN }) taken as a whole. This captures whether the sequence presents a unified and non-contradictory line of thought. This score is a single evaluation of the complete sequence: $$ORCS = Eval_{coh}(A_1, ..., A_N).$$ (3) The MLLM assesses the overall narrative flow, consistency across all steps, maintenance of focus on the core problem, and absence of internal contradictions or logical flaws throughout the entire sequence. Decision Justification Strength Score (DJSS). Measures how well the final step or decision (AN ) is supported and justified by the preceding reasoning steps (A1, . . . , AN−1). This is crucial for understanding if the model's conclusion is a logical outcome of its analysis. This score evaluates the relationship between the reasoning process and the conclusion: $$DJSS = \text{Eval}_{\text{just}}((A_1, \dots, A_{N-1}), A_N). \tag{4}$$ The MLLM assesses whether AN is a direct and logical consequence of the arguments, evidence, or analysis presented in steps A1 through AN−1. It checks if the preceding steps provide sufficient grounds for the final answer. Together, these four metrics provide a multi-faceted framework for evaluating the internal quality of CoT reasoning in autonomous driving models, complementing traditional task-based evaluations by offering insights into the underlying thought process. # 4 Experiments We conducted extensive experiments with detailed analyses and discussions in AD2 -Bench. #### 4.1 Experimental Setups We evaluated 12 current mainstream open-source Multimodal Large Language Models on AD2 - Bench. These mainly include the InternVL series (InternVL2, 2.5, 3) [\[62,](#page-12-7) [13,](#page-9-11) [14\]](#page-9-12), the Qwen series (Qwen2-VL [\[48\]](#page-11-10), Qwen2.5-VL [\[4\]](#page-9-9)), the MiniCPM (V2.5, V2.6, o2.6) series [\[60\]](#page-12-8), and the LLaVA series (1.5, Next, onevision) [\[33,](#page-10-1) [27,](#page-10-16) [34\]](#page-11-9), as well as Janus-Pro [\[11\]](#page-9-13) which is based on DeepSeek. The compact parameter scale endows the models with the potential for on-device deployment in the field of autonomous driving More details and perception metrics in Appendix [A.4](#page-14-0) and [A.5.](#page-14-1) # 4.2 Observations & Discussions Base Perception. As shown in Tab. [2](#page-7-0) and [5,](#page-16-0) in adverse weather and complex scenes, the accuracy of perception of existence of objects (Existence) generally falls below 60% for most models. We believe that the main reasons for this phenomenon are as follows: (1) the limited visual conditions; (2) the inherent long-tail distribution of objects in driving scenes (where humans and cars are predominant); (3) the querying strategy we adopted that often targets less frequent object categories, causing Table 2: Overall Model Performance Comparison (Sorted by Average with CoT) | Models | LLM | | | | | Base Perc. Adv. Perc. Relation Und. Event Res. Avg-S (w/ CoT) | |--------------------------|----------------|-------|-------|-------|-------|---------------------------------------------------------------| | InternVL3-8B | Qwen2.5-7B | 41.08 | 70.52 | 54.66 | 63.04 | 57.33 | | Qwen2.5-VL | Qwen2.5-7B | 59.29 | 62.53 | 45.90 | 59.33 | 56.76 | | InternVL2.5 | InternLM2.5-7B | 36.69 | 68.30 | 53.71 | 61.64 | 55.09 | | MiniCPM-o-2.6 | Qwen2.5-7B | 40.49 | 67.93 | 52.81 | 58.19 | 54.86 | | InternVL2 | InternLM2.5-7B | 40.23 | 62.03 | 46.99 | 60.57 | 52.46 | | MiniCPM-V-2.6 | Qwen2-7B | 40.58 | 61.67 | 47.72 | 55.27 | 51.31 | | Qwen2-VL | Qwen2-7B | 40.75 | 52.90 | 51.57 | 50.45 | 48.92 | | Janus-Pro | DeepSeek-7B | 34.31 | 49.29 | 51.19 | 58.84 | 48.41 | | LLaVA-OneVision Qwen2-7B | | 21.85 | 51.80 | 55.06 | 57.29 | 46.50 | | MiniCpm-V-2.5 | Llama3-8B | 35.83 | 52.90 | 39.30 | 52.50 | 45.13 | | LLaVA-1.5 | Vicuna-7B | 30.08 | 39.90 | 54.72 | 43.87 | 42.14 | | LLaVA-NeXT | Vicuna-7B | 28.76 | 40.92 | 51.09 | 47.22 | 42.00 | models to overlook them. Grounding tasks, such as detection and counting, reveal stark performance disparities: Qwen2.5-VL significantly excels, whereas MiniCPM and LLaVA series models exhibit minimal grounding capabilities. Notably, LLaVA models often erroneously default to outputting '10' for counts, indicating failures in quantitative reasoning. While performance correlates partly with training data composition, input processing strategies are also critical. Many models output proportional coordinates; however, common image patching or sampling techniques for input images prevent the reliable reconstruction of absolute bounding boxes from these outputs. This highlights that enhancing basic perception necessitates a dual approach: incorporating targeted training data and adopting input processing methods like dynamic resolution (used by Qwen2.5-VL) that preserve holistic image information and scale context. Advanced Perception. In our Optical Character Recognition (OCR) evaluation for advanced perception (see Appendix [A.5](#page-14-1) for detailed metrics), the InternVL series achieved the best results. Qwen and MiniCPM series demonstrated good proficiency. In contrast, several other models (*e.g*., LLaVA variants) underperformed due to inadequate visual feature extraction and poor instruction adherence; their excessively long outputs were truncated for fair evaluation. We conclude that high-quality OCR necessitates targeted optimization and precise generation capabilities beyond general pre-training. For overall advanced perception performance (detailed per-model scores are provided in Appendix [A.7\)](#page-16-1). Relation Understanding. Fine-grained relational understanding is pivotal for MLLM scene comprehension, bridging perception and reasoning. Our evaluation results (Tab. [7\)](#page-17-0) show that LLaVA series models generally excel in this dimension, particularly those using the Vicuna LLM backbone (LLaVA-1.5, NeXT). Notably, these Vicuna-based models [\[45\]](#page-11-17) substantially outperform newer architectures (InternVL3, Qwen2.5-VL) when reasoning about occlusion, maintaining around 50% accuracy. In contrast, LLaVA-OneVision (using the Qwen LLM [\[58\]](#page-12-9)) struggles significantly with occlusion tasks compared to its Vicuna-based counterparts. We found this discrepancy stems from differing model behaviors, possibly linked to pre-training: Vicuna's potential exposure to "completion"-style tasks might enhance speculation under visual uncertainty (beneficial for occlusion), while Qwen appears more logically conservative, avoiding guesses about poorly discerned objects and often defaulting to generic 'not found' answers (*e.g*., options 'E' or 'F'). Overall, despite advances, MLLM proficiency in complex relational understanding, especially under occlusion, requires further improvement. Visual Scene Reasoning and Decision Analysis. Direct reasoning without CoT yields suboptimal performance for most evaluated LMMs, with scores typically below 40% (Tab. [2\)](#page-7-0); even powerful models like Qwen2.5-VL struggle in this setting. Notably, models employing input image patching strategies (MiniCPM) tend to outperform those using dynamic resolution (Qwen2.5-VL) for these reasoning tasks. While dynamic resolution excels elsewhere (*e.g*., detection), patches resized appears to better preserve fine visual details crucial for nuanced reasoning in adverse/complex scenes (Fig. [1\)](#page-1-0), albeit at a higher computational cost. Intriguingly, we observed a counter-intuitive trend within the InternVL series, where newer iterations underperformed older ones on ADBench. We found two primary factors: 1) Compared to InternVL2, InternVL3 generated more diffuse and less coherent reasoning outputs when dealing with multifaceted autonomous driving scenarios. 2) The relative proportion of domain-specific autonomous driving data likely decreased within their progressively larger general training datasets, potentially diluting specialized reasoning capabilities. | Model | w/o CoT | w/ CoT | Avg. Scores (CoT Steps) | | | | | |-----------------|---------|--------|-------------------------|-------|-------|-------|--| | | | | SACS | SLPS | ORCS | DJSS | | | InternVL3 | 42.21 | 63.04 | 63.42 | 67.23 | 61.00 | 60.53 | | | InternVL2.5 | 42.92 | 61.64 | 61.80 | 63.27 | 57.37 | 64.14 | | | InternVL2 | 43.77 | 60.57 | 60.30 | 62.71 | 56.25 | 63.02 | | | Qwen2.5-VL | 41.73 | 59.33 | 59.24 | 61.30 | 55.03 | 61.75 | | | Janus-Pro | 39.27 | 58.84 | 59.11 | 61.35 | 54.63 | 60.28 | | | MiniCPM-o-2.6 | 43.97 | 58.19 | 58.38 | 59.90 | 53.66 | 60.84 | | | LLaVA-OneVision | 43.83 | 57.29 | 57.24 | 60.44 | 53.60 | 57.90 | | | MiniCPM-V-2.6 | 42.13 | 55.27 | 55.62 | 58.35 | 51.74 | 55.36 | | | MiniCPM-V-2.5 | 40.25 | 52.49 | 52.41 | 56.21 | 48.72 | 52.64 | | | Qwen2-VL | 39.17 | 50.45 | 50.91 | 52.65 | 46.84 | 51.37 | | | LLaVA-NeXT | 34.22 | 47.22 | 47.06 | 49.62 | 43.34 | 48.87 | | | LLaVA-1.5 | 32.02 | 43.87 | 43.64 | 45.83 | 39.81 | 46.22 | | Empowerment Effect of Hierarchical CoT. Introducing a hierarchical CoT framework (Fig. [4\)](#page-4-0) substantially enhances performance for most models (Tab. [3\)](#page-8-0). Notably, InternVL2.5/3 and Qwen2.5- VL saw large score increases (*e.g*., +20.2 for InternVL3), with InternVL3 becoming the top performer post-CoT, likely due to improved reasoning coherence and focus. In contrast, MiniCPM-o-2.6, the strongest model without CoT, showed comparatively modest gains. We attribute this difference primarily to training methodology and prompt adherence: models like InternVL3 and Qwen2.5-VL, likely exposed to rationale-augmented data during instruction fine-tuning (SFT), adapt better to CoT prompting. Conversely, the edge-optimized MiniCPM series appears to exhibit weaker adherence to complex, extended CoT prompts, limiting its gains despite potential input strategy advantages. Granular Analysis of CoT-Driven Reasoning Steps. Further analysis dissects CoT performance by evaluating intermediate reasoning steps across correctness, logical consistency, coherence, and decision support (Tab. [3\)](#page-8-0). These intermediate metrics correlate positively with final post-CoT scores. A key observation is that average correctness during the CoT process often exceeds the ORCS score (*e.g*., for Qwen2.5-VL: 59.3 *vs.* 55.0). This suggests MLLMs, trained predominantly on question-answer pairs lacking intermediate supervision, often excel at approximating final answers but struggle to maintain consistent logical coherence throughout complex reasoning chains. #### 4.3 Bad Case Analysis and Findings Qualitative Error Analysis. Qualitative analysis in Fig. [5](#page-6-0) reveals significant MLLM failure modes. Models sometimes misinterpret subtle weather conditions (*e.g*., light fog, Fig. [5\(](#page-6-0)a)) easily perceived by humans. More critically, under adverse conditions like rain (Fig. [5\(](#page-6-0)b)), models frequently fail to detect crucial safety hazards, such as an unexpected U-turning vehicle directly ahead, while sometimes exhibiting dangerous object hallucinations (*e.g*., MiniCPM-o-2.6 perceiving a non-existent 'tram'). These errors highlight the urgent need for improved fine-grained perception and robust grounding in visual input, strictly avoiding hallucinations, for safe autonomous driving deployment. More analysis in Appendix [A.11.](#page-19-0) Instruction Following Ability. We observed limitations in models' adherence to specific formatting instructions. Several MLLMs generated extraneous analysis beyond the required single-letter answers for multiple-choice questions, produced excessively verbose outputs often exceeding token limits (particularly for reasoning VQA), and failed to consistently apply the required structural tags (*e.g*., '...') for CoT step evaluation. This indicates substantial room for improvement in the instruction following capabilities of current MLLMs, especially regarding structured output formats. # 5 Conclusion In this work, we introduced AD2 -Bench to address critical evaluation gaps for Multimodal Large Models (MLLMs) in autonomous driving, specifically the scarcity of benchmarks for Chain-of-Thought (CoT) reasoning and for comprehensive real-world adverse weather and complex scenarios. AD2 -Bench fills these voids, providing 10k real-world images from such demanding conditions. As the first CoT-focused benchmark in this domain, its core contribution includes over 5,400 meticulously, manually annotated CoT instances where each intermediate reasoning step forms an atomic ground truth, enabling deep analysis of inferential integrity. Unique multi-level visual prompts further aid in evaluating reasoning capabilities beyond perceptual limitations, complemented by a novel multidimensional CoT evaluation metric. AD2 -Bench significantly challenges current state-of-the-art models (<60% accuracy), underscoring its instrumental role in guiding the development of more robust, interpretable, and truly end-to-end autonomous driving systems. # References - [1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. *arXiv preprint arXiv:2404.14219*, 2024. - [2] Felix Assion, Florens Gressner, Nitin Augustine, Jona Klemenc, Ahmed Hammam, Alexandre Krattinger, Holger Trittenbach, Anja Philippsen, and Sascha Riemer. A-bdd: Leveraging data augmentations for safe autonomous driving in adverse weather and lighting. *arXiv preprint arXiv:2408.06071*, 2024. - [3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. *arXiv preprint arXiv:2308.12966*, 2023. - [4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. *arXiv preprint arXiv:2502.13923*, 2025. - [5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. *arXiv preprint arXiv:2307.15818*, 2023. - [6] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 11621–11631, 2020. - [7] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 8748–8757, 2019. - [8] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 14455–14465, 2024. - [9] Junliang Chen, Huaiyuan Xu, Yi Wang, and Lap-Pui Chau. Occprophet: Pushing efficiency frontier of camera-only 4d occupancy forecasting with observer-forecaster-refiner framework. *arXiv preprint arXiv:2502.15180*, 2025. - [10] Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision-language models on self-driving corner cases. In *2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pages 7817–7826. IEEE, 2025. - [11] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. *arXiv preprint arXiv:2501.17811*, 2025. - [12] Yuntao Chen, Yuqi Wang, and Zhaoxiang Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. *arXiv preprint arXiv:2412.18607*, 2024. - [13] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. *arXiv preprint arXiv:2412.05271*, 2024. - [14] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 24185–24198, 2024. - [15] Qing Cheng, Niclas Zeller, and Daniel Cremers. Vision-based large-scale 3d semantic mapping for autonomous driving applications. In *2022 International Conference on Robotics and Automation (ICRA)*, pages 9235–9242. IEEE, 2022. - [16] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 3213–3223, 2016. - [17] Yaodong Cui, Shucheng Huang, Jiaming Zhong, Zhenan Liu, Yutong Wang, Chen Sun, Bai Li, Xiao Wang, and Amir Khajepour. Drivellm: Charting the path toward full autonomous driving with large language models. *IEEE Transactions on Intelligent Vehicles*, 9(1):1450–1464, 2024. - [18] Thierry Deruyttere, Dusan Grujicic, Matthew B Blaschko, and Marie-Francine Moens. Talk2car: Predicting physical trajectories for natural language commands. *Ieee Access*, 10:123809–123834, 2022. - [19] Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. *arXiv preprint arXiv:2411.14432*, 2024. - [20] Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human: Rethinking autonomous driving with large language models. In *IEEE/CVF Winter Conference on Applications of Computer Vision*, pages 910–919, 2024. - [21] Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Chenming Zhang, Shuai Liu, and Long Chen. Drivemllm: A benchmark for spatial understanding with multimodal large language models in autonomous driving. *arXiv preprint arXiv:2411.13112*, 2024. - [22] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. A benchmark for anomaly segmentation. *arXiv preprint arXiv:1911.11132*, 2(7), 2019. - [23] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 14281–14290, 2024. - [24] Yuichi Inoue, Yuki Yada, Kotaro Tanahashi, and Yu Yamaguchi. Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pages 930–938, 2024. - [25] Mourad A Kenk and Mahmoud Hassaballah. Dawn: vehicle detection in adverse weather nature dataset. *arXiv preprint arXiv:2008.05402*, 2020. - [26] Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. In *ECCV*, 2018. - [27] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. *arXiv preprint arXiv:2407.07895*, 2024. - [28] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. In *International Conference on Machine Learning (ICML)*, 2023. - [29] Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. In *European Conference on Computer Vision*, pages 406–423. Springer, 2022. - [30] Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 9087–9098, 2023. - [31] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 45(3):3292– 3310, 2022. - [32] Benlin Liu, Yuhao Dong, Yiqin Wang, Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondences elicit 3d spacetime understanding in multimodal language model. *arXiv preprint arXiv:2408.00754*, 2024. - [33] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 26296–26306, 2024. - [34] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. - [35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In *Advances in Neural Information Processing Systems*, pages 34892–34916, 2023. - [36] Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. *arXiv preprint arXiv:2403.12966*, 2024. - [37] Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. *arXiv preprint arXiv:2312.00438*, 2023. - [38] Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In *WACV*, 2023. - [39] Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driving. In *European Conference on Computer Vision*, pages 252–269. Springer, 2024. - [40] Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 4542–4550, 2024. - [41] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 10765–10775, 2021. - [42] Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In *European Conference on Computer Vision*, pages 256–274, 2024. - [43] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world object manipulation using pre-trained vision-language models. *arXiv preprint arXiv:2303.00905*, 2023. - [44] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 2446–2454, 2020. - [45] The Vicuna Team. Vicuna: An open-source chatbot impressing gpt-4 with 90%\* chatgpt quality. [https:](https://lmsys.org/blog/2023-03-30-vicuna/) [//lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/). Accessed: 2023. - [46] Kexin Tian, Jingrui Mao, Yunlong Zhang, Jiwan Jiang, Yang Zhou, and Zhengzhong Tu. Nuscenesspatialqa: A spatial understanding and reasoning benchmark for vision-language models in autonomous driving. *arXiv preprint arXiv:2504.03164*, 2025. - [47] Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. *arXiv preprint arXiv:2402.12289*, 2024. - [48] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. *arXiv preprint arXiv:2409.12191*, 2024. - [49] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. *Advances in neural information processing systems*, 35:24824–24837, 2022. - [50] Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to autonomous driving with large language models. In *International Conference on Learning Representations*, 2024. - [51] Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. *arXiv preprint arXiv:2309.04379*, 2023. - [52] Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives. *arXiv preprint arXiv:2501.04003*, 2025. - [53] Shuo Xing, Hongyuan Hua, Xiangbo Gao, Shenzhe Zhu, Renjie Li, Kexin Tian, Xiaopeng Li, Heng Huang, Tianbao Yang, Zhangyang Wang, et al. Autotrust: Benchmarking trustworthiness in large vision language models for autonomous driving. *arXiv preprint arXiv:2412.15206*, 2024. - [54] Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. *arXiv preprint arXiv:2412.14446*, 2024. - [55] Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable object-induced action decision for autonomous vehicles. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9523–9532, 2020. - [56] Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K. Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. *arXiv preprint arXiv:2310.01412*, 2023. - [57] Shaotian Yan, Chen Shen, Wenxiao Wang, Liang Xie, Junjie Liu, and Jieping Ye. Don't take things out of context: Attention intervention for enhancing chain-of-thought reasoning in large language models. *arXiv preprint arXiv:2503.11154*, 2025. - [58] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. *arXiv preprint arXiv:2412.15115*, 2024. - [59] Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. In *European Conference on Computer Vision*, pages 20–38. Springer, 2025. - [60] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. *arXiv preprint arXiv:2408.01800*, 2024. - [61] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 2636–2645, 2020. - [62] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. *arXiv preprint arXiv:2504.10479*, 2025. # A Appendix #### A.1 Annotation Formats. AD-Bench is released in both Chinese and English and offers two complementary formats—*standard VQA* and *multiple-choice (MC)*—within the same annotation layer. Objective perception queries with a single ground-truth answer are expressed in MC form, whereas relation-understanding and decision-making tasks that require descriptive responses remain in VQA form; the benchmark, however, contains both formats for every item, and the evaluator automatically adopts the format best suited to each task. Every MC question provides four carefully designed options: one correct answer and three hard distractors drawn from text appearing in the image or phrased to closely resemble the ground truth. Two additional choices, E and F, allow the model to abstain—E when none of the options match its model's answer, and F when the relevant target is absent from the image. Tab. [4](#page-13-2) gives a side-by-side example of a basic QA pair and its MC counterpart; during testing we prefer a model's native prompt style in object localization or detection tasks, defaulting to the template in Fig. [3](#page-3-0) if none is specified. The full chain-of-thought data-flow, including format conversion, is illustrated in Fig. [4.](#page-4-0) #### Table 4: Prompt Settings of AD2 -Bench #### A. Bounding Box Identification [Image] [Question] This task aims to identify the bounding box coordinates [min\_x, min\_y, max\_x, max\_y] of a specific object in an image, where (min\_x, min\_y) is the top-left corner and (max\_x, max\_y) is the bottom-right corner of the rectangle, with the output format specified as [min\_x, min\_y, max\_x, max\_y]. The answer is: #### B. Multiple-Choice Question based on Image [Image] [Question] Here are the options below: - (A) [Choice A] - (B) [Choice B] - (C) [Choice C] - (D) [Choice D] - (E) [Choice E] - (F) [Choice F] Select the best answer to the above multiple-choice question based on the image. Please reply with the letter (A, B, C, D, E or F) corresponding to the correct option. The best answer is: # A.2 Data Collection and Scene Partition Most public autonomous-driving datasets (e.g., KITTI [\[31\]](#page-10-17), nuScenes [\[6\]](#page-9-10), Waymo [\[44\]](#page-11-18), Argoverse [\[7\]](#page-9-14)) were captured in clear weather, while synthetic sets such as A-BDD [\[2\]](#page-9-3) or Cityscapes Foggy/Rainy [\[16\]](#page-10-8) still deviate from real environments. To fully cover genuine adverse conditions, we curated images under rain, snow, fog, sand-storm, and nighttime from CODA [\[29\]](#page-10-4), ACDC [\[41\]](#page-11-16), DAWN [\[25\]](#page-10-15), BDD100K [\[61\]](#page-12-4), nuScenes [\[6\]](#page-9-10) and additional web resources. For datasets with explicit weather labels, we first isolated adverse-weather samples and then performed expert verification; where labels were absent, we employed the proprietary multi-modal model Gemini2.5-Pro for automatic filtering followed by manual review. The resulting images were grouped by weather and illumination into seven classes—*overcast*, *dawn*, *rainy*, *snowy*, *fog*, *night*, and *sand*. *Daytime* and *dawn* scenes focus on dense objects or complex road layouts, whereas the remaining classes emphasise natural factors that perturb perception and reasoning. Each weather type is further divided into *light*, *moderate*, and *severe* levels, with cross-class expert validation to ensure consistent labelling. ### A.3 Annotation Pipeline and Prompt Strategy All annotations, from pre-processing to final audit, were performed manually in four batches, each involving five domain experts and overseen by an independent lead annotator. During pre-annotation, images were grouped by coarse scene categories; cluster analysis then refined the four evaluation axes into 33 sub-tasks, and primary objects in every image were roughly boxed to reduce bias. Task design proceeds from three perspectives—*visual perception*(base and advanced), *relation understanding*, and *decision reasoning*. An initial pool of four question demensions was subdivided into 33 subcategories after preprocess. Each record Pi stores a six-tuple (Qi , Ci , Ii , Li , Bi , Ai) capturing the question text, sub-task index (Ci ∈ [1, 33]), image ID, prompt type Li (text, box, or point), prompt coordinates Bi , and ground-truth answer Ai . To mitigate the visual challenges posed by adverse weather, we introduce three granular prompt levels in Fig. [3:](#page-3-0) *image-level* (no extra cues) for holistic reasoning, *region-level* bounding boxes for multi-object or area-specific queries, and an innovative *point-level* prompt that avoids box overlap in heavy occlusion. When finer reference is required, we additionally employ *text-level* prompts. Experiments show that point prompts deliver the best trade-off between annotation cost and accuracy under severe occlusion while preserving the lowest labour overhead. The aforementioned vision prompts establish a multi-level and diverse methodology for evaluating MLLMs in adverse scenarios. Regarding input formats, our benchmark allows MLLMs to perform inference directly on entire images or on images combined with various prompts; the latter approach aids model perception while enabling a deeper probe of its understanding and reasoning capabilities. In terms of inference style, it supports not only direct outputs from image inputs but also step-by-step modes like Chain-of-Thought (CoT), facilitating the assessment of the model's thinking process at each stage. These comprehensive features render our benchmark highly suitable for multifaceted and multi-dimensional MLLM evaluation. ### A.4 Experimental Setups in Details In this work, we evaluated 12 current mainstream open-source models on ADBench. These mainly include the InternVL series (InternVL2, 2.5, 3) [\[62,](#page-12-7) [13,](#page-9-11) [14\]](#page-9-12), the Qwen series (Qwen2-VL [\[48\]](#page-11-10), Qwen2.5-VL [\[4\]](#page-9-9)), the MiniCPM (V2.5, V2.6, o2.6) series [\[60\]](#page-12-8), and the LLaVA series (1.5, Next, onevision) [\[33,](#page-10-1) [27,](#page-10-16) [34\]](#page-11-9), as well as Janus-Pro [\[11\]](#page-9-13) which is based on DeepSeek. The parameter count of these models is around 7/8B, which gives them the potential for on-device deployment in autonomous driving. Considering computational resource limitations and the possibility of deploying models with higher parameter counts in autonomous driving terminals, we have not yet evaluated models with larger parameter counts. The entire evaluation relies on the MS-SWIFT open-source framework. In the inference stage, GPT-4o is used as an evaluator for the similarity between the predicted output and the ground truth. We set the temperature to 0 and fixed the random seed to ensure accuracy when evaluating different models. Our experiments are conducted on 8×RTX4090 GPUs. Instruction-following issues are prevalent in many MLLMs, particularly for object localization tasks in perception stages and Optical Character Recognition (OCR) tasks in advanced perception. For instance, in object localization, where models are instructed to output a point in the format < x, y >, some models, such as those from the MiniCPM series, instead provide a bounding box (e.g., formatted as < x1, y1, x2, y2 > or similar, representing two diagonal points or other box parameters). In such cases, for metric calculation, we utilize the center point of the provided bounding box as the definitive localization output. Similarly, in OCR tasks, which primarily involve recognizing license plates and traffic signs, models like the LLaVA series tend to generate overly verbose responses. Our evaluation protocol addresses this by considering output length: answers exceeding 50 words are automatically classified as incorrect, and their accuracy score is consequently set to zero. #### A.5 Perception Metrics The accuracy metrics quantify the correctness of the model's predictions for different types of tasks. The indi vidual accuracy for each sample is defined based on the task type as follows: Binary Classification Accuracy. For binary classification tasks, such as Left/Right existance judgment and choice judgment, the individual accuracy acci is defined as: $$acc_i = \begin{cases} 1, & \text{if } a_i = y_i \\ 0, & \text{if } a_i \neq y_i \end{cases}$$ (5) where, ai is the model's predicted answer for sample i, yi is the ground truth answer for sample i. Object Localization Accuracy For the pixel object localization task, the individual accuracy acci is based on the Euclidean distance error between the predicted and ground truth coordinates: $$acc_i = \frac{1}{1 + \alpha_p \|\mathbf{x}_i - \mathbf{x}_i^{\mathsf{gt}}\|_2} \tag{6}$$ where, xi = (xi , yi) are the model's predicted coordinates for sample i, x gt i = (x gt i , y gt i ) are the ground truth coordinates for sample i, αp is a scaling factor, set to αp = 0.005. Object Detection Accuracy. For the object detection task, accuracy is measured using the Intersection over Union (IoU) between the predicted and ground truth bounding boxes: $$acc_i = IoU(B_i, B_i^{gt}) = \frac{|B_i \cap B_i^{gt}|}{|B_i \cup B_i^{gt}|}$$ $$(7)$$ where, Bi is the model's predicted bounding box for sample i, B gt i is the ground truth bounding box for sample i, | · | denotes the area of the bounding box. Character Error Rate (CER) for OCR. For evaluating the transcriptional accuracy of Optical Character Recognition (OCR) systems, the Character Error Rate (CER) quantifies errors at the character level based on the Levenshtein distance: $$CER = \frac{S + D + I}{N} \tag{8}$$ where, S is the number of substitutions, D is the number of deletions, I is the number of insertions required to change the predicted string to the ground truth string, and N is the total number of characters in the ground truth string. A lower CER indicates better performance. Character-Level F1-score for OCR. The F1-score is utilized to assess OCR performance by computing the harmonic mean of character-level Precision (P) and Recall (R), which are defined as: $$P = \frac{H}{M}, \quad R = \frac{H}{N} \tag{9}$$ The F1-score is then calculated by: $$F_1 = 2 \cdot \frac{P \cdot R}{P + R} \tag{10}$$ where, H is the number of correctly recognized characters (hits) common to both the predicted string and the ground truth string, M is the total number of characters in the predicted string, and N is the total number of characters in the ground truth string. A higher F1-score (closer to 1) indicates better performance. #### A.6 Details of Base Perception The performance comparision of basic perception capability among different models is reported in Tab. [5.](#page-16-0) The experimental results reveal the limitations of the basic perception capabilities of current mainstream multimodal large models in adverse weather conditions and complex scenes. As shown in Tab. [5,](#page-16-0) in adverse weather and complex scenes, the accuracy of perception of existence of objects (Existence) generally falls below 60% for most models. We believe that the main reasons for this phenomenon are as follows: (1) the limited visual conditions; (2) the inherent long-tail distribution of objects in driving scenes (where humans and cars are predominant); (3) the querying strategy we adopted that often targets less frequent object categories, causing models to overlook them. This difficulty likely arises from challenging visual conditions combined with the inherent long-tail distribution of objects in driving scenes (where humans and cars are predominant) and our querying strategy that often targets less frequent object categories, causing models to overlook them. In addition, performance disparities are particularly stark in detection and counting. Qwen2.5 significantly exceeds other models. The MiniCPM and LLaVA series exhibit minimal grounding capabilities. Notably, LLaVA models frequently default to outputting the number '10' for counting tasks, irrespective of the actual object count, indicating potential systemic biases or failures in quantitative reasoning. These performance variations correlate partly with training data; for example, Qwen2.5-VL's improved detection over Qwen2-VL likely stems from incorporating more relevant grounding data. Furthermore, analyzing model outputs reveals a tendency for many MLLMs to provide results using proportional coordinates. However, common input processing methods like image patching or sampling make it infeasible to reconstruct accurate, absolute bounding boxes from these relative outputs using the original image dimensions. This suggests that enhancing basic perception requires a two-pronged approach: incorporating targeted training data and employing input strategies, such as the dynamic resolution used by Qwen2.5-VL, that preserve the integrity and scale of the full image context. Table 5: Basic Perception | Models | LLM | Existence | Counting | Location | Detection | Avg-S | |-----------------|----------------|-----------|----------|----------|-----------|-------| | Qwen2.5-VL | Qwen2.5-7B | 58.70 | 59.4 | 68.10 | 50.95 | 59.29 | | InternVL3 | Qwen2.5-7B | 59.58 | 57.1 | 41.35 | 6.28 | 41.08 | | Qwen2-VL | Qwen2-7B | 58.06 | 63.3 | 37.16 | 4.46 | 40.75 | | MiniCPM-V-2.6 | Qwen2-7B | 55.49 | 56.2 | 42.91 | 7.70 | 40.58 | | MiniCPM-o-2.6 | Qwen2.5-7B | 63.07 | 65.6 | 27.70 | 5.59 | 40.49 | | InternVL2 | InternLM2.5-7B | 61.21 | 55.5 | 38.03 | 6.16 | 40.23 | | InternVL2.5 | InternLM2.5-7B | 62.13 | 54.0 | 25.11 | 5.51 | 36.69 | | MiniCPM-V-2.5 | Llama3-8B | 59.52 | 53.6 | 24.51 | 5.67 | 35.83 | | Janus-Pro | DeepSeek-7B | 56.59 | 58.6 | 21.92 | 0.11 | 34.31 | | LLaVA-1.5 | Vicuna-7B | 50.76 | 50.5 | 19.07 | 0.00 | 30.08 | | LLaVA-NeXT | Vicuna-7B | 52.76 | 43.2 | 19.07 | 0.00 | 28.76 | | LLaVA-OneVision | Qwen2-7B | 43.70 | 22.9 | 20.70 | 0.11 | 21.85 | #### A.7 Details of Advanced Perception In the Optical Character Recognition (OCR) component of advanced perception (results summarized in Tab. [6\)](#page-17-1), we employ the F1 score (measuring the detection of target characters) and Character Error Rate (CER, focusing more on single-character accuracy; details in [A.5\)](#page-14-1) as evaluation metrics. The InternVL series models perform best in this section, demonstrating excellent text recognition capabilities. The Qwen and MiniCPM series models also recognize most text well, with only minor character-level deviations in some cases. The LLaVA-OneVision model can identify text in most scenes but suffers from some misidentifications and content omissions. In contrast, the recognition results from LLaVA-1.5, LLaVA-NeXT, and Janus-Pro are often unsatisfactory, frequently outputting irrelevant and meaningless sequences of numbers or letters. This reflects significant deficiencies in their visual feature extraction and semantic translation capabilities for text; they also exhibit poor instruction following, occasionally generating abnormal outputs far exceeding the typical length of OCR ground truth (usually <100 characters). To ensure fair evaluation, for single recognition results exceeding 100 characters, we only consider the first 100 characters for metric calculation. We posit that achieving high-quality and reliable OCR capabilities in MLLMs does not simply stem from the generalization effects of large-scale pre-training, but rather relies heavily on targeted training data optimization (e.g., images with diverse scene text and precise annotations), effective representation and extraction of fine-grained visual features by the model, and the integrated ability to perform precise, controllable text generation under task-specific instructions. Summarizing the overall performance in advanced perception (as shown in Tab. [6\)](#page-17-1), the InternVL3-8B model is indeed exceptionally prominent. It not only achieves an F1 score as high as 89.7% on OCR tasks but also reaches accuracies of 57.93% and 56.37% on the more challenging tasks of object attribute judgment and action/position reasoning, respectively. In comparison, while the Qwen series models perform excellently in basic perception tasks, their performance in advanced perception tasks exhibits considerable fluctuation. For example, although the Qwen-2.5-VL model scores well on several perception problems, it performs relatively weakly in judging target action/position. The LLaVA series and Janus-Pro models generally underperform in advanced perception tasks, clearly indicating their deficiencies in capturing subtle object attributes and dynamic positional changes in complex scenes. Table 6: Model Performance Evaluation on Advanced Perception | Models | LLM | | OCR-F1 OCR-SER | Attr. | Pos. | Cap. & Others Avg-S | | |--------------------------|----------------|------|----------------|-------|-------------|---------------------|-------| | InternVL3 | Qwen2.5-7B | 89.7 | 15.1 | | 57.93 56.37 | 78.09 | 70.52 | | InternVL2.5 | InternLM2.5-7B | 89.7 | 15.0 | | 59.89 45.18 | 78.41 | 68.30 | | MiniCPM-o-2.6 | Qwen2.5-7B | 84.2 | 27.3 | | 56.75 51.88 | 78.88 | 67.93 | | Qwen2.5-VL | Qwen2.5-7B | 86.5 | 15.4 | | 54.23 33.74 | 75.65 | 62.53 | | InternVL2 | InternLM2.5-7B | 83.3 | 35.3 | | 59.05 38.24 | 67.54 | 62.03 | | MiniCPM-v-2.6 | Qwen2-7B | 78.5 | 27.5 | | 53.61 40.36 | 74.19 | 61.67 | | MiniCPM-V-2.5 | Llama3-8B | 67.3 | 41.0 | | 50.25 22.47 | 71.59 | 52.90 | | Qwen2-VL | Qwen2-7B | 59.0 | 49.1 | | 49.30 30.56 | 72.72 | 52.90 | | LLaVA-OneVision Qwen2-7B | | 56.5 | 65.7 | | 49.69 29.08 | 71.93 | 51.80 | | Janus-Pro | DeepSeek-7B | 22.0 | 143.6 | | 54.01 46.00 | 75.16 | 49.29 | | LLaVA-NeXT | Vicuna-7B | 32.8 | 125.5 | | 37.54 25.98 | 67.34 | 40.92 | | LLaVA-1.5 | Vicuna-7B | 24.1 | 173.1 | | 37.20 36.11 | 62.20 | 39.90 | ### A.8 Details of Relation Understanding Serving as a crucial link between perception and reasoning, the ability to comprehend fine-grained relationships between objects is pivotal for assessing a Multi-Modal Large Model's (MLLM's) scene understanding capabilities. Tab. [7](#page-17-0) presents the evaluation results for the twelve advanced models benchmarked on the relational understanding dimension. Our findings indicate that the LLaVA series models generally exhibit strong performance across multiple relational categories, demonstrating proficiency in accurately judging both occlusion and social relationships. Particularly noteworthy are the results concerning occlusion relationships. We observe that LLaVA-1.5 and LLaVA-NeXT, which utilize Vicuna as their Large Language Model (LLM) backbone, substantially outperform newer models like InternVL3 and Qwen2.5-VL, despite the latter possessing more recent architectures. Furthermore, within the LLaVA family, LLaVA-OneVision, which employs Qwen as its LLM, shows a marked decrease in performance when reasoning about occlusion compared to its Vicuna-based counterparts. The Vicuna-based models maintain accuracy levels around 50% on this task. Our analysis suggests this discrepancy might stem from differing pre-training objectives or inherent model behaviors: Vicuna-based models, potentially trained with extensive "completion"-style tasks, may possess an enhanced capacity for reasoning or speculation under visual uncertainty, aiding them in occluded scenarios. Conversely, Qwen-based models appear more logically conservative; they exhibit a lower propensity to speculate about entities that are difficult for the visual encoder to clearly discern, often defaulting to responses such as 'E (answer not listed)' or 'F (object not depicted in the image)'. Overall, while contemporary MLLMs demonstrate advancements in logical coherence and judgmental precision, their proficiency in handling complex relational understanding tasks, especially under challenging conditions like occlusion, indicates that significant room for improvement remains. Table 7: Model Performance Evaluation on Relation Understanding | Models | LLM | Spatial | Occlusion | Social | Causal | Avg-S | |-----------------|----------------|---------|-----------|--------|--------|-------| | LLaVA-OneVision | Qwen2-7B | 50.49 | 42.63 | 73.68 | 53.44 | 55.06 | | LLaVA-1.5 | Vicuna-7B | 50.08 | 51.04 | 67.37 | 50.38 | 54.72 | | InternVL3 | Qwen2.5-7B | 58.80 | 45.62 | 60.00 | 54.20 | 54.66 | | InternVL2.5 | InternLM2.5-7B | 53.70 | 43.20 | 65.26 | 52.67 | 53.71 | | MiniCPM-o-2.6 | Qwen2.5-7B | 55.59 | 39.32 | 62.11 | 54.20 | 52.81 | | Qwen2-VL | Qwen2-7B | 48.19 | 45.28 | 71.58 | 41.22 | 51.57 | | Janus-Pro-7B | DeepSeek-7B | 49.51 | 40.17 | 63.16 | 51.91 | 51.19 | | LLaVA-NeXT | Vicuna-7B | 44.49 | 50.92 | 63.16 | 45.80 | 51.09 | | MiniCPM-V-2.6 | Qwen2-7B | 43.26 | 41.01 | 54.71 | 51.91 | 47.72 | | InternVL2 | InternLM2.5-7B | 45.89 | 41.94 | 53.68 | 46.46 | 46.99 | | Qwen2.5-VL | Qwen2.5-7B | 46.05 | 42.17 | 52.63 | 42.75 | 45.90 | | MiniCPM-V-2.5 | Llama3-8B | 49.51 | 37.48 | 22.34 | 47.88 | 39.30 | #### A.9 Details of Reasoning and Decision Visual scene reasoning and subsequent decision-making represent the culminating and most critical evaluation stage within our AD2Bench. This phase demands that models not only achieve comprehensive perception and understanding of the current driving scene but also exhibit profound inferential and predictive capabilities regarding scene dynamics and potential risks, thereby guiding the ego-car towards prudent actions. Our evaluation methodology for this component is bifurcated into two settings: direct inference without Chain-of-Thought (CoT) prompting, and reasoning augmented by a hierarchical CoT process. Performance without Chain-of-Thought (CoT) Guidance. As delineated in Tab. [2,](#page-7-0) a majority of the evaluated Large Multimodal Models (LMMs) exhibit suboptimal performance when tasked with direct reasoning without CoT guidance, with scores typically ranging between 35% and 45%. Notably, even powerful models such as Qwen2.5-VL struggle to achieve satisfactory results under this setting. Our analysis reveals a discernible trend: models employing input image patching strategies (e.g., MiniCPM, InternVL), which involves segmenting the input image into multiple sub-patches and resizing each to the model's optimal input dimensions, tend to outperform those utilizing dynamic resolution scaling (e.g., Qwen2.5-VL). The latter approach adapts the aspect ratio based on the original image size and moderately increases input token counts. For scenarios depicted in Fig. [1,](#page-1-0) such as those involving adverse weather conditions or complex urban environments, decision-making accuracy is heavily predicated on the model's fine-grained perceptual acuity of scene details. While Qwen2.5-VL's dynamic resolution mechanism excels in standard detection tasks, the patch-and-upscale strategy appears to preserve greater visual detail crucial for nuanced reasoning in complex scenes. However, this advantage comes at the cost of increased input token lengths and, consequently, higher computational resource consumption. Intriguingly, we observed a counter-intuitive performance trend within the InternVL series. Despite the architectural advancements from InternVL2 through InternVL2.5 to InternVL3, their efficacy on AD2Bench demonstrated an inverse correlation with model progression. We attribute this phenomenon primarily to two factors: 1) In contrast to InternVL2, the more advanced iterations tend to generate responses that are somewhat diffuse when confronted with the multifaceted elements characteristic of autonomous driving scenarios. Although they articulate a broad range of perceived elements, the final reasoning output often lacks sufficient focus and coherence. 2) While the training datasets for the InternVL series have expanded progressively, the proportion of domain-specific autonomous driving data (largely concentrated in datasets like nuScenes and Waymo) has effectively diminished relative to the total volume. This data dilution may inadvertently temper their specialized reasoning capabilities for autonomous driving tasks. Empowerment Effect of Hierarchical Chain-of-Thought (CoT). Conversely, the introduction of a hierarchical CoT framework, as illustrated in Fig[.4,](#page-4-0) yields substantial performance enhancements across most models, as evidenced in Tab[.2.](#page-7-0) Specifically, InternVL2.5, InternVL3, and Qwen2.5-VL demonstrate remarkable improvements, with their scores increasing by 18.725, 20.166, and 17.60 points, respectively. Particularly noteworthy is that InternVL3, when augmented with CoT, surpasses other models of comparable parameter counts. We attribute this to CoT's ability to significantly bolster the overall coherence of InternVL3's outputs, thereby channeling its reasoning process more effectively towards factors critically impacting the ego-car. In contrast, MiniCPM-o-2.6, which registered the highest performance in the absence of CoT, demonstrates a comparatively modest improvement with CoT integration. Our investigation suggests that models like InternVL3 and Qwen2.5-VL, having been exposed to considerable volumes of rationale-augmented data during their second-stage instruction fine-tuning (SFT), exhibit stronger adaptability to CoT prompting. Conversely, the MiniCPM series, primarily optimized for on-device deployment, while benefiting from its patch-based input strategy for detail perception, appears to exhibit weaker adherence to extended CoT prompts, thus limiting its incremental gains relative to models like InternVL3. Granular Analysis of CoT-Driven Reasoning Steps. To further dissect the influence of CoT, Tab[.3](#page-8-0) presents a granular evaluation of model performance during the CoT reasoning process, averaged across all intermediate steps. This assessment encompasses four core dimensions: correctness (similarity to ground truth), logical consistency between individual reasoning steps, overall coherence of the reasoning chain, and the efficacy of the CoT in supporting the final decision. These intermediate step metrics exhibit a strong positive correlation with the final comprehensive evaluation scores post-CoT guidance. A salient observation is that the average correctness scores during the CoT process consistently surpass the ORCS scores for most models (e.g., Qwen2.5-VL: *59.330 vs. 55.028*). This suggests that LMMs may often achieve higher accuracy in individual reasoning steps than in maintaining the integrity of the entire reasoning process leading to the final answer. We hypothesize that this discrepancy arises because prevalent training paradigms predominantly utilize questionanswer pairs, lacking explicit supervision for intermediate reasoning pathways. Consequently, while models can effectively learn to approximate correct final answers through strong fitting, they may still face challenges in constructing robust, step-by-step, logically sound, and coherent reasoning chains when addressing complex multi-hop reasoning tasks. #### A.10 Chain of Thought(CoT) Format Fig. [6](#page-22-0) illustrates our Chain-of-Thought (CoT) process for decomposing complex autonomous driving decision-making problems. This methodology employs a hierarchical reasoning process, structured around key elements within the traffic scene, to address intricate decision-making tasks. The reasoning process commences with an assessment of environmental factors, which typically constitute the most readily perceived and predominant visual information. This foundational environmental perception then grounds the identification of key vehicles pertinent to the ego-vehicle's driving behavior. As other vehicles often represent the most significant immediate threats, the CoT process, upon establishing environmental awareness, prioritizes reasoning about these potentially hazardous agents. The reasoning hierarchy further extends to Vulnerable Road Users (VRUs), such as pedestrians and cyclists, whose dynamic presence critically influences the ego-vehicle's maneuvers. Although potentially posing a less immediate physical threat than other vehicles, VRUs demand dedicated perceptual analysis and reasoning. Subsequently, the fourth reasoning layer, designated herein as Step 4, addresses traffic rules. The accurate extraction and interpretation of relevant traffic regulations from the scene are paramount for ensuring sound subsequent decision-making. Finally, by consolidating insights derived from these hierarchically processed layers of perception and understanding, the Chain-of-Thought process culminates in a rational and justifiable driving decision. #### A.11 Case Study of Chain of Thought(CoT) Weather Analysis. Fig. [7](#page-23-0) and [8](#page-24-0) present responses from two advanced models, InternVL3 and Qwen2.5-VL, to challenges posed by adverse weather conditions in complex scenarios. In these figures, areas shaded yellow indicate reasoning errors, while red or blue highlights denote correct portions of the answer. For the first image in Fig. [7,](#page-23-0) we observe that Qwen2.5-VL, despite perceiving road fog, still concludes the weather is 'overcast' and fails to correctly distinguish whether the current scene is rainy or foggy. Such light fog conditions, being similar in appearance to rainy scenes, can easily lead to model misjudgment. In contrast, InternVL3 accurately identifies the current weather and also provides a fine-grained description of pedestrians. However, both models exhibit distinct focuses when describing traffic signs and road markings: Qwen2.5-VL identifies a yellow pedestrian crossing, whereas InternVL3 points out a sign on the left. Our expectation is for models to comprehensively perceive and understand the entire scene; thus, their current capabilities in understanding this specific scenario still reveal certain deficiencies. Vision Prompt Analysis. Regarding the second image in Fig. [8,](#page-24-0) the most critical element is a black sedan ahead, obscured by rain, which is either making a U-turn or crossing the road. Qwen2.5-VL perceives several white cars parked on the roadside; however, these are irrelevant to the ego car's current driving status. The priority is key information that can impact the ego car's driving decisions. InternVL3 performs poorly in this instance. Although it segments the input into patches and resizes them to a fixed dimension for the vision encoder, this method appears ineffective here, as the model fails to detect any vehicles. This oversight is extremely detrimental to safe driving. To further evaluate the reasoning capabilities of MLLMs and to mitigate the issue of insufficient perceptual acuity in vision encoders, we experimented with incorporating vision prompts in Fig. [9.](#page-25-0) By overlaying annotated points onto the image and providing this composite input to the MLLM, we found that Qwen2.5-VL, while misidentifying the vehicle type, correctly recognized the vehicle's color and direction of movement (from right to left). This indicates that our vision prompts are largely effective, at least enabling the model to perceive the presence and movement of the vehicle ahead. Similarly, InternVL3 demonstrated even better performance: its vehicle type identification was more accurate than that of Qwen2.5-VL, and its overall responses were largely correct. This showcases the potential of vision prompts for conducting in-depth assessments of MLLMs' advanced reasoning capabilities. #### A.12 Case Study of Other Tasks Fig. [10](#page-26-0) and [11](#page-27-0) present further case studies across various tasks. From Fig. [10,](#page-26-0) it is evident that adverse weather and complex scenarios significantly impair the perceptual capabilities of MLLMs. This is particularly apparent in a specific image from this figure (referred to as ), concerning the detection of pedestrian presence. In this nighttime intersection scene, although several pedestrians are situated on the sidewalk across from the ego-vehicle's front-right, most models fail to detect them. This failure is attributed to occlusion by a white car turning from the front-right, compounded by the low-light conditions. Such perceptual misses are highly detrimental to subsequent understanding and reasoning processes. Fig. [11](#page-27-0) showcases MLLM performance on localization, detection, and Optical Character Recognition (OCR) tasks. For localization and detection, Qwen2.5-VL notably outperforms other models. Many models tend to output coordinates in a proportional (normalized) format; however, due to image patching and sampling operations during input processing, relying on these proportions often fails to accurately reconstruct true bounding boxes. In contrast, approaches similar to those used by Qwen2.5-VL and Qwen2-VL, which involves passing the entire image to subsequent modules via a dynamic resolution scheme—achieve higher scores on these tasks. This improved performance stems from the fact that the final bounding box and localization point coordinates are established within the entire image's coordinate system, rendering full-image input superior to patched or sampled inputs in terms of overall perceptual integrity and output format consistency. Regarding OCR tasks, especially in low-light conditions, most models manage to output text in the expected format. LLaVA-1.5, however, only outputs coordinates, indicating poor instructionfollowing capabilities. Our findings show that InternVL series models achieve significantly better OCR recognition results compared to others. Conversely, MiniCPM series models, which also utilize patched image inputs, deliver underwhelming performance. This suggests that challenges in OCR perception are not solely attributable to the input format and may necessitate greater consideration of training data. We observe that MiniCPM models, designed for edge-device deployment, possess training datasets with substantially less OCR-specific content than more general-purpose MLLMs like the InternVL series, a factor that critically contributes to their diminished OCR efficacy. #### A.13 Limitations The core contribution of this research is the development of AD2 -Bench, the first benchmark for evaluating the Chain-of-Thought (CoT) capabilities of Multimodal Large Language Models (MLLMs) in autonomous driving under adverse weather and complex scenarios. By collecting approximately 10k real-world images and dedicating substantial effort from domain experts to create over 5,400 meticulous, purely manual CoT annotations, we ensure that each reasoning step has an explicit ground truth, thereby enabling a fine-grained evaluation of model reasoning processes. While AD2 - Bench achieves a notable level in the diversity of challenging real-world scenarios covered and the granularity of its annotations, providing a valuable tool for research in this domain, we acknowledge certain limitations. To ensure a high standard of annotation quality and depth, although we have designed an atomic annotation system, our current approach for atomic questions relies on manual annotation by domain experts to guarantee quality at this stage. This stringent commitment to quality, while establishing a solid foundation for the benchmark, inherently incurs considerable construction costs. Concurrently, we must recognize that the complexity and dynamism of real-world driving environments are effectively boundless. For instance, while AD2 -Bench includes a variety of typical adverse weather conditions and complex traffic situations, the current version may not exhaustively cover certain extremely rare or presently unforeseeable combinations of edge cases. This also reflects the inherent challenge of constructing a finite dataset that can perfectly capture all facets of the real world. Looking ahead, as the capabilities of Multimodal Large Models (MLLMs) continue to evolve—a key technological area that AD2 -Bench focuses on evaluating—we aim to more deeply integrate these increasingly powerful MLLMs into our proposed atomic annotation system. This direction not only offers a potential avenue for exploring cost reduction and efficiency improvements in building larger-scale and more diverse datasets while upholding high annotation standards, but also further underscores AD2 -Bench's role as an important platform for advancing methodological developments in this field. # Case Study of Chain of Thought: Question: Analyze the provided traffic scenario from the ego car's perspective. Perform a step-by-step Chain-of-Thought analysis. Structure your entire response using only the specific tags `` and `` to enclose the answer for each step N (where N is the step number). Do not include any other text, prefixes, or labels outside of these start and end tags for each step's answer. What are the current weather and road conditions, and how will these natural conditions affect the ego car's driving behavior? Step2: Besides the aforementioned natural conditions, what other vehicles will affect the ego car's driving behavior? Please provide their attributes (color, appearance), position, and movement (relative to the ego car) respectively. Step3: What other vulnerable road users will affect the ego car's driving behavior(eg. pedestrian,Rider,cyclist)? Please provide their attributes (color, appearance), position, and movement (relative to the ego car) respectively. Step4: In addition to the aforementioned objectives, observe objective traffic information including traffic lights, traffic signs, lane lines, and road markings. Step5: From the ego car's perspective, what should it do to ensure safety? > Figure 6: The visualization of Chain of Thought for MLLM. By breaking down unified scene problems into fine-grained perception, understanding, and reasoning tasks, CoT assists MLLM in achieving a hierarchical and comprehensive understanding of the current scene for making the most reasonable decisions. # Case Study ofChain of Thought(): ![](_page_23_Picture_1.jpeg) ![](_page_23_Picture_2.jpeg) ![](_page_23_Picture_3.jpeg) #### Qwen2.5-VL: Step1: [The current natural conditions appear overcast with a gray sky, suggesting possible rain or fog. This could reduce visibility and make road surfaces slippery, affecting the ego car's driving behavior by requiring slower speeds and increased braking distances...] Step2: [Besides the weather, there are vehicles ahead on the road. They are white cars, positioned in the distance, moving towards the ego car.The approaching vehicles may require the ego car to slow down or stop to avoid collisions...] Step3: [There is a pedestrian crossing the street, indicated by the yellow stripes and the pedestrian sign. The pedestrian crossing the street necessitates the ego car to yield the right of way, ensuring the pedestrian's safety...] Step4: [Objective traffic information observed includes yellow stripes (road markings) and a pedestrian sign (traffic sign)...] Step5: [From the ego car's perspective, it should maintain a safe following distance from the vehicles ahead, reduce speed due to the overcast weather, and be prepared to stop at any time. It should also be vigilant for pedestrians crossing the street and be ready to yield the right of way accordingly...] # InternVL3: Step1: [The current natural condition is foggy,which reduces visibility. This can affect the ego car's drive behavior by requiring more cautious driving, slower speeds, and increased reliance on other sensors or systems for ...] Step2: [Several cars are visible on the road, some approaching and some moving away. Positioned in the left lane.Some are stationary, others are moving.These vehicles may slow down or stop, affecting the ego car's speed and....] Step3: [Wearing a white hat and dark clothing. Walking on the right side of the road, near the building. Stationary relative to the ego car. This pedestrian walking on the right side may enter the crosswalk, necessitating vigilance....] Step4: [Objective traffic information includes a Pedestrian crossing sign: Blue with a white pedestrian symbol. Positioned on the left side of the road. Stationary. This sign indicates a potential crossing area, requiring the ego car to be cautious of pedestrians....] Step5: [The ego car should reduce speed, maintain a safe distance from the vehicles ahead, and be prepared to stop if a pedestrian enters the crosswalk. It should also be aware of the reduced visibility due to fog and use additional sensors if available...] Figure 7: The visualization of Qwen2.5-VL and InternVL3's perception and reasoning under adverse weather conditions focuses mainly on the current thin fog weather, with yellow background representing output errors and blue/red highlights representing correctly expressed parts. # Case Study ofChain of Thought(): ![](_page_24_Picture_1.jpeg) ![](_page_24_Picture_2.jpeg) #### Qwen2.5-VL: Step1: [The current natural condition is rainy, as indicated by the wet road surface and raindrops on the windshield. This will likely reduce visibility and traction, requiring the ego car to drive more cautiously...] Step2: [There are other cars visible in the distance, appearing as dark silhouettes against the wet road. They are stationary or moving slowly, which could indicate reduced speed due to the weather, thus affecting the ego car's driving behavior...] Step3: [There are no vulnerable road users (such as pedestrians, riders, or cyclists) that would affect the ego car's driving behavior...] Step4: [Objective traffic information includes the wet road surface,which can cause hydroplaning, reducing the car's ability to stop quickly. There are also trees lining the street, which might obstruct visibility of oncoming traffic...] Step5: [The ego car should maintain a safe following distance from other vehicles, reduce speed, and be prepared to brake suddenly. It should also be cautious of pedestrians and be ready to adjust its path if necessary. The driver should use headlights and turn signals appropriately to increase visibility and...] #### InternVL3: conditions...] Step1: [The current natural condition is rainy, as indicated by the wet road and raindrops on the windshield. This weather condition can affect the ego car's drive by reducing visibility and increasing the risk of hydroplaning, requiring the driver to slow down and maintain a greater following distance...] Step2: [From the current perspective, there are no other vehicles visible on the road ahead that would affect the ego car's driving behavior...] Step3: [Information regarding vulnerable road users (such as pedestrians, riders, or cyclists) affecting the ego car's drive is not included in this scene...] Step4: [Besides the rainy weather, other factors affecting the ego car's behavior include buildings and trees lining the street. The buildings are described as light green and beige multi-story structures on the right side, while the trees are various types lining the street. Both are stationary. Although these stationary objects do not pose a direct threat or affect the car's movement directly, they can potentially affect visibility if not properly managed...] Step5: [To ensure safety, the ego car should maintain a safe following distance from the white car ahead, be prepared to slow down or stop if the black car further ahead slows down, and be vigilant of the pedestrian crossing the street. The driver should also reduce speed and use caution due to the wet and slippery road Figure 8: The visualization of perception and reasoning of Qwen2.5-VL and InternVL3 in adverse weather conditions mainly focuses on black cars that are blocked by rainwater and cross the road ahead. The yellow background represents output errors, and the blue/red highlights represent the correct parts. # Vision Prompt for Chain of Thought(): ![](_page_25_Picture_1.jpeg) #### Qwen2.5-VL: Original Step2: [There are other cars visible in the distance, appearing as dark silhouettes against the wet road. They are stationary or moving slowly, which could indicate reduced speed due to the weather, thus affecting the ego car's driving behavior...] Prompted Step2: [The highlighted vehicle is a dark-colored (possibly black or dark gray) standard motorcycle with a visible rider. It is positioned ahead of the ego car, slightly to the left within the same lane, at a moderate distance. The motorcycle is moving forward in the same direction as the ego car, likely at a similar or slightly faster speed. Its presence will affect the ego car's driving behavior by necessitating careful attention to maintaining a safe following distance and being prepared for any sudden movements or changes in speed by the motorcycle.] #### InternVL3: Original Step2: [From the current perspective, there are no other vehicles visible on the road ahead that would affect the ego car's driving behavior...] Prompted Step2: [The highlighted dark-colored SUV/crossover, with visible headlights, is positioned ahead and to the right of the ego car, currently appearing to merge or turn into the ego car's path from a side entry. It is moving from right to left, actively maneuvering into the ego car's vicinity, initially at a relatively slower speed. This vehicle's actions will significantly affect the ego car, requiring deceleration, possible steering adjustments (such as moving to the left if clear), and heightened caution to avoid a collision, especially given the wet road conditions, while anticipating the SUV's complete maneuver into the path.] #### GroundTruth: Step2: [In addition to the wet and slippery road conditions due to rain,there is a black sedan making a U-turn/across in the distance ahead, which will significantly affect the ego car's driving behavior. This car is positioned far ahead, and its U-turn/across maneuver means its direction of travel could change abruptly, requiring the ego car's driver to remain vigilant and prepared for emergency braking or evasive actions.If the ego car is moving too quickly or is too close to the vehicle ahead, the risk of a rear-end collision is further amplified, particularly as the current slippery road conditions will increase braking distances.] Figure 9: The visualization of changes in perception and reasoning under adverse weather conditions for Qwen2.5-VL and InternVL3 after providing a vision prompt. ![](_page_26_Figure_0.jpeg) Figure 10: The visualization of Basic Perception Tasks of Multimodal large language Models under Adverse Weather and Complex Scene Conditions, with yellow highlights representing correct answers. # Case Study of Perception: Question1(Localization): Where is motorcycle located in the image? (Output the object's center point coordinates as [x, y]. True: [1778.0, 786.5] Answers: Qwen2.5-VL: (1720, 805); Qwen2-VL: (920, 730); MiniCPM-o-2.6: (881, 340); MiniCPM-V-2.6: (885.5, 694.5); MiniCPM-V-2.5: (0.775, 0.775); InternVL2: (800, 700); InternVL2.5: (0.550, 0.850); InternVL3: (890, 680); LLaVA-1.5: (0.720, 0.745); LLaVA-NeXT: (0.893, 0.742); LLaVA-OneVision: (1, 0); Janus-Pro: (10, 10) Question2(Detection): What is the bounding box of the bus in the image? (Just respond with the object's one most probable bounding box coordinates as (x\_min,y\_min, x\_max, y\_max) without any othertext caption.) True: [765.0, 765.0, 832.0, 800.0] Answers: Qwen2.5-VL: (768, 772, 840, 815); Qwen2-VL: (415, 708, 450, 744); MiniCPM-o-2.6: (0.54, 0.70, 0.60, 0.77); MiniCPM-V-2.6: (456, 425, 586, 575); MiniCPM-V-2.5: (383, 488, 448, 663); InternVL2: (0, 0, 998, 998); InternVL2.5: (300, 200, 400, 300); InternVL3: (420, 650, 460, 760); LLaVA-1.5: (0, 0.66, 0.2, 0.82); LLaVA-NeXT: (0.138, 0.652, 0.322, 0.842); LLaVA-OneVision: (10, 20, 30, 40); Janus-Pro: (100, 100, 200, 200) Question3(OCR): This is an OCR task. We have provided an image with a red rectangular box. Please recognize the text within the red rectangular box. The output should only include digits (0-9), English letters (A-Z and a-z), and spaces. Please ignore and remove all punctuation marks (such as ".", ",", "-", etc.) and any other characters that are not letters or numbers. The specific bounding box for the text recognition is [405.5, 422.8, 495.9, 461.1] (format: x\_min, y\_min, x\_max, y\_max). True: T730884C Answers: Qwen2.5-VL: 730884C; Qwen2-VL: 17308840; MiniCPM-o-2.6: LZ0884C; MiniCPM-V-2.6: 408840; MiniCPM-V-2.5: 30B84C; InternVL2: 17308840; InternVL2.5: T730884C; InternVL3: T730884C; LLaVA-1.5: [0.19, 0.52, 0.49, 0.72]; LLaVA-NeXT: 130888; LLaVA-OneVision: 7730884C; Janus-Pro: Case Study of Other Tasks: Figure 11: The visualization of Advanced Perception Tasks of Multimodal large language Models under Adverse Weather and Complex Scene Conditions, with yellow highlights representing correct answers. 405542284959 # NeurIPS Paper Checklist #### 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction provide a comprehensive overview of the background and the motivation of this study, efficiently outlining its main contributions, thus accurately reflecting the paper's scope and significance. # Guidelines: - The answer NA means that the abstract and introduction do not include the claims made in the paper. - The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. - The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. - It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. #### 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper provides the limitation discussion in Appendix [A.13.](#page-20-0) #### Guidelines: - The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. - The authors are encouraged to create a separate "Limitations" section in their paper. - The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. - The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. - The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. - The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. - If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. - While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. #### 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The method in this article includes theoretical formulation. Moreover, it covers detailed ablative studies and qualitative Studies, ensuring completeness and accuracy in the theoretical presentation. #### Guidelines: - The answer NA means that the paper does not include theoretical results. - All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. - All assumptions should be clearly stated or referenced in the statement of any theorems. - The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. - Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. - Theorems and Lemmas that the proof relies upon should be properly referenced. #### 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We have provided a detailed description of our proposed benchmark and metric, experimental settings and dataset description for reproduction. #### Guidelines: - The answer NA means that the paper does not include experiments. - If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. - If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. - Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. - While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example - (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. - (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. - (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). - (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. #### 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the anonymous URLs in our paper to provide the codes and datasets in the reviewing period. After the acceptance, we will make our codes publicly available. # Guidelines: - The answer NA means that paper does not include experiments requiring code. - Please see the NeurIPS code and data submission guidelines ([https://nips.cc/](https://nips.cc/public/guides/CodeSubmissionPolicy) [public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details. - While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). - The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ([https:](https://nips.cc/public/guides/CodeSubmissionPolicy) [//nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details. - The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. - The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. - At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). - Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. #### 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We give the experimental settings in Section [4.1](#page-6-1) and give the more detailed discription in Appendix [A.4.](#page-14-0) # Guidelines: - The answer NA means that the paper does not include experiments. - The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. - The full details can be provided either with the code, in appendix, or as supplemental material. #### 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Generally, papers in our field do not include error bars, and we found that the model evaluating is quite stable with little variation across multiple runs. In addition, we do not have enough compute resources to report error bars for each experiment. # Guidelines: • The answer NA means that the paper does not include experiments. - The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. - The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). - The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) - The assumptions made should be given (e.g., Normally distributed errors). - It should be clear whether the error bar is the standard deviation or the standard error of the mean. - It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. - For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). - If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. #### 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We give the information that all experiments are conducted on 8×RTX4090 GPUs and give more details about memory, training time, and compute resources for further study in Appendix [A.4.](#page-14-0) #### Guidelines: - The answer NA means that the paper does not include experiments. - The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. - The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. - The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). ### 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ? Answer: [Yes] Justification: After carefully reviewing the referenced document, we certify that our research preserves anonymity and we comply with the Code of Ethics. # Guidelines: - The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. - If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. - The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). #### 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The paper primarily focuses on vision question answer evaluation benchmark using publicly available datasets and information on web that have undergone thorough validation. This study only serves as a evaluation benchmark, which is not directly applicable to practical scenarios until now. ### Guidelines: - The answer NA means that there is no societal impact of the work performed. - If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. - Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. - The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. - The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. - If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). #### 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The proposed benchmark are contructed on public datasets such as ACDC, BDD100K, etc. These datasets have been extensively used in the computer vision community and have undergone comprehensive safety risk assessments. # Guidelines: - The answer NA means that the paper poses no such risks. - Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. - Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. - We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. #### 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: In the paper, we clearly specified the datasets and code sources used, and provided appropriate citations in the reference section. We also mark that our evaluation codes are based on the ms-swift (version 3.4.1) library. #### Guidelines: - The answer NA means that the paper does not use existing assets. - The authors should cite the original paper that produced the code package or dataset. - The authors should state which version of the asset is used and, if possible, include a URL. - The name of the license (e.g., CC-BY 4.0) should be included for each asset. - For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. - If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. - For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. - If this information is not available online, the authors are encouraged to reach out to the asset's creators. #### 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We have provided the URLs of the code and data, along with detailed usage instructions in our paper. After completing the review process, we will make the code and data publicly available to the community. #### Guidelines: - The answer NA means that the paper does not release new assets. - Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. - The paper should discuss whether and how consent was obtained from people whose asset is used. - At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. #### 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: This research does not involve any crowdsourcing experiments or studies with human subjects. ### Guidelines: - The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. - Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. - According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. #### 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: No crowdsourcing experiments or research with human subjects were involved in this study. All experiments were conducted using codes and GPU servers. # Guidelines: - The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. - Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. - We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. - For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. ### 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required. Answer: [Yes] Justification: In the process of creating our benchmark, we called the LLM API to help us filter data, evaluate the validity of the data, and calculate the similarity between experimental results and correct answers. These are all detailed in Section [3.1,](#page-3-1) [3.2](#page-4-1) and appendix [A.2.](#page-13-0) #### Guidelines: - The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. - Please refer to our LLM policy () for what should or should not be described.