Obsidian/Zotero/001_artiklid/changtimeimmdatasetbenchmark2025.md

---

category: literaturenote citekey: changtimeimmdatasetbenchmark2025 title: "Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series" authors: "Chang, Ching; Hwang, Jeehyun; Shi, Yidan; Wang, Haixin; Peng, Wen-Chih; Chen, Tien-Fu; Wang, Wei" year: 2025 date: 2025-00-00 2025 url: "https://arxiv.org/abs/2506.10412v1" zotero_key: IWB3MCA5 zotero_storage: GJ3XMSRE collections: transpordi_planeerimine folder: 001_artiklid firstAuthor: "Chang, Ching"

status: converted

Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series

Ching Chang1,2^∗ Jeehyun Hwang¹ Yidan Shi¹ Haixin Wang¹ Wen-Chih Peng² Tien-Fu Chen² Wei Wang^{1 1}University of California, Los Angeles ²National Yang Ming Chiao Tung University

Abstract

Time series data in real-world applications such as healthcare, climate modeling, and finance are often irregular, multimodal, and messy, with varying sampling rates, asynchronous modalities, and pervasive missingness. However, existing benchmarks typically assume clean, regularly sampled, unimodal data, creating a significant gap between research and real-world deployment. We introduce Time-IMM, a dataset specifically designed to capture cause-driven irregularity in multimodal multivariate time series. Time-IMM represents nine distinct types of time series irregularity, categorized into trigger-based, constraintbased, and artifact-based mechanisms. Complementing the dataset, we introduce IMM-TSF, a benchmark library for forecasting on irregular multimodal time series, enabling asynchronous integration and realistic evaluation. IMM-TSF includes specialized fusion modules, including a timestamp-to-text fusion module and a multimodality fusion module, which support both recency-aware averaging and attention-based integration strategies. Empirical results demonstrate that explicitly modeling multimodality on irregular time series data leads to substantial gains in forecasting performance. Time-IMM and IMM-TSF provide a foundation for advancing time series analysis under real-world conditions. The dataset is publicly available at https://www.kaggle.com/datasets/ blacksnail789521/time-imm/data, and the benchmark library can be accessed at https://anonymous.4open.science/r/IMMTSF\_NeurIPS2025.

1 Introduction

Time series analysis plays a foundational role across diverse fields such as healthcare [1, 2], climate science [3, 4], and finance [5, 6]. It underpins critical applications such as patient health monitoring, weather forecasting, disaster response coordination, and financial trend prediction. These applications increasingly require modeling time series data that exhibit irregular sampling, asynchronous observation patterns, and pervasive missingness, often across multiple modalities. However, traditional time series research has largely relied on clean, unimodal, regularly-sampled datasets where observations arrive at fixed intervals. These assumptions rarely hold in real-world scenarios. In practice, time series data are often irregular, multimodal, and messy, with variable sampling rates, asynchronous modalities, and pervasive missingness.

Despite the importance of addressing such irregularities, most publicly available time series benchmarks largely overlook these complexities. Datasets such as the UCR archive [7], M4 [8], and many multimodal datasets like Time-MMD [9] assume regular temporal grids and fixed sampling intervals. They fail to reflect the diverse irregularities arising in operational systems, human behavior, and natural processes. This lack of realism limits the development and evaluation of models capable of handling the challenges faced in real-world deployments.

^∗Correspondence to: Ching Chang chingchang0730@ucla.edu

The primary obstacle to advancing time series analysis under real-world conditions lies in the absence of benchmarks that accurately reflect the irregular, heterogeneous nature of practical data. Despite the growing body of research on both time series forecasting and multimodal learning, three critical challenges remain: (1) The oversight of real-world irregularity in existing time series benchmarks, as most prior datasets [9, 8, 7] prioritize clean, regularly-sampled settings and fail to capture the pragmatic complexities of operational systems and natural processes; (2) The limited exploration of multimodal integration within irregular time series modeling, where existing work [10, 3, 11, 12, 13] primarily focuses on unimodal numerical sequences and neglects the asynchronous nature of textual and other auxiliary modalities; and (3) The missing systematic understanding of irregularity causes, as prior efforts rarely distinguish between the major categories of irregularity such as trigger-based events, constraint-based sampling restrictions, and artifact-based system failures — that underlie observed irregular patterns, limiting causal interpretability and model robustness.

To address these challenges, we introduce a new dataset and benchmark library designed for systematic study of irregular multimodal multivariate time series analysis. Our main contributions are:

• Cause-Driven Irregular Multimodal Dataset. We construct Time-IMM, the first benchmark that not only explicitly categorizes real-world irregularities into nine cause-driven types—spanning trigger-based, constraint-based, and artifact-based mechanisms—but also incorporates richly annotated textual data alongside numerical observations. The textual modality captures asynchronous, auxiliary information such as clinical notes, sensor logs, or event descriptions, providing critical context for interpreting and forecasting multivariate time series. This enables comprehensive modeling of complex, cross-modal interactions and irregular sampling behaviors across multiple domains.
• Benchmark Library for Irregular Multimodal Time Series Forecasting. We develop IMM-TSF, a plug-and-play benchmarking library for forecasting on irregular multimodal time series. It supports asynchronous integration of numerical and textual data through modular components for encoding and fusion, enabling flexible and realistic experimentation.
• Empirical Validation of Irregularity and Multimodality Modeling. Through extensive experiments across diverse types and causes of irregularity, we show that incorporating multimodality on top of irregular time series modeling yields substantial forecasting improvements—achieving an average MSE reduction of 6.71%, and up to 38.38% in datasets with highly informative textual signals. These results underscore the superiority of multimodal approaches in handling the complexities of real-world time series forecasting.

By introducing Time-IMM and IMM-TSF, we aim to catalyze research in robust, generalizable time series analysis, moving the field beyond simplified assumptions toward truly real-world scenarios. Further details on related work and dataset limitations are provided in Appendix A and Appendix B, respectively.

2 Time-IMM: A Dataset for Irregular Multimodal Multivariate Time Series

While prior benchmarks in time series forecasting and multimodal learning have expanded the diversity of application domains, they largely overlook the intrinsic irregularities that characterize real-world data collection. Existing datasets predominantly assume regular sampling patterns, simple temporal alignments, and domain-driven organization, abstracting away the diverse causes that drive irregular observations [14, 15, 16]. Even recent multimodal datasets [9] focus on domain variety rather than addressing the deeper methodological challenge of irregularity. As a result, current benchmarks fail to capture the pragmatic complexities introduced by trigger-based events, constraintdriven sampling, and artifact-induced disruptions. To bridge this gap, Time-IMM explicitly centers irregularity as a core characteristic of time series data, categorizing its causes and curating multimodal datasets that reflect real-world irregular structures.

2.1 Taxonomy of Irregularity in Time Series

Irregularities in real-world time series are driven by diverse and systematic causes, not just random or arbitrary occurrences. Understanding the origins of these irregularities is crucial for developing models that are robust to realistic conditions and capable of meaningful interpretation. To structure

Figure 1: Overview of the taxonomy of irregularities in time series data. Each type is exemplified by a corresponding dataset curated as part of the Time-IMM dataset.

Figure 2: Visualization of three representative features from each dataset in Time-IMM, illustrating the diverse sampling patterns and distinct types of irregularity.

this complexity, we propose a taxonomy based on the primary nature of the factors causing irregularity. An overview of our taxonomy is illustrated in Figure 1.

First, Trigger-Based Irregularities arise when data collection is initiated by external events or internal system triggers, meaning observations occur only in response to specific happenings rather than at predetermined intervals. Second, Constraint-Based Irregularities are driven by limitations on when data can be collected, such as operational schedules, resource availability, or human factors, where the sampling process is shaped by external restrictions rather than an ideal or intended schedule. Third, Artifact-Based Irregularities result from imperfections in the data collection process itself, including failures, delays, and asynchrony, where the intended regularity is disrupted by technical or systemic artifacts.

This taxonomy highlights that real-world irregularities are not caused by a single factor but arise from fundamentally different sources, each requiring tailored modeling strategies to capture their structure and implications. Each dataset in Time-IMM exemplifies a distinct type of irregularity, as shown in Figure 1. To complement the taxonomy, Figure 2 presents representative visualizations of three features per dataset, illustrating the diversity of sampling patterns and irregular behaviors. Table 1 summarizes dataset statistics in Time-IMM. We also introduce a set of metrics to characterize time series irregularity—feature observability entropy, temporal observability entropy, and mean inter-observation interval—which respectively quantify: (1) the variability of missingness across features via the normalized Shannon entropy of the feature-presence distribution, (2) the dispersion of events over time using the normalized Shannon entropy of the event-time distribution, and (3) the average time gap between successive observations. Formal definitions of these metrics are provided in Appendix C.

2.1.1 Trigger-Based Irregularities

Trigger-based irregularities are characterized by observations that occur only in response to discrete events or system triggers. We further categorize this group into three representative types, each illustrating different real-world mechanisms leading to event-driven sampling patterns.

Event-Based Logging captures scenarios where observations occur only when notable external events arise, such as geopolitical protests or seismic activity. To represent this pattern, we collect event-driven time series from the GDELT Global Database [17], where each entry corresponds to a discrete global event like a political crisis or major policy change.

Dataset		# Entities # Features	# Unique	# Observations	Feature Observability	Temporal Observability	Mean Inter-Obseration	# Text Entries†	Textual Temporal Observability
			Timestamps		Entropy	Entropy	Interval		Entropy†
GDELT	8	5	34317	193205	1	0.9964	7.2364 hours	14357	0.9896
RepoHealth	4	10	6783	67830	1	0.9658	1.8217 days	12310	0.9821
MIMIC	20	30	91098	219949	0.8461	0.6556	14.6157 minutes	1593	0.6758
FNSPID	10	6	3659	209688	1	0.9969	1.4507 days	20826	0.9488
ClusterTrace	3	11	12615	69001	0.893	0.9753	18.1425 minutes	688	0.9971
StudentLife	20	9	1743	153610	0.92	0.9775	1.0191 days	6623	0.9761
ILINet	1	11	4918	4918	0.9267	1	6.989 days	650	1
CESNET	30	10	51107	512760	1	1	1.17 hours	224	0.9869
EPA-Air	8	4	6587	49552	0.3777	0.9576	1.0242 hours	1244	0.9956

Adaptive or Reactive Sampling describes cases where the system dynamically adjusts its sampling rate based on observed activity levels, such as environmental monitoring systems increasing sampling during severe weather or networks logging more frequently during high congestion periods. We illustrate this in our benchmark with RepoHealth, a dataset collected from GitHub project histories where sampling intervals adaptively shorten during periods of high commit activity and lengthen during quieter phases.

Human-Initiated Observations involve data recording triggered by human judgment or manual action, such as doctors measuring patient vitals based on clinical necessity or users reporting symptoms only when they perceive significant changes. To capture this behavior, we include samples from the MIMIC clinical database [18, 19], where vital signs like heart rate and blood pressure are recorded irregularly according to clinicians' assessments of patient needs.

2.1.2 Constraint-Based Irregularities

Constraint-based irregularities occur when limitations on resources, schedules, or human factors restrict when data can be collected, shaping the sampling pattern according to external constraints rather than an intended schedule. We further categorize this group into three representative types that reflect how operational and resource factors introduce structured irregularities.

Operational Window Sampling refers to situations where observations are possible only during predefined operational periods, such as stock exchanges being open only during business hours or satellites collecting imagery only during orbital passes. To represent this type of irregularity, we construct datasets from FNSPID[5], where financial time series are naturally sampled only during market operating hours, leading to regular gaps outside trading periods.

Resource-Aware Collection describes cases where data is sampled conditionally based on system activity, often to reduce overhead or avoid redundant logging. In large-scale compute clusters, for example, telemetry may be collected only while jobs are running, leading to activity-dependent irregularity. We illustrate this with the ClusterTrace dataset[20], which captures GPU cluster activity at Alibaba over eight weeks. Here, machine-level metrics are recorded only when containers are active, producing structured gaps tied to workload presence and resource scheduling.

Human Scheduling / Availability involves data collection governed by human activity schedules, such as smartphone sensor logs being recorded primarily when users are awake, or clinical measurements being taken during scheduled patient rounds rather than continuously. To illustrate this phenomenon, we include data from the StudentLife[21] study, where smartphone sensing data is inherently irregular due to human behavioral patterns and daily schedules.

2.1.3 Artifact-Based Irregularities

Artifact-based irregularities arise when technical imperfections, failures, or system noise disrupt the intended regularity of data collection, creating unintended gaps or distortions in the observation sequence. We further categorize this group into three representative types that reflect different sources of system-induced irregularity.

Missing Data / Gaps occur when expected data points are not recorded, leaving empty time steps in the sequence. These gaps often result from incomplete submissions, system errors, or data quality issues. In public health settings, missing values can arise when clinics do not submit reports or when technical problems affect data collection. We use the ILINet dataset [9] from the CDC to demonstrate this type of irregularity. While ILINet provides weekly records of influenza-like illness across the U.S., some weeks contain missing or partial data due to non-reporting providers or processing issues.

Scheduling Jitter / Delay happens when data points are recorded at uneven time intervals due to system delays or timing issues. Instead of arriving on a regular schedule, the data shows random gaps and timing shifts. This can be caused by logging processes competing for resources, background maintenance tasks, or data throttling during peak hours. We use the CESNET dataset [22] to show this kind of irregularity since its network flow records often appear at inconsistent times due to internal scheduling and system constraints.

Multi-Source Asynchrony arises when observations from multiple data streams are integrated despite differing internal clocks, sampling rates, or synchronization policies. Examples include environmental sensor networks where temperature, humidity, and air quality sensors report measurements at unsynchronized intervals, and health monitoring systems where wearable devices and clinical records log data at different temporal granularities. To represent this form of irregularity, we compile datasets from EPA Outdoor Air Sensors [23], where heterogeneous sensors report at varying frequencies, creating inherent asynchrony across measurement streams.

2.2 Dataset Construction

To construct Time-IMM, we start with real-world time series data exhibiting diverse forms of irregularity, then follow a two-stage pipeline: curating relevant textual data to pair with each time series, and preserving asynchronous timestamps across both modalities to reflect real-world conditions.

Multimodal Textual Data Curation. Each time series in Time-IMM is paired with contextual textual information, such as situational narratives, observational reports, or human-generated logs, depending on the nature of the dataset. Since the modality and format of text differ across sources, we adopt dataset-specific strategies for acquisition and preprocessing. Details on data sources and collection pipelines for each dataset are provided in Appendix D.

To ensure the textual modality is both relevant and informative, we apply a preprocessing strategy inspired by Time-MMD [9]. This includes an initial filtering step to retain only semantically relevant documents, followed by summarization to improve clarity and compress dense information into concise, interpretable content. Summarization is performed using GPT-4.1 Nano [24] with prompt instructions tailored to preserve factual grounding, avoid hallucination, and allow abstention when relevance is uncertain. Prompt templates and preprocessing examples are included in Appendix E and Appendix F.

Handling Asynchronous Timestamps. Unlike traditional multimodal datasets [9, 25] that assume or enforce temporal alignment between modalities, Time-IMM explicitly preserves asynchronous timestamping. Numerical time series and textual observations are allowed to occur on independent, irregular schedules. We do not impose alignment between the timestamps of numerical and textual data—this reflects real-world conditions where modalities are generated through fundamentally different processes (e.g., continuous sensors vs. episodic reports). This design choice enables more flexible modeling, supporting realistic multimodal fusion strategies that can account for both recency and content semantics.

2.3 Ethical Considerations and Data Access

All datasets in Time-IMM are derived from publicly available sources, with appropriate handling of sensitive domains such as healthcare. We ensure responsible data curation and do not redistribute any content that may violate licensing terms or privacy safeguards. Further details on ethical practices, licensing, and data access policies are provided in Appendix G.

Figure 3: Problem formulation for irregular multimodal multivariate time series forecasting.

Figure 4: IMM-TSF architecture. The library includes modular fusion layers that combine irregular numerical sequences with asynchronous text via timestamp-to-text fusion and multimodal forecast fusion.

3 IMM-TSF: A Benchmark Library for Irregular Multimodal Multivariate Time Series Forecasting

3.1 Problem Formulation

In standard irregular unimodal time series forecasting (TSF), we aim to predict future values of a numerical sequence using its past, irregularly sampled observations. Let X = {[(t (n) i , x (n) i )]P^{n i}=1} N ⁿ=1, where there are N variables and the n-th variable contains Pⁿ past observations. Each time series observation x (n) ^{i ∈ R} occurs at time ^t (n) ⁱ ∈ [Tstart, Tcut], where Tstart denotes the start time and Tcut denotes the cut-off time separating past from future data. We define a set of future query times as Q = {[q (n) k F^{n k}=1} N ⁿ=1, where each q (n) ^k ∈ (Tcut, Tend] and Fⁿ denotes the number of future queries for variable n. The forecasting objective is to learn a function f^θ : (X, Q) 7→ Y , where the predictions are given by Y = {[ˆy (n) k F^{n k}=1} N ⁿ=1, with each yˆ (n) k approximating the true time series value at time q (n) k

To incorporate unstructured textual information, we extend the irregular TSF setup to a multimodal setting. In addition to X, we observe a sequence of time-stamped textual data S = {(τ^j , s^j )} L^{S j}=1, where s^j is a text input observed at time τ^j ∈ [Tstart, Tcut], and L^S denotes the number of text observations. Here, Tstart is the shared start time for both the time series and the text modality, and no alignment is assumed between the time series timestamps {t (n) i } and the text timestamps {τj}. The multimodal forecasting objective is to learn a function g^θ : (X, S, Q) 7→ Y that integrates the temporal dynamics of the irregular time series and contextual signals from the text to produce accurate predictions Y = {[ˆy (n) k F^{n k}=1} N ⁿ=1 at the query times Q ⊂ (Tcut, Tend].

3.2 Multimodal TSF Library

To support research on forecasting under realistic multimodal irregularity, we introduce IMM-TSF, a modular benchmark library designed for plug-and-play experimentation. An overview of the library architecture is shown in Figure 4. IMM-TSF supports flexible composition of forecasting pipelines by exposing unified interfaces for numerical encoders, textual encoders, and fusion strategies. Each component is designed to be self-contained and easily swappable, allowing researchers to explore different architectural choices without needing to modify the rest of the pipeline.

Irregular Unimodal TSF Model IMM-TSF supports both irregular time series models and regular forecasting architectures adapted to irregular settings. To adapt regular time series models for irregular data, we follow the canonical pre-alignment strategy introduced in t-PatchGNN [10]. This method interpolates irregular data to a fixed temporal grid, treating relative time and missingness masks as additional input variables. Since forecast queries remain part of the input in our problem formulation, we append their timestamps to the pre-aligned sequence during inference, enabling the model to condition predictions on the desired query times. Further implementation details are provided in Appendix H.

Text Encoder Textual inputs are processed using pretrained language models, which are frozen during training to prevent overfitting. IMM-TSF supports a variety of open-source LLMs and compact encoders, including GPT-2 [26], BERT [27], LLaMA-3.1-8B [28], and DeepSeek-7B [29]. The resulting embeddings form the asynchronous textual input stream for downstream fusion.

Timestamp-to-Text Fusion (TTF) Module In real-world settings, textual data typically arrives asynchronously and is not temporally aligned with the timestamps of numerical time series. The Timestamp-to-Text Fusion module addresses this by using the timestamps of past textual data to produce temporally aware representations for each forecast query. IMM-TSF provides two variants. (1) Recency-Weighted Averaging (RecAvg) performs a Gaussian-weighted aggregation of past text embeddings, where weights decrease with the squared time difference between the text and forecast timestamps. This allows the model to emphasize temporally proximate text while remaining simple and efficient. (2) Time2Vec-Augmented Cross-Attention (T2V-XAttn) enriches each past text embedding with a Time2Vec [30] encoding of its timestamp, then uses cross-attention with a learned forecast query to selectively attend to temporally and semantically relevant past text.

Multimodality Fusion (MMF) Module The Multimodality Fusion module combines text-derived representations with numerical features before prediction, allowing textual context to directly influence the final forecast. IMM-TSF includes two variants. (1) GRU-Gated Residual Addition (GR-Add) encodes a residual correction from the concatenated text and numerical features using a GRU, and blends it into the forecast via a learned gating mechanism that adapts the contribution of text dynamically. (2) Cross-Attention Addition (XAttn-Add) applies multi-head attention between numerical queries and text representations, and adds a scaled residual update to the forecast using a convex combination controlled by a fixed mixing weight. Further architectural details for both TTF and MMF modules are provided in Appendix I.

4 TSF Experiments on Irregular Multimodal Multivariate Time Series

4.1 Experimental Setup

Datasets and Forecasting Setup. We evaluate all models on the nine Time-IMM datasets, each corresponding to a distinct cause of real-world time series irregularity. Each dataset contains multiple entities (e.g., patients, devices, repositories), and we define an entity-specific forecasting task, where both the input and target windows are sampled from the same entity without crossing entity boundaries. We apply a sliding window approach, dividing each window into a past (context) segment and a future (query) segment, enabling models to learn from historical observations and predict future values. Data is split chronologically into 60% training, 20% validation, and 20% test. Window sizes for both context and query segments are dataset-specific, reflecting native timestamp distributions and sampling patterns. Further configuration details are provided in Appendix J.

Baselines. We compare a broad range of models categorized into three groups: (1) Regular Time Series Forecasting Models, including Informer [6], DLinear [31], PatchTST [32], TimesNet [33], and TimeMixer [34]; (2) Large Time Series Models, including the LLM-based TimeLLM [35] and the foundation model TTM [36]; and (3) Irregular Time Series Forecasting Models, including CRU [11], Latent-ODE [3], Neural Flow [12], and t-PatchGNN [10]. We evaluate each model in both unimodal and multimodal settings, using mean squared error (MSE) as our primary performance metric.

Training and Optimization. All models are trained using the Adam optimizer with a learning rate of 1e-3 and a batch size of 8. Training proceeds until early stopping based on validation loss. For model-specific hyperparameters such as the number of layers or hidden dimensions, we use the default settings provided by the original implementations. In multimodal settings, we use the best-performing combination of text encoder, Timestamp-to-Text Fusion (TTF) module, and Multimodality Fusion (MMF) module selected via validation performance. Full details for each baseline and multimodal configuration are provided in Appendix K.

4.2 Results and Analysis

Effectiveness of Multimodality. Across all baselines, we observe consistent performance gains when incorporating textual signals alongside irregular time series data. Figure 5 shows that multimodal variants outperform unimodal counterparts in nearly every case. While prior work like Time-MMD demonstrated the value of text in regular time series forecasting, our results extend this insight to irregular settings. On average, the best multimodal configurations reduce MSE by 6.71%, with gains reaching up to 38.38% in datasets with informative text—confirming that textual context offers complementary cues under real-world irregularities. Detailed results are provided in Appendix L. Among irregular models, t-PatchGNN achieves the strongest results—consistent with its status as the state-of-the-art in irregular unimodal forecasting—and further validates the quality of our carefully curated irregular benchmark. We also find that models suited to small-scale data—such as the few-shot capable TimeLLM and TTM, and the lightweight DLinear—remain

Figure 5: Radar chart comparing forecasting performance across all baseline models. Shaded regions highlight the relative improvement from multimodal over unimodal variants.

competitive in the multimodal setting. This may be due to the smaller size of multimodal datasets compared to unimodal ones, where models that are both simple and possess strong few-shot capabilities are easier to train effectively.

Impact of Irregularity Patterns. Forecasting difficulty in the unimodal setting varies substantially across different subtypes of time series irregularity. Figure 6a plots per-dataset forecasting errors across baselines, grouped by the irregularity subtypes defined in our taxonomy. Datasets characterized by Missing Data (ILINet), Resource-Aware Collection (ClusterTrace), or Event-Based Logging (GDELT) are notably more difficult, as their observation patterns are sparse, erratic, or driven by latent semantic triggers that are inherently harder to model. In contrast, datasets such as RepoHealth (Adaptive Sampling), FNSPID (Operational Window Sampling), and EPA-Air (Multi-Source Asynchrony) tend to be easier to model due to more predictable or recoverable observation structures. For instance, in the FNSPID dataset, which involves stock price forecasting, the irregularity stems from known business-hour trading windows—an expected pattern that simplifies temporal reasoning.

When extending to the multimodal setting, we find that performance gains from incorporating text are not determined solely by the type of irregularity. Instead, improvements vary by dataset and are closely tied to the predictive value of the textual modality. For instance, in FNSPID, where numerical forecasting is already strong due to the regularity of stock market hours, textual news adds limited additional benefit. Conversely, ClusterTrace sees substantial gains from textual integration because its workload annotations describe the currently running processes, which are directly relevant for forecasting future resource usage. These findings highlight that the value of textual data in multimodal irregular forecasting depends more on the semantic alignment and predictive strength of the text than on the irregularity pattern itself.

Ablation of Fusion Modules. To evaluate the design choices within our fusion framework, we conduct ablations over the Timestamp-to-Text Fusion (TTF) and Multimodality Fusion (MMF) modules, as summarized in Figure 6b. For TTF, we observe no notable performance difference between the Time2Vec-augmented cross-attention (T2V-XAttn) and the simpler recency-based averaging (RecAvg), suggesting that both strategies for handling irregular timestamps in textual data are comparably effective. For MMF, GRU-gated residual addition (GR-Add) consistently outperforms cross-attention addition (XAttn-Add), suggesting that learnable gating provides a more effective means of integrating text-derived context into time series representations. By selectively controlling the flow of information, the gating mechanism in GR-Add helps suppress noisy or less relevant textual signals, leading to more stable and robust fusion.

Figure 6: Multimodal forecasting analysis: (a) gains across datasets; (b) fusion strategies; (c) frozen LLM backbones.

Role of the Text Encoder. We evaluate the impact of different LLM backbones on forecasting performance in the irregular multimodal TSF setting by varying the text encoder within IMM-TSF. Specifically, we test GPT-2, BERT, LLaMA 3.1 (8B), and DeepSeek (7B), and assess their performance across representative datasets. As shown in Figure 6c, we observe no significant performance differences across these models, suggesting that the choice of LLM backbone does not strongly influence results under irregular conditions. This lack of sensitivity may reflect the nature of the irregular TSF task: because forecasting hinges more on temporal alignment and contextual anchoring than on deep semantic reasoning, increasing LLM capacity offers no clear benefit. Additionally, while our TTF and MMF modules enable integration of textual signals into the forecasting pipeline, the overall scale of our datasets remains relatively small compared to the large-scale data used to pretrained LLMs, which may limit the advantages of larger models in this setting.

5 Future Work

Forecasting Without Known Query Timestamps. Following prior work such as t-PatchGNN [10], we assume that forecast query times are given, simplifying the task to predicting values at predefined future timestamps. A natural extension is to consider more open-ended settings where the model must first infer when important events or state changes are likely to occur, and then predict their values. This setup—akin to timestamp prediction or event forecasting—introduces additional uncertainty and demands joint modeling of temporal dynamics and event saliency.

Beyond Forecasting Tasks. While IMM-TSF currently focuses on forecasting as the primary task, real-world irregular multimodal time series often require broader capabilities such as anomaly detection, classification, or retrieval. Adapting our framework to support these tasks would enable more comprehensive evaluation of modeling techniques and open new directions for multimodal irregular time series learning under operational constraints.

Beyond Textual Modalities. Our current benchmark focuses on unstructured text as the secondary modality, but many real-world applications involve other asynchronous inputs such as images, audio, or tabular data. Extending irregular time series datasets to support multimodal integration beyond text remains an important direction for building truly general purpose time series models.

6 Conclusion

We introduce Time-IMM, the first benchmark explicitly designed around cause-driven irregularities in multimodal multivariate time series. Our taxonomy covers three distinct categories of irregularity—trigger-based, constraint-based, and artifact-based—captured across nine real-world datasets. To complement this, we present IMM-TSF, a plug-and-play library for forecasting on irregular multimodal time series, with modular support for textual encoders and fusion strategies. Empirical results show that integrating textual information improves forecasting accuracy across all irregularity types, underscoring the need for robust multimodal modeling under realistic conditions. Time-IMM and IMM-TSF are positioned to serve as strong foundations for advancing time series research toward more general and practically relevant solutions.

A Related Dataset Work

Recent progress in time series analysis has been supported by a growing number of publicly available benchmarks [37, 38, 39, 40]. However, most existing datasets make simplified assumptions that limit their realism and utility for modeling complex real-world phenomena. In particular, prior time series datasets often fall short along several key dimensions that Time-IMM is explicitly designed to address.

Lack of Irregularity. Many standard time series datasets assume regularly sampled data where all features are observed at fixed intervals [6, 41, 42, 43]. Even in multimodal contexts, existing benchmarks often maintain regular sampling, smoothing over the irregular behaviors common in realworld systems [9, 44]. This assumption neglects the complexities found in domains like healthcare, environmental sensing, and finance, where sampling can be triggered by events, human decisions, or operational constraints [45, 46, 47].

Lack of Asynchronous Textual Modality. Most existing time series datasets include only numerical observations and lack any unstructured textual modality [7, 8]. While some benchmarks incorporate alternative modalities such as images, audio, or video [48, 49, 50, 51], this work focuses specifically on unstructured textual data as the complementary modality to time series. Yet in many domains, textual signals—such as clinical notes, system logs, or policy descriptions—are crucial for interpretability and context-aware forecasting [52, 53, 54]. Even in datasets that include both time series and text [9, 25], the modalities are often artificially aligned by enforcing shared timestamps, disregarding the fact that different modalities in real-world settings typically follow inherently asynchronous temporal patterns. Our work explicitly supports asynchronous timestamps across modalities, preserving their natural temporal structure for more realistic modeling.

Lack of Reasoning About Irregularity Causes. While some prior datasets contain irregularly sampled time series [10, 55, 56, 57], they do not provide any structured explanation or annotation of why the irregularity occurs. Irregular sampling is typically treated as a nuisance to be smoothed or interpolated away, rather than a meaningful signal that could inform model design or interpretation [58, 59, 60]. This lack of causal reasoning limits model robustness and interpretability, especially in high-stakes domains such as healthcare, environmental monitoring, or fault diagnostics [61, 62]. In contrast, our work introduces a structured taxonomy of irregularity, categorizing it into trigger-based, constraint-based, and artifact-based causes to support more interpretable and realistic modeling.

B Limitations

While our proposed dataset and benchmark provide a valuable foundation for studying irregular multimodal time series forecasting, several limitations remain.

First, the current version of our dataset focuses exclusively on English-language textual data. This choice simplifies preprocessing and alignment but limits the applicability of our benchmark in multilingual or cross-lingual settings, which are common in global-scale systems such as public health surveillance or international finance [63, 64, 65].

Second, our benchmark is currently limited to the task of time series forecasting. We chose this task in part because it does not require additional label collection—making it easier to construct realistic datasets from naturally occurring logs or signals [66, 67]. Moreover, forecasting serves as a critical upstream task in time series analysis: models that accurately capture future dynamics can support a range of downstream applications including anomaly detection, classification, and decision support [68, 69]. Nonetheless, expanding the benchmark to cover these broader tasks would enable more comprehensive evaluation of modeling techniques.

Third, our focus on textual modality reflects the increasing utility of natural language as a rich, unstructured source of auxiliary information. This design decision is motivated by the recent success of large language models (LLMs), which have demonstrated strong capabilities in extracting and contextualizing semantic information from text. However, we do not currently support other modalities such as images, audio, or tabular records—despite their relevance in domains like remote sensing, clinical medicine, or human activity recognition [70, 71, 72].

Fourth, we use pretrained LLMs as frozen text encoders and do not explore task-specific fine-tuning. This design simplifies benchmarking and reduces computational cost, but it may underutilize the

full potential of the language models, especially in domains where domain-specific adaptation could improve alignment with time series signals [73, 74]. To isolate the value of pretrained representations, we conducted additional experiments using Doc2Vec [75] as a train-from-scratch alternative. These results, presented in Appendix L.1, confirm that frozen LLMs consistently outperform Doc2Vec, reinforcing their utility even without fine-tuning.

Finally, while we introduce modular mechanisms for timestamp-aware fusion of text and time series, we do not explore more advanced strategies such as retrieval-augmented generation (RAG) [76, 77], memory modules [78, 79], or reasoning [80, 81], which could further enhance model interpretability and performance.

Despite these limitations, our work offers a structured and extensible platform for modeling real-world time series data under conditions of irregularity and multimodality, and we hope it encourages future research into more general, robust, and semantically grounded forecasting systems.

C Formalization of Irregularity Metrics

We introduce three metrics to quantitatively characterize the irregularity of time series data across entities and modalities. These metrics are designed to be interpretable, normalized where appropriate, and sensitive to structural patterns in missingness and temporal spacing.

C.1 Feature Observability Entropy

This metric quantifies how evenly features are observed across the dataset. Let N be the number of features, and fⁱ the number of non-missing observations for feature i. Define the proportion of observations for each feature as:

$$p_i = \frac{fi}{\sum{j=1}^N f_j} \tag{1}$$

The normalized entropy is computed as:

$$\hat{H}{\text{feat}} = -\frac{1}{\log N} \sum{i=1}^{N} p_i \log(p_i + \epsilon)$$ (2)

where ϵ is a small constant added for numerical stability.

Interpretation: A higher value of Hˆ feat (closer to 1) indicates that observations are evenly distributed across all features. A lower value (closer to 0) suggests that only a subset of features dominate the observation space, revealing imbalance in feature observability.

C.2 Temporal Observability Entropy

This metric measures the dispersion of observations over time by dividing the full time span into K equal-width bins and evaluating the entropy of observation counts within each bin. Let the total time range be divided into K bins of equal width, and let c^k be the number of observations falling into bin k. The normalized count in each bin is:

$$p_k = \frac{ck}{\sum{j=1}^K c_j} \tag{3}$$

The normalized temporal entropy is computed as:

$$\hat{H}{\text{temp}} = -\frac{1}{\log K} \sum{k=1}^{K} p_k \log(p_k + \epsilon)$$ (4)

where ϵ is a small constant added for numerical stability. We fix K = 10 across all datasets to provide a stable and consistent measure of temporal dispersion.

Interpretation: A higher value of Hˆ temp (closer to 1) indicates that observations are evenly distributed across time, suggesting regular sampling behavior. A lower value (closer to 0) reflects temporal clustering or bursty updates.

C.3 Mean Inter-Observation Interval (IOI)

This metric quantifies the average time gap between consecutive observations.

Let ∆tⁱ = tⁱ − ti−¹ denote the interval between successive timestamps across all entities, and let M be the total number of such intervals. The mean IOI is:

$$Mean IOI = \frac{1}{M} \sum_{i=1}^{M} \frac{\Delta t_i}{s}$$ (5)

where s is the number of seconds per time unit (e.g., 86,400 for days).

Interpretation: Smaller values indicate denser data (frequent sampling), while larger values suggest sparser time series. This metric is unnormalized and reported in interpretable units (e.g., hours or days).

D Dataset Details and Data Sources

We organize Time-IMM's nine datasets according to our irregularity taxonomy. For each, we describe the data, point to the source, and summarize how numerical and textual streams were collected.

D.1 Event-Based Logging (Trigger-Based)

Description Observations occur only when external events trigger data collection. This pattern reflects systems that respond to real-world happenings such as political crises, protests, or policy changes.

Source GDELT Event Database 2.0 (2021–2024) [17]. Raw data was downloaded from: http: //data.gdeltproject.org/gdeltv2/masterfilelist.txt

Entity Each entity corresponds to a unique (country, event type) pair. We select 4 countries—USA, GBR, AUS, and CAN—and 2 event types: AGR (Agriculture) and LAB (Labor Union). This results in 8 distinct entities in total.

Collection

• Numerical Time Series: We use GDELT's event dataset from 2021/01/01 to 2024/12/31 and extract five key quantitative features per event: GoldsteinScale, NumMentions, NumSources, NumArticles, and AvgTone. Events are grouped by (country, type) to create entity-specific time series.
• Textual Data: For each event, we download the associated news article using the source URL provided in GDELT. These articles are filtered for relevance and summarized into five-sentence narratives using GPT-4.1 Nano, following prompt instructions designed to ensure factuality, brevity, and contextual clarity.

D.2 Adaptive or Reactive Sampling (Trigger-Based)

Description Sampling frequency is adjusted dynamically based on system activity levels. In this case, repository activity determines when state is logged—shorter intervals during development bursts and longer intervals during quiet periods.

Source We use GitHub repository data from the GitHub Archive via Google BigQuery.

Entity Each entity corresponds to a single GitHub repository. This results in four total entities in the dataset:

• 1. facebook/react
• 2. huggingface/transformers
• 3. kubernetes/kubernetes
• 4. pytorch/pytorch

Collection

• Numerical Time Series: We compute a 10-dimensional numerical time series from GitHub metadata, including issue activity, pull request lifecycle statistics, backlog growth, and age metrics. The data is sampled on an adaptive schedule determined by commit frequency: 1 day during high activity, 7 days for baseline, and 14 days during low activity. Each timestamp reflects a state summary at the chosen interval.
• Textual Data: For each sampled interval, we gather all issue and pull request comments made since the previous timestamp. We limit the number of comments to 5 per issue to control input length. These comments are aggregated by repository and date, then summarized using GPT-4.1 Nano with a prompt designed to extract key concerns, proposals, and unresolved questions. The generated summaries consist of up to five sentences and serve as the unstructured modality aligned with each adaptive sample.

D.3 Human-Initiated Observation (Event-Based)

Description Data is recorded when humans initiate measurement or documentation, such as clinicians logging vitals based on clinical necessity. This results in inherently irregular sampling tied to human judgment and workflow.

Source MIMIC-IV clinical database and MIMIC-Notes [18, 19]. Available at: https:// physionet.org/content/mimiciv/3.1/

Entity Each entity is a single ICU patient who satisfies the following inclusion criteria:

• Patients included in the Metavision system.
• Exactly one hospital admission (to avoid multi-stay confounds).
• Length of stay exceeds 50 days, ensuring sufficiently long time-series.
• Age at admission is ≥ 15 years.
• Has at least one chart-events record and appears in the MIMIC-IV Notes table.

From the remaining population we draw a simple random sample of n = 20 patients to capture diverse clinical courses and recording patterns. The choice of criteria and the preprocessing code mostly follow the logic from the GRU-ODE-Bayes preprocessing pipeline: https://github.com/ edebrouwer/gru\_ode\_bayes/tree/master/data\_preproc/MIMIC

Collection

• Numerical Time-Series

For each selected patient we extract 105 variables spanning: IV infusion and boluses, laboratory chemistry and hematology, blood products and fluids, outputs such as urine and stool. Measurements are documented only when clinically indicated, yielding patientspecific, irregular sampling.

• Textual Data

All free-text notes (e.g. progress, nursing, radiology) linked to the same patients are retained in raw form together with their original timestamps. These notes provide asynchronous narrative context that can precede, coincide with, or follow numerical measurements.

This protocol produces a multimodal dataset—irregular physiological signals paired with temporally aligned clinical narratives—tailored for longitudinal modelling tasks.

D.4 Operational Window Sampling (Constraint-Based)

Description This form of irregularity occurs when observations are limited to predefined operational hours, such as trading windows on financial markets. No data is collected during weekends, holidays, or outside business hours, resulting in structured temporal gaps.

Source FNSPID: Financial News & Stock Prices [5].

Entity Each entity corresponds to an individual publicly traded company. We first filter out companies that lack recorded time stamps or related articles, and then randomly sample 10 different companies.

Collection

• Numerical Time Series: We download daily closing prices and related trading statistics for each selected company from https://huggingface.co/datasets/Zihan1004/ FNSPID. Only trading days are included, with no entries for weekends or market holidays, leading to regularly spaced but discontinuous sampling.
• Textual Data: For each trading day, we collect news headlines and article excerpts related to the corresponding company from https://huggingface.co/datasets/Zihan1004/ FNSPID. Articles are time-stamped and filtered for relevance. We then use GPT-4.1 Nano to summarize the article into concise 5-sentence narratives, highlighting market developments and company-specific events.

D.5 Resource-Aware Collection (Constraint-Based)

Description This irregularity arises when data is recorded only while compute resources are active. In this setting, telemetry is missing not due to failure or noise, but because the monitoring system is deliberately inactive during idle periods.

Source Alibaba GPU Cluster Trace v2020 [20]. Main repository: https://github.com/ alibaba/clusterdata/tree/master?tab=readme-ov-file

Entity Each entity corresponds to a distinct physical GPU machine in the Alibaba cluster. We select 3 machines from the dataset, identified by their anonymized worker_name, to represent heterogeneous task schedules and resource usage patterns.

Collection

• Numerical Time Series: We process eight weeks of Alibaba GPU cluster logs from July–August 2020. For each selected machine, we generate a time series where measurements are sampled only during active periods (i.e., when at least one container is running). At each event timestamp, we log:
  • The number of concurrent instances, tasks, and jobs
  • Eight sensor metrics aggregated across all currently active containers

The result is an 11-dimensional time series with naturally irregular intervals, reflecting operational constraints and runtime dynamics.

• Textual Data: For every container start event, we extract the corresponding job_name, task_name, gpu_type_spec, and workload description from the pai_group_tag_table.csv. These fields are treated as an asynchronous textual context. We aggregate all textual entries within a 6-hour window and use GPT-4.1 Nano to summarize them into concise narratives of no more than 5 sentences, capturing key workload patterns and system activity during that period.

D.6 Human Scheduling / Availability (Constraint-Based)

Description This form of irregularity occurs when data collection is driven by human activity patterns or availability—such as sensors only logging when users are awake or interacting with their devices.

Source StudentLife study (College Experience Dataset) [21].

Entity Each entity corresponds to an individual participant in the study. We first filter out students with less than 1000 records, then randomly select 20 students from the full cohort to represent a diverse range of behavioral patterns and schedules.

Collection

• Numerical Time Series: We use smartphone sensor logs—including activity durations on e.g., bike, foot, sleep—recorded at naturally irregular intervals driven by device usage, battery state, and user activity.
• Textual Data: We extract self-reported survey responses (e.g., anxiety, depression, stress level), each time-stamped by the participant's device. These serve as asynchronously generated text signals that reflect personal context, mood, or behavior at different points in time.

D.7 Missing Data / Gaps (Artifact-Based)

Description This type of irregularity results from data that is expected but not recorded, often due to reporting failures, technical issues, or data loss. The result is structured gaps in the time series.

Source ILINet influenza surveillance (CDC) [9].

Entity This dataset contains a single entity: the United States as a national-level reporting unit. All observations are aggregated at the country level.

Collection

• Numerical Time Series: We download weekly counts of influenza-like illness (ILI) cases as reported by the CDC through ILINet. When reporting is incomplete or unavailable for a given week, we preserve these as missing values (NaNs) to reflect true observational gaps.
• Textual Data: We include weekly CDC public health summaries and provider notes curated from the report and search result collection process introduced by Time-MMD [9]. These documents are treated as asynchronous textual observations and provide contextual insight into public health events, seasonal flu trends, and potential reporting disruptions.

D.8 Scheduling Jitter / Delay (Artifact-Based)

Description This irregularity arises when data points are recorded at uneven intervals due to system delays, congestion, or throttling. Instead of arriving on a regular schedule, observations appear with jitter caused by internal scheduling constraints or resource contention.

Source CESNET network flow data [22].

Entity Each entity corresponds to a distinct networked device in the infrastructure. We include 11 entities in total, spanning various device types such as IP endpoints, firewalls, routers, servers, and switches.

Collection

• Numerical Time Series: We ingest flow-level metrics such as byte and packet counts using real-world arrival timestamps. These metrics naturally exhibit jitter due to internal logging delays, variable sampling rates, and system load effects.

• Textual Data: We collect network device logs that include maintenance messages, traffic control alerts, and system warnings. Each log entry is treated as an independent, asynchronously occurring textual observation, preserving its original timestamp without forcing alignment to numeric data.

D.9 Multi-Source Asynchrony (Artifact-Based)

Description This type of irregularity arises when multiple data streams—such as sensors—operate with different clocks, sampling rates, or synchronization policies. These asynchronous signals pose challenges for fusion and temporal alignment, especially when modalities are collected independently.

Source EPA Outdoor Air Quality Sensors [23]. Numerical time series data was downloaded from: https://aqs.epa.gov/aqsweb/airdata/download\_files.html

Textual news articles were retrieved from Common Crawl News, using monthly archives available at: https://data.commoncrawl.org/crawl-data/CC-NEWS/

Entity Each entity corresponds to a U.S. county monitored by the EPA sensor network. We select 8 counties for inclusion in this dataset:

• Los Angeles, CA
• Maricopa (Phoenix), AZ
• Philadelphia, PA
• Bexar (San Antonio), TX
• Dallas, TX
• Richmond, VA
• Hillsborough (Tampa), FL
• Denver, CO

Collection

• Numerical Time Series: We extract four environmental variables from the EPA's air monitoring dataset: AQI, Ozone, PM2.5, and Temperature. Each feature is recorded at its native sensor-specific frequency, resulting in asynchronous observations across features and counties.
• Textual Data: To provide contextual background for environmental conditions, we collect news articles from Common Crawl spanning January to October 2024. Articles are filtered to include only those that mention both the county name and the keyword "weather." Relevant articles are summarized using GPT-4.1 Nano into concise 5-sentence narratives, highlighting key environmental developments or disruptions affecting the region.

E Prompt Designed for LLM Summarization

This appendix provides the prompt templates used for GPT-4.1 Nano to generate textual summaries for each dataset in Time-IMM. All prompts are designed to produce concise, factual, and context-aware outputs within a 5-sentence limit.

E.1 GDELT (Event-Based Logging)

PROMPTS = """
You are an expert in { domain }.
This task is part of building the GDELT 2.0 Event dataset .
Below is a news article :
{ article }

Write a concise summary of the information related to { domain } , using no more than 5 sentences . If the article is not relevant to { domain } , reply with exactly : NA """

E.2 RepoHealth (Adaptive or Reactive Sampling)

PROMPTS = """

The following are comments from multiple GitHub issue threads in the repository '{ repo_name } ' ,

including all historical comments accumulated up to and including those made on a specific day .

{ comments }

Write a concise summary of the main technical concerns , questions , or suggestions raised across these discussions . Focus on key issues , proposed solutions ,

and unresolved questions . Your summary should be no more than 5 sentences .

"""

E.3 FNSPID (Operational Window Sampling)

PROMPTS = """

You are an expert in financial markets .

This task is part of building the stock price prediction dataset .

Below is a news article : { article }

Write a concise summary of the information related to financial markets , using no more than 5 sentences . If the article is not relevant to financial markets , reply with

exactly : NA """

E.4 ClusterTrace (Resource-Aware Collection)

PROMPTS = """

You are an expert in large - scale distributed computing and cloud infrastructure .

This task is part of building the Alibaba Cluster Trace Program dataset .

Below is a collection of logs and system behavior summaries : { article }

Write a concise summary of the system behavior and operational insights relevant to distributed computing , using no more than 5 sentences .

If the content is not relevant to the dataset , or the logs / summaries are not provided , reply with exactly : NA """

E.5 EPA-Air (Multi-Source Asynchrony)

PROMPTS = """

You are an expert in weather .

This task is part of building the Air Quality Data Collected at Outdoor Monitors Across the US dataset .

Below is a news article :
{ article }
Write a concise summary of the information related to weather , using
   no more than 5 sentences .
If the article is not relevant to weather , reply with exactly : NA
"""

F Showcase of Raw and Summarized Text

This appendix illustrates how raw textual inputs are preprocessed into concise, informative summaries using LLMs (GPT-4.1 Nano). Below is an example taken from the EPA-Air dataset, where the input news article contains broader event details, but the summarization process emphasizes weather conditions relevant to environmental sensing.

EPA-Air Example

Raw News Excerpt:

Seven migrant smuggling suspects have been arrested and 11 migrants were hospitalized after a law enforcement sting Thursday near San Antonio, authorities said. . . .

Temperatures in San Antonio were in the high 90s Thursday afternoon and were expected to top 100, according to the National Weather Service.

. . . The incident came two days after President Joe Biden unveiled plans to enact immediate significant restrictions on migrants seeking asylum . . .

San Antonio was the site of the nation's deadliest human smuggling episode in June 2022 . . .

Summarized Version:

On Thursday in San Antonio, temperatures reached the high 90s and were expected to exceed 100 degrees Fahrenheit. These hot conditions contributed to heat-related injuries among migrants found in a trailer during a law enforcement operation. The extreme heat poses a significant risk for dehydration, heat exhaustion, and other heat-related health issues. Past incidents, such as the deadly smuggling episode in June 2022, also occurred under similar sweltering conditions. The current weather highlights the dangers faced by individuals exposed to high temperatures in the area.

G Additional Ethical Considerations and Data Access

Data Sources and Privacy. All datasets in Time-IMM are derived from publicly available sources with appropriate access permissions. For sensitive domains such as healthcare (e.g., MIMIC-IV), we take additional precautions to ensure data privacy and regulatory compliance. MIMIC-IV is a restricted-access resource hosted on https://physionet.org/content/mimiciv/3.1/. To access the raw files, users must be credentialed, complete the CITI "Data or Specimens Only Research" training, and sign the official data use agreement on PhysioNet. In Time-IMM, we do not redistribute raw data from MIMIC-IV. Instead, we provide preprocessing scripts and instructions to enable authorized users to reproduce our setup. We strictly follow access policies and do not feed MIMIC-IV clinical notes into closed-source LLMs (e.g., GPT-4.1 Nano). This ensures alignment with privacy safeguards and ethical standards for clinical data handling.

Responsible Redistribution. To comply with upstream licenses and ethical research norms, we do not redistribute raw web-crawled text, GitHub comments, or clinical notes where direct sharing is restricted. Instead, we offer derived summaries, structured metadata, and links or instructions for dataset reconstruction from original public sources.

Figure 7: Overview of the canonical pre-alignment process.

Open Access and Licensing. Time-IMM is released under a Creative Commons Attribution 4.0 (CC BY 4.0) license, which permits broad use, redistribution, and adaptation with appropriate citation. Scripts and benchmarks are openly available to ensure reproducibility, transparency, and extensibility by the research community.

Hosting and Maintenance Plan. All datasets and code are hosted on publicly accessible and stable platforms. The dataset is maintained on Kaggle to facilitate broad usability, while the benchmark library is versioned and published through a persistent code repository. We commit to periodically reviewing contributions and issues reported by the community to ensure continued compatibility and correctness.

Accessing. The Time-IMM dataset can be accessed at: https://www.kaggle.com/datasets/ blacksnail789521/time-imm/data

The accompanying benchmark code and processing tools are available at: https://anonymous. 4open.science/r/IMMTSF\_NeurIPS2025

Ethical Usage Encouragement. We strongly encourage responsible usage of Time-IMM and IMM-TSF in research contexts that prioritize fairness, transparency, and real-world safety. Applications involving downstream decision-making—especially in domains such as finance or healthcare—should incorporate appropriate validation and interpretability safeguards.

H Adapting Regular Time Series Models via Canonical Pre-Alignment

To adapt regular time series forecasting models to irregular data, we follow the canonical prealignment strategy introduced in t-PatchGNN [10], as illustrated in Figure 7. This method transforms irregularly sampled time series into a fixed-length, regularly spaced representation that can be consumed by standard forecasting architectures.

Let X = {[(t (n) i , x (n) i )]^{Pn i}=1} N ⁿ=1 denote the set of past irregular time series observations, where N is the number of variables and Pⁿ is the number of past observations for variable n. Each observation x (n) occurs at time t (n) ⁱ ∈ [Tstart, Tcut]. We also define a set of future query times Q = {[q (n) k F^{n k}=1} N ⁿ=1, with each query q (n) ^k ∈ (Tcut, Tend].

Step 1: Determine Global Temporal Resolution. We first scan the entire dataset to determine the global maximum number of past input timestamps and future query timestamps across all training windows. This defines the fixed-length resolution L for both input and output time series, covering the combined past and forecast windows.

Step 2: Temporal Grid Construction, Alignment, and Padding. For each example, we create a unified, chronologically sorted list of timestamps {t˜l} L ^l=1 that covers both the observation period

and the forecast horizon. We construct a matrix X˜ ∈ R ^L×^N , where each entry x˜ (n) l is the observed value of variable n at time t˜^l , or zero if no observation exists. A binary mask matrix M ∈ {0, 1} L×N is used to indicate whether a value was observed (1) or imputed (0). Both X˜ and M are padded as necessary to reach the globally fixed length L, ensuring uniform tensor shapes across all examples.

Step 3: Add Input Features: Timestamp and Mask. To help the model reason about timing and missingness, we expand the feature dimension from N to 2N + 1. Each input row includes: (i) the observed values or zeros, (ii) the binary mask, and (iii) the normalized timestamp t˜^l . This preserves all temporal and structural information in a dense, aligned format.

Step 4: Add Query Features. The forecast queries Q are appended to the end of the input sequence using the same 2N + 1 feature schema. Each query timestamp q (n) k is represented by a feature row that includes: (i) zero values for all variable entries (since ground truth is unknown at inference time), (ii) a binary mask set to 0 (indicating no observation—not a forecast target), and (iii) the normalized query timestamp. This mask entry should not be confused with the evaluation-time forecast mask used to compute prediction errors. Including query timestamps in the input allows the model to explicitly condition on the temporal locations of future targets.

Summary. This pre-alignment pipeline transforms irregular, variable-length time series into padded, fixed-length tensors that include time, observation, and mask features. By appending forecast queries and exposing the full temporal grid, regular forecasting models can be repurposed to handle irregular time series in a structured and query-aware manner.

I Formal Definitions of TTF and MMF Modules

The detailed architectures of the TTF and MMF modules are illustrated in Figures 8 and 9, respectively. We now present their formal definitions below.

I.1 Timestamp-to-Text Fusion (TTF)

Let S = {(τ^j , s^j )} L^{S j}=1 denote the sequence of past text observations, where s^j is a text snippet observed at timestamp τ^j ∈ [Tstart, Tcut]. Let v^j ∈ R ^d be the embedding of s^j produced by a fixed language model. Let Q = {tk} T^{f k}=1 denote the future forecast query times.

Recency-Weighted Averaging (RecAvg). This variant computes a time-weighted average of all previous embeddings using a Gaussian kernel centered at the forecast time:

$$\alpha_{jk} = \exp\left(-\left(\frac{t_k - \tau_j}{\sigma}\right)^2\right), \quad \text{for } \tau_j \le t_k$$ (6)

$$ek = \frac{\sum{j=1}^{LS} \alpha{jk} vj}{\sum{j=1}^{LS} \alpha{jk}} \in \mathbb{R}^d$$ (7)

Time2Vec-Augmented Cross-Attention (T2V-XAttn). Each embedding is augmented with a Time2Vec representation of its timestamp:

$$\phi(\tau_j) = [\omega_0 \tau_j + b_0, \sin(\omega_1 \tau_j + b1), \dots, \sin(\omega{d_{\tau} - 1} \tauj + b{d_{\tau} - 1})]$$ (8)

$$\tilde{v}_j = \llbracket v_j; \phi(\tauj) \rrbracket \in \mathbb{R}^{d+d\tau} \tag{9}$$

Let W^q ∈ R ^dq×¹ be a learnable forecast query vector. Then the aligned representation at time t^k is computed by soft attention:

Figure 8: Detailed architecture of TTF modules.

$$a_{jk} = \frac{\exp\left((W_q^{\top} \tilde{v}j)\right)}{\sum{\tau_{j'} \le t_k} \exp\left((Wq^{\top} \tilde{v}{j'})\right)}$$ (10)

$$ek = \sum{\tau_j \le tk} a{jk} \cdot \tilde{v}_j \tag{11}$$

I.2 Multimodality Fusion (MMF)

Let Y ts ∈ R ^{Tf ×N} denote the numerical forecast sequence, and let E = {ek} T^{f k}=1 ^{∈ R Tf ×d} be the aligned text representations from TTF.

GRU-Gated Residual Addition (GR-Add). Let H ∈ R ^{Tf ×h} be the hidden state sequence from a GRU applied over the concatenated input:

$$z_k = [y_k^{\text{ts}}; e_k] \in \mathbb{R}^{N+d} \tag{12}$$

$$H = \operatorname{GRU}({zk}{k=1}^{T_f}) \tag{13}$$

$$\Delta Y = W{\Delta}H + b{\Delta} \tag{14}$$

$$G = \sigma(W_g Z + b_g)$$ where $Z = [Y^{ts}; E]$ (15)

$$Y^{\text{fused}} = G \odot Y^{\text{ts}} + (1 - G) \odot (Y^{\text{ts}} + \Delta Y) \tag{16}$$

where W^∆ ∈ R N×h , W^g ∈ R N×(N+d) , and ⊙ denotes element-wise multiplication.

Cross-Attention Addition (XAttn-Add). Let Q = Y tsWQ, K = EWK, and V = EW^V with learnable projection matrices WQ, WK, W^V . The residual is derived via scaled dot-product attention:

Figure 9: Detailed architecture of MMF modules.

$$A = \operatorname{softmax}\left(\frac{QK^{\top}}{\sqrt{d}}\right) \cdot V \tag{17}$$

$$\Delta Y = AW{\text{res}} + b{\text{res}} \tag{18}$$

$$Y^{\text{fused}} = \frac{Y^{\text{ts}} + \kappa \cdot \Delta Y}{1 + \kappa} \tag{19}$$

Here, κ ∈ R is a scalar hyperparameter, and all terms are defined over the entire sequence of query times.

J Dataset-Specific Forecasting Configuration

Each dataset in Time-IMM is configured with a dataset-specific forecasting setup that reflects its native timestamp distribution and sampling behavior. For each entity, we apply a sliding window strategy with fixed-length context and query segments. The context window corresponds to past observations X ∈ [Tstart, Tcut], while the query horizon defines the future timestamps Q ∈ (Tcut, Tend] at which the model is expected to make predictions. Durations are defined using dataset-native time units (e.g., hours, days, or weeks), ensuring semantic consistency across domains.

K Model-Specific and Fusion Module Configuration

K.1 Model-Specific Configuration

Each baseline model is configured using recommended default hyperparameters from its original implementation. The following settings summarize key architecture and training parameters for each model class:

• Informer [6]: 2 encoder layers, 1 decoder layer, attention factor 3.
• DLinear [31]: Default configuration without modification.

Table 2: Dataset-specific forecasting parameters: context window (past input) and query horizon (forecast target).

Dataset	Context Window Query Horizon
GDELT	14 days	14 days
RepoHealth	31 days	31 days
MIMIC	24 hours	24 hours
FNSPID	31 days	31 days
ClusterTrace	12 hours	12 hours
StudentLife	31 days	31 days
ILINet	4 weeks	4 weeks
CESNET	7 days	7 days
EPA-Air	7 days	7 days

• PatchTST [32]: 1 encoder layer, 1 decoder layer, 2 attention heads.
• TimesNet [33]: 2 encoder layers, 1 decoder layer, top-k = 5, dmodel = 16, dff = 32.
• TimeMixer [34]: 2 encoder layers, dmodel = 16, dff = 32, with 3 downsampling layers (window size 2, average pooling).
• TimeLLM [35]: GPT2 backbone with 6 layers, input/output token lengths 16 and 96, respectively; dmodel = 32, dff = 128.
• TTM [36]: 3-level attention patching, 3 encoder and 2 decoder layers; dmodel = 1024, decoder dimension 64.
• CRU [11]: Latent state and hidden units set to 32, with square/exponential activations and gravity gates enabled.
• Latent-ODE [3]: Recognition GRU with 32 hidden units, 1 RNN and 1 generator layer, ODE units set to 32.
• Neural Flow [12]: Latent dimension 20, 3-layer hidden network (dimension 32), GRU units 32, coupling flow with 2 layers, time network: TimeLinear.
• tPatchGNN [10]: Patch size 24, 1 Transformer layer, 1 GNN layer, 10-dimensional time and node embeddings, dhidden = 32.

K.2 Fusion Module Configuration

In multimodal experiments, we integrate textual context into the forecasting pipeline using specialized fusion modules. Each model uses the best-performing fusion configuration selected via validation performance.

Timestamp-to-Text Fusion (TTF).

• RecAvg: Recency-weighted averaging of past text embeddings using a Gaussian decay function based on timestamp proximity. The maximum weight is assigned when the text timestamp exactly matches the forecast time. We set the standard deviation parameter to σ = 1.0.
• T2V-XAttn: Each text embedding is augmented with a Time2Vec encoding of its timestamp and fused via a single-layer, single-head attention mechanism.

Multimodality Fusion (MMF).

• GR-Add: A residual correction is computed from the concatenated numerical forecast and text embedding using a GRU followed by an MLP. The output is blended with the base forecast through a learned gating function.

Figure 10: Performance comparison between frozen LLM-based text encoders and Doc2Vec, averaged across all 11 baselines for each dataset.

• XAttn-Add: Numerical forecasts attend to aligned text representations using a single-layer, single-head attention module. The result is scaled by a fixed convex mixing weight κ = 0.5 and added to the base forecast.

Text Encoder. We utilize four kinds of pretrained large language models as text encoders—GPT-2, BERT-base, LLaMA 3.1 (8B), and DeepSeek (7B)—each used with its full transformer stack and no truncation of layers. BERT-base supports a maximum context length of 512 tokens, while GPT-2, LLaMA 3.1, and DeepSeek allow longer sequences but are forcibly truncated to 1024 tokens in our setup. To ensure consistent dimensionality, we project all encoder outputs to a shared 768-dimensional space using a learned linear transformation before fusion.

L More Experimental Results

This appendix contains two sets of additional results that further validate the effectiveness of multimodal modeling in irregular time series forecasting.

• Section L.1 investigates the role of the text encoder by replacing large language models (LLMs) with a classical alternative—Doc2Vec [75]—and comparing the resulting performance.
• Section L.2 reports per-dataset comparisons between unimodal (time series only) and multimodal (time series + text) model variants across all nine datasets in Time-IMM.

L.1 Ablation: Replacing LLMs with Doc2Vec

To evaluate the importance of using pretrained language models, we conduct an ablation study replacing frozen LLM-based text encoders with a train-from-scratch alternative—Doc2Vec. Doc2Vec learns fixed-length embeddings for each text input by training a shallow neural model directly on the textual corpus of each dataset, without relying on large-scale pretraining.

Figure 10 summarizes performance across all datasets. We observe that models using frozen LLMs consistently outperform those using Doc2Vec, confirming the benefit of pretrained semantic representations—even in the presence of asynchronous and domain-specific text. These results highlight the importance of strong general-purpose encoders for effective multimodal forecasting under irregular sampling.

L.2 Multimodal vs. Unimodal Performance

To quantify the contribution of textual signals under irregular sampling, we report per-dataset comparisons between unimodal (time series only) and multimodal (time series + text) model variants. Each result reflects the best-performing fusion configuration selected via validation on the corresponding dataset. Performance is measured using mean squared error (MSE), and improvements are reported as relative percentage change from the unimodal baseline.

Multimodal models consistently outperform their unimodal counterparts across all datasets, with especially large improvements observed in settings where text provides rich, temporally relevant context. The following tables summarize detailed results for each dataset:

• GDELT (Event-Based Logging) Table 3
• RepoHealth (Adaptive Sampling) Table 4
• MIMIC (Human-Initiated Observation) Table 5
• FNSPID (Operational Window Sampling) Table 6
• ClusterTrace (Resource-Aware Collection) Table 7
• StudentLife (Human Scheduling) Table 8
• ILINet (Missing Data) Table 9
• CESNET (Scheduling Jitter) Table 10
• EPA-Air (Multi-Source Asynchrony) Table 11

Table 3: Unimodal vs. multimodal forecasting results on the GDELT dataset.

Model	Modal	MSE	MAE
Informer	Uni	1.0448	0.6935
	Multi	1.0295	0.6897
DLinear	Uni	1.0482	0.6943
	Multi	1.0487	0.6945
PatchTST	Uni	1.0670	0.7051
	Multi	1.0511	0.6944
TimesNet	Uni	1.0251	0.6846
	Multi	1.0250	0.6916
TimeMixer	Uni	1.0471	0.6842
	Multi	1.0333	0.6973
TimeLLM	Uni	1.0411	0.6923
	Multi	1.0402	0.6894
TTM	Uni	1.0400	0.6948
	Multi	1.0238	0.6893
CRU	Uni	1.0282	0.6968
	Multi	1.0267	0.6912
LatentODE	Uni	1.0271	0.6886
	Multi	1.0268	0.6856
NeuralFlow	Uni	1.0265	0.6942
	Multi	1.0260	0.6958
tPatchGNN	Uni	1.0267	0.6893
	Multi	1.0265	0.6958

Table 4: Unimodal vs. multimodal forecasting results on the RepoHealth dataset.

Model	Modal	MSE	MAE
Informer	Uni	0.6391	0.4143
	Multi	0.6296	0.4037
DLinear	Uni	0.5376	0.4081
	Multi	0.5291	0.4021
PatchTST	Uni	0.5727	0.4185
	Multi	0.5520	0.4180
TimesNet	Uni	0.5415	0.4158
	Multi	0.5599	0.4275
TimeMixer	Uni	0.7267	0.4321
	Multi	0.6890	0.4230
TimeLLM	Uni	0.5655	0.3994
	Multi	0.5601	0.3919
TTM	Uni	0.5442	0.4088
	Multi	0.5386	0.4079
CRU	Uni	1.0362	0.8030
	Multi	0.8538	0.6596
LatentODE	Uni	0.7555	0.6104
	Multi	0.7341	0.6285
NeuralFlow	Uni	0.8231	0.6736
	Multi	0.7720	0.6409
tPatchGNN	Uni	0.6196	0.5000
	Multi	0.6090	0.4718

Table 5: Unimodal vs. multimodal forecasting results on the MIMIC dataset.

Model	Modal	MSE	MAE
Informer	Uni	0.8519	0.5588
	Multi	0.7988	0.6083
DLinear	Uni	0.8770	0.6152
	Multi	0.8515	0.6456
PatchTST	Uni	0.9394	0.6794
	Multi	0.9281	0.6799
TimesNet	Uni	1.0374	0.7348
	Multi	0.9335	0.6821
TimeMixer	Uni	0.8856	0.5686
	Multi	0.8179	0.5821
TimeLLM	Uni	0.7901	0.5457
	Multi	0.7535	0.5620
TTM	Uni	0.8620	0.6283
	Multi	0.9105	0.6707
CRU	Uni	1.0122	0.7235
	Multi	0.9847	0.7087
LatentODE	Uni	0.9325	0.6868
	Multi	0.9423	0.6785
NeuralFlow	Uni	1.0086	0.7219
	Multi	0.9884	0.7079
tPatchGNN	Uni	0.7250	0.5479
	Multi	0.7656	0.5794

Table 6: Unimodal vs. multimodal forecasting results on the FNSPID dataset.

Model	Modal	MSE	MAE
Informer	Uni	0.1277	0.2153
	Multi	0.1282	0.2149
DLinear	Uni	0.1239	0.2121
	Multi	0.1248	0.2145
PatchTST	Uni	0.1301	0.2240
	Multi	0.1277	0.2212
TimesNet	Uni	0.1253	0.2144
	Multi	0.1239	0.2128
TimeMixer	Uni	0.1161	0.1965
	Multi	0.1157	0.1964
TimeLLM	Uni	0.1261	0.2138
	Multi	0.1240	0.2124
TTM	Uni	0.1355	0.2373
	Multi	0.1335	0.2312
CRU	Uni	0.1484	0.2545
	Multi	0.1476	0.2471
LatentODE	Uni	0.1203	0.2134
	Multi	0.1157	0.2051
NeuralFlow	Uni	0.1510	0.2572
	Multi	0.1420	0.2470
tPatchGNN	Uni	0.1247	0.2195
	Multi	0.1212	0.2132

Table 7: Unimodal vs. multimodal forecasting results on the ClusterTrace dataset.

Model	Modal	MSE	MAE
Informer	Uni	1.0793	0.8039
	Multi	0.7473	0.6527
DLinear	Uni	0.8418	0.6765
	Multi	0.6440	0.6504
PatchTST	Uni	2.0366	0.9896
	Multi	1.0370	0.8389
TimesNet	Uni	0.9739	0.7187
	Multi	0.6706	0.6579
TimeMixer	Uni	1.1262	0.7861
	Multi	0.7377	0.6946
TimeLLM	Uni	0.9596	0.7325
	Multi	0.7250	0.6973
TTM	Uni	1.0030	0.7409
	Multi	0.7166	0.6813
CRU	Uni	1.2111	0.8287
	Multi	0.5753	0.5988
LatentODE	Uni	1.1613	0.7818
	Multi	0.5336	0.5780
NeuralFlow	Uni	1.4458	0.8910
	Multi	0.6248	0.6384
tPatchGNN	Uni	0.9671	0.7119
	Multi	0.6105	0.6104

Table 8: Unimodal vs. multimodal forecasting results on the StudentLife dataset.

Model	Modal	MSE	MAE
Informer	Uni	0.9432	0.6674
	Multi	0.8910	0.6790
DLinear	Uni	0.8850	0.6760
	Multi	0.8733	0.6629
PatchTST	Uni	0.8964	0.6793
	Multi	0.8764	0.6814
TimesNet	Uni	0.9069	0.6827
	Multi	0.8892	0.6889
TimeMixer	Uni	0.9437	0.6720
	Multi	0.8867	0.6768
TimeLLM	Uni	0.9355	0.6797
	Multi	0.8870	0.6781
TTM	Uni	0.9024	0.6830
	Multi	0.8888	0.6838
CRU	Uni	0.9101	0.6876
	Multi	0.9101	0.6890
LatentODE	Uni	0.9028	0.6965
	Multi	0.9040	0.6907
NeuralFlow	Uni	0.9237	0.6995
	Multi	0.9150	0.7024
tPatchGNN	Uni	0.8661	0.6672
	Multi	0.8633	0.6705

Table 9: Unimodal vs. multimodal forecasting results on the ILINet dataset.

Model	Modal	MSE	MAE
Informer	Uni	1.2714	0.8118
	Multi	1.2338	0.7832
DLinear	Uni	1.0402	0.6811
	Multi	0.9795	0.6374
PatchTST	Uni	1.1606	0.7532
	Multi	1.1727	0.7373
TimesNet	Uni	1.0518	0.6670
	Multi	1.0510	0.6436
TimeMixer	Uni	0.9825	0.6456
	Multi	1.0530	0.6845
TimeLLM	Uni	1.1243	0.7255
	Multi	1.0333	0.6901
TTM	Uni	1.1010	0.7003
	Multi	1.0383	0.6898
CRU	Uni	0.9718	0.6652
	Multi	1.0423	0.6900
LatentODE	Uni	1.1512	0.7531
	Multi	1.1098	0.7276
NeuralFlow	Uni	1.1946	0.7688
	Multi	1.1501	0.7460
tPatchGNN	Uni	1.6163	0.9255
	Multi	1.4877	0.8689

Table 10: Unimodal vs. multimodal forecasting results on the CESNET dataset.

Model	Modal	MSE	MAE
Informer	Uni	0.9591	0.7559
	Multi	0.8754	0.7233
DLinear	Uni	0.9695	0.7517
	Multi	0.8763	0.7221
PatchTST	Uni	1.3177	0.8797
	Multi	1.1242	0.8192
TimesNet	Uni	0.9581	0.7499
	Multi	0.8686	0.7201
TimeMixer	Uni	0.9500	0.7481
	Multi	0.8588	0.7160
TimeLLM	Uni	0.9891	0.7676
	Multi	0.9125	0.7364
TTM	Uni	0.9869	0.7653
	Multi	0.9205	0.7422
CRU	Uni	0.9462	0.7559
	Multi	0.8726	0.7257
LatentODE	Uni	0.9655	0.7642
	Multi	0.8977	0.7400
NeuralFlow	Uni	1.0154	0.7915
	Multi	0.9495	0.7685
tPatchGNN	Uni	0.9796	0.7713
	Multi	0.9122	0.7483

Table 11: Unimodal vs. multimodal forecasting results on the EPA-Air dataset.

Model	Modal	MSE	MAE
Informer	Uni	0.6301	0.5983
	Multi	0.5812	0.5728
DLinear	Uni	0.5361	0.5279
	Multi	0.5223	0.5223
PatchTST	Uni	0.6196	0.5947
	Multi	0.6204	0.5992
TimesNet	Uni	0.5599	0.5524
	Multi	0.5892	0.5650
TimeMixer	Uni	0.6086	0.5770
	Multi	0.5641	0.5663
TimeLLM	Uni	0.5835	0.5643
	Multi	0.5334	0.5303
TTM	Uni	0.6002	0.5653
	Multi	0.6218	0.5761
CRU	Uni	0.7026	0.6306
	Multi	0.7982	0.6739
LatentODE	Uni	0.8025	0.6665
	Multi	0.7556	0.6523
NeuralFlow	Uni	0.7821	0.6488
	Multi	0.8202	0.6790
tPatchGNN	Uni	0.6258	0.6022
	Multi	0.5840	0.5793

References

• [1] Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rnns: Improved classification of clinical time series. In Machine learning for healthcare conference, pages 253–270. PMLR, 2016.
• [2] Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. Scalable and accurate deep learning with electronic health records. NPJ digital medicine, 1(1):18, 2018.
• [3] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
• [4] Ching Chang, Wei-Yao Wang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Aligning pre-trained llms as data-efficient time-series forecasters. ACM Trans. Intell. Syst. Technol., 16(3), April 2025.
• [5] Zihan Dong, Xinyu Fan, and Zhiyuan Peng. Fnspid: A comprehensive financial news dataset in time series. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4918–4927, 2024.
• [6] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11106–11115, 2021.
• [7] Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML. The ucr time series classification archive, October 2018. https://www.cs.ucr.edu/~eamonn/time\_series\_ data\_2018/.
• [8] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of forecasting, 34(4):802– 808, 2018.
• [9] Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Prabhakar Kamarthi, Aditya Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, et al. Time-mmd: Multi-domain multimodal dataset for time series analysis. Advances in Neural Information Processing Systems, 37:77888–77933, 2024.
• [10] Weijia Zhang, Chenlong Yin, Hao Liu, Xiaofang Zhou, and Hui Xiong. Irregular multivariate time series forecasting: A transformable patching graph neural networks approach. In Forty-first International Conference on Machine Learning, 2024.
• [11] Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In International conference on machine learning, pages 19388–19405. PMLR, 2022.
• [12] Marin Biloš, Johanna Sommer, Syama Sundar Rangapuram, Tim Januschowski, and Stephan Günnemann. Neural flows: Efficient alternative to neural odes. Advances in neural information processing systems, 34:21325–21337, 2021.
• [13] Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021.
• [14] Kai Kim, Howard Tsai, Rajat Sen, Abhimanyu Das, Zihao Zhou, Abhishek Tanpure, Mathew Luo, and Rose Yu. Multi-modal forecaster: Jointly predicting time series and textual data. arXiv preprint arXiv:2411.06735, 2024.

• [15] Jialin Chen, Aosong Feng, Ziyu Zhao, Juan Garza, Gaukhar Nurbek, Cheng Qin, Ali Maatouk, Leandros Tassiulas, Yifeng Gao, and Rex Ying. Mtbench: A multimodal time series benchmark for temporal reasoning and question answering. arXiv preprint arXiv:2503.16858, 2025.

• [16] Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, and Jianxin Liao. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12694–12702, 2025.

• [17] Kalev H Leetaru and Philip A Schrodt. Gdelt: Global data on events, location, and tone, 1979–2012. ISA annual convention, 2(4):1–49, 2013.

• [18] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pages 49–55, 2020.

• [19] Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-ivnote: Deidentified free-text clinical notes (version 2.2). physionet, 2023.

• [20] Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. MLaaS in the wild: Workload analysis and scheduling in large-scale heterogeneous GPU clusters. In 19th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 22), 2022.

• [21] Subigya Nepal, Wenjun Liu, Arvind Pillai, Weichen Wang, Vlado Vojdanovski, Jeremy F Huckins, Courtney Rogers, Meghan L Meyer, and Andrew T Campbell. Capturing the college experience: a four-year mobile sensing study of mental health, resilience and behavior of college students during the pandemic. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 8(1):1–37, 2024.

• [22] Josef Koumar, Karel Hynek, Tomáš Cejka, and Pavel Šiška. Cesnet-timeseries24: Time series ˇ dataset for network traffic anomaly detection and forecasting. Scientific Data, 12(1):338, 2025.

• [23] U.S. Environmental Protection Agency. Air quality data. https://www.epa.gov/ air-quality, n.d. Accessed: 2025-04-15.

• [24] OpenAI. Gpt-4.1 nano. https://platform.openai.com/docs/models/gpt-4.1-nano, 2025. Accessed: 2025-04-15.

• [25] Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving medical predictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning, pages 41300–41313. PMLR, 2023.

• [26] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.

• [27] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.

• [28] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024.

• [29] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, and et al. Deepseek llm: Scaling open-source language models with longtermism. CoRR, abs/2401.02954, 2024.

• [30] Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321, 2019.

• [31] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121–11128, 2023.

• [32] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2022.

• [33] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023.

• [34] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.

• [35] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023.

• [36] Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam Nguyen, Wesley M Gifford, Chandra Reddy, and Jayant Kalagnanam. Tiny time mixers (ttms): Fast pre-trained models for enhanced zero/few-shot forecasting of multivariate time series. Advances in Neural Information Processing Systems, 37:74147–74181, 2024.

• [37] Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark. arXiv preprint arXiv:2407.13278, 2024.

• [38] Qinghua Liu and John Paparrizos. The elephant in the room: Towards a reliable time-series anomaly detection benchmark. Advances in Neural Information Processing Systems, 37:108231– 108261, 2024.

• [39] Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S Jensen, Zhenli Sheng, et al. Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods. arXiv preprint arXiv:2403.20150, 2024.

• [40] Haotian Si, Jianhui Li, Changhua Pei, Hang Cui, Jingwen Yang, Yongqian Sun, Shenglin Zhang, Jingjing Li, Haiming Zhang, Jing Han, et al. Timeseriesbench: An industrial-grade benchmark for time series anomaly detection models. In 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE), pages 61–72. IEEE, 2024.

• [41] UCI Machine Learning Repository. Electricityloaddiagrams20112014 data set. https:// archive.ics.uci.edu/dataset/321/electricityloaddiagrams20112014, 2014. Accessed: 2025-04-15.

• [42] California Department of Transportation. Performance measurement system (pems), 2024. Accessed: 2025-04-15.

• [43] National Centers for Environmental Information. Local climatological data, 2024. Accessed: 2025-04-15.

• [44] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A, 379(2194):20200209, 2021.

• [45] Weijia Zhang, Hao Liu, Lijun Zha, Hengshu Zhu, Ji Liu, Dejing Dou, and Hui Xiong. Mugrep: A multi-task hierarchical graph representation learning framework for real estate appraisal. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 3937–3947, 2021.

• [46] Zhen-Jie Yao, Jie Bi, and Yi-Xin Chen. Applying deep learning to individual and community health monitoring data: A survey. International Journal of Automation and Computing, 15:643– 655, 2018.

• [47] Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems, 32, 2019.

• [48] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: a large-scale dataset for visual speech recognition. arXiv preprint arXiv:1809.00496, 2018.

• [49] Henintsoa S Andrianarivony and Moulay A Akhloufi. Machine learning and deep learning for wildfire spread prediction: A review. Fire, 7(12):482, 2024.

• [50] Yushan Jiang, Kanghui Ning, Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, and Dongjin Song. Multi-modal time series analysis: A tutorial and survey. arXiv preprint arXiv:2503.13709, 2025.

• [51] Xiongxiao Xu, Yue Zhao, S Yu Philip, and Kai Shu. Beyond numbers: A survey of time series analysis in the era of multimodal llms. Authorea Preprints, 2025.

• [52] Christopher Brooks, Craig Thompson, and Stephanie Teasley. A time series interaction analysis method for building predictive models of learners using log data. In Proceedings of the fifth international conference on learning analytics and knowledge, pages 126–135, 2015.

• [53] Iman Deznabi, Mohit Iyyer, and Madalina Fiterau. Predicting in-hospital mortality by combining clinical notes with time-series data. In Findings of the association for computational linguistics: ACL-IJCNLP 2021, pages 4026–4031, 2021.

• [54] Patrick T Brandt and John R Freeman. Advances in bayesian time series modeling and the study of politics: Theory testing, forecasting, and policy analysis. Political Analysis, 14(1):1–36, 2006.

• [55] Ikaro Silva, George B. Moody, Daniel J. Scott, and Roger G. Mark. Predicting mortality of icu patients: The physionet/computing in cardiology challenge 2012. https://archive. physionet.org/challenge/2012, 2012. Accessed: 2025-05-18.

• [56] Vedrana Vidulin, Mitja Lustrek, Bostjan Kaluza, Rok Piltaver, and Jana Krivec. Localization data for person activity. https://archive.ics.uci.edu/dataset/196/localization+ data+for+person+activity, 2010. Accessed: 2025-04-15.

• [57] MJ Menne, CN Williams Jr, RS Vose, and Data Files. Long-term daily climate records from stations across the contiguous united states, 2015.

• [58] Maksims Kazijevs and Manar D Samad. Deep imputation of missing values in time series health data: A review with benchmarking. Journal of biomedical informatics, 144:104440, 2023.

• [59] Jun Wang, Wenjie Du, Yiyuan Yang, Linglong Qian, Wei Cao, Keli Zhang, Wenjia Wang, Yuxuan Liang, and Qingsong Wen. Deep learning for multivariate time series imputation: A survey. arXiv preprint arXiv:2402.04059, 2024.

• [60] A Yarkın Yıldız, Emirhan Koç, and Aykut Koç. Multivariate time series imputation with transformers. IEEE Signal Processing Letters, 29:2517–2521, 2022.

• [61] Siddhartha Bhandari, Neil Bergmann, Raja Jurdak, and Branislav Kusy. Time series analysis for spatial node selection in environment monitoring sensor networks. Sensors, 18(1):11, 2017.

• [62] Ting-Yun Ou, Ching Chang, and Wen-Chih Peng. Coke: Causal discovery with chronological order and expert knowledge in high proportion of missing manufacturing data. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4803–4810, 2024.

• [63] Cheng He, Susanne Breitner-Busch, Veronika Huber, Kai Chen, Siqi Zhang, Antonio Gasparrini, Michelle Bell, Haidong Kan, Dominic Royé, Ben Armstrong, et al. Rainfall events and daily mortality across 645 global locations: two stage time series analysis. bmj, 387, 2024.

• [64] Noppawit Aiumtrakul, Charat Thongprayoon, Supawadee Suppadungsuk, Pajaree Krisanapan, Preyarat Pinthusopon, Michael A Mao, Chinnawat Arayangkool, Kristine B Vo, Chalothorn Wannaphut, Jing Miao, et al. Global trends in kidney stone awareness: a time series analysis from 2004–2023. Clinics and Practice, 14(3):915–927, 2024.

• [65] Haotian Zheng, Jiang Wu, Runze Song, Lingfeng Guo, and Zeqiu Xu. Predicting financial enterprise stocks and economic data trends using machine learning time series analysis. Applied and Computational Engineering, 87:26–32, 2024.

• [66] Ching Chang, Chiao-Tung Chan, Wei-Yao Wang, Wen-Chih Peng, and Tien-Fu Chen. Timedrl: Disentangled representation learning for multivariate time-series. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 625–638. IEEE, 2024.

• [67] Ching Chang, Wei-Yao Wang, Wen-Chih Peng, Tien-Fu Chen, and Sagar Samtani. Align and fine-tune: Enhancing llms for time-series forecasting. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024.

• [68] Cheng-Ming Lin, Ching Chang, Wei-Yao Wang, Kuang-Da Wang, and Wen-Chih Peng. Root cause analysis in microservice using neural granger causal discovery. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 206–213, 2024.

• [69] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 8980–8987, 2022.

• [70] Ross Koval, Nicholas Andrews, and Xifeng Yan. Financial forecasting from textual and tabular time series. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8289–8300, 2024.

• [71] Pratibha Kumari and Mukesh Saini. An adaptive framework for anomaly detection in time-series audio-visual data. IEEE Access, 10:36188–36199, 2022.

• [72] Nasir Hayat, Krzysztof J Geras, and Farah E Shamout. Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images. In Machine Learning for Healthcare Conference, pages 479–503. PMLR, 2022.

• [73] Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, and Shu-Tao Xia. Calf: Aligning llms for time series forecasting via cross-modal fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18915–18923, 2025.

• [74] Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, and Junhua Zhao. From news to forecast: Integrating event analysis in llm-based time series forecasting with reflection. Advances in Neural Information Processing Systems, 37:58118–58153, 2024.

• [75] Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188–1196. PMLR, 2014.

• [76] Silin Yang, Dong Wang, Haoqi Zheng, and Ruochun Jin. Timerag: Boosting llm time series forecasting via retrieval-augmented generation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025.

• [77] Kanghui Ning, Zijie Pan, Yu Liu, Yushan Jiang, James Y Zhang, Kashif Rasul, Anderson Schneider, Lintao Ma, Yuriy Nevmyvaka, and Dongjin Song. Ts-rag: Retrieval-augmented generation based time series foundation models are stronger zero-shot forecaster. arXiv preprint arXiv:2503.07649, 2025.

• [78] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. arXiv preprint arXiv:2408.03615, 2024.

• [79] Honghao Gao, Binyang Qiu, Ramon J Duran Barroso, Walayat Hussain, Yueshen Xu, and Xinheng Wang. Tsmae: a novel anomaly detection approach for internet of things time series data using memory-augmented autoencoder. IEEE Transactions on network science and engineering, 10(5):2978–2990, 2022.

• [80] Winnie Chow, Lauren Gardiner, Haraldur T Hallgrímsson, Maxwell A Xu, and Shirley You Ren. Towards time series reasoning with llms. arXiv preprint arXiv:2409.11376, 2024.

• [81] Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, and Dan Pei. Chatts: Aligning time series with llms via synthetic data for enhanced understanding and reasoning. arXiv preprint arXiv:2412.03104, 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the paper's contributions: (1) a dataset categorizing irregularities into nine cause-driven types, (2) a modular benchmark library (IMM-TSF) for multimodal forecasting, and (3) empirical validation showing multimodality boosts performance. These claims are supported by experimental results and theoretical framing throughout the paper.

Guidelines:

• The answer NA means that the abstract and introduction do not include the claims made in the paper.
• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 5 and Appendix B.

Guidelines:

• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
• The authors are encouraged to create a separate "Limitations" section in their paper.
• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA] Justification: NA.

Guidelines:

• The answer NA means that the paper does not include theoretical results.
• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.
• All assumptions should be clearly stated or referenced in the statement of any theorems.
• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
• Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 4 and Appendices J and K.

• The answer NA means that the paper does not include experiments.
• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
• (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
• (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
• (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
• (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] Justification: Abstract

Guidelines:

• The answer NA means that paper does not include experiments requiring code.
• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.
• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.
• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 4 and Appendix K.

Guidelines:

• The answer NA means that the paper does not include experiments.
• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
• The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Section 4 and Appendix K.

• The answer NA means that the paper does not include experiments.
• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

• The assumptions made should be given (e.g., Normally distributed errors).

• It should be clear whether the error bar is the standard deviation or the standard error of the mean.

• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section 4 and Appendix K.

Guidelines:

• The answer NA means that the paper does not include experiments.
• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: Section 2 and Appendix G

Guidelines:

• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Section 5 and Conclusion.

• The answer NA means that there is no societal impact of the work performed.
• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: For healthcare and sensitive domains (e.g., MIMIC), raw data is not redistributed. Only preprocessing instructions are provided to respect licensing and privacy agreements, as documented in Appendix G.

Guidelines:

• The answer NA means that the paper poses no such risks.
• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Appendix G

• The answer NA means that the paper does not use existing assets.
• The authors should cite the original paper that produced the code package or dataset.
• The authors should state which version of the asset is used and, if possible, include a URL.
• The name of the license (e.g., CC-BY 4.0) should be included for each asset.
• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

• If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Section 2 and Appendix D.

Guidelines:

• The answer NA means that the paper does not release new assets.
• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
• The paper should discuss whether and how consent was obtained from people whose asset is used.
• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] Justification: NA. Guidelines:

• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] Justification: NA. Guidelines:

• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA] Justification: NA.

• The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
• Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be described.