Distribution Shift Analysis in Generalizable Modelling: Intensive Care Time-Series Data
In this paper, we investigate how distribution shift affects model generalizability in medical time-series from intensive care. We define clinically relevant domain adaptation scenarios for predicting hypotension and hypoxemia using the MIMIC-III Matched Waveform Database v1.0. Distribution shift is quantified via the Kolmogorov–Smirnov test (KS) applied to an interpretable, lower-dimensional data representation. Our results highlight the challenges of measuring covariate, prior probability, and semantic shifts in non-stationary multivariate time series. We observed linear correlation (0.74 and 0.99) between KS(source train, target) - KS(source train, source test) and RMSE difference when comparing in-distribution and out-of-distribution model performance. We argue that developing and benchmarking data-driven models should explicitly account for modality-specific data shift characteristics.
