М. О. Dobrokhvalov , А.Y. Filatov , Е.А. Chegodaeva
2026-02-27
Abstract ▼
Experimental reproducibility constitutes a critical cornerstone of modern machine learning research, yet random initialization seed selection substantially influences final model performance, creating challenges for principled comparison of different architectures and methods. Random seed effects on convolutional time series classifiers were quantified, and a principled comparison criterion was established. Two 1D architectures, FCN and ResNet, were trained on seven public datasets containing different data. 55 independent runs for each combination of model and dataset were performed nder controlled pseudorandomness in Python, NumPy, and PyTorch. Deterministic backends were enabled, and identical hyperparameters were used across runs. Normality of seed-wise accuracy distributions was assessed with the Shapiro–Wilk and Anderson–Darling tests. Accuracy variability attributable to seed choice reached up to 12 percentage points in some settings, with magnitude dependent on dataset and architecture. The distributions were found to be non-normal in most cases, indicating that confidence intervals predicated on normality are unreliable. To enable fair comparison across runs, a reproducibility meta-metric, RM, was introduced that subtracts a dispersion penalty from the mean and depends on the number of runs and a tunable coefficient λ. RM was shown to lie between the empirical minimum and the mean, to approach the lower bound for small sample sizes, and to converge toward the mean as the number of runs increases. Portability of the approach was examined on an additional architecture, DenseNet, confirming expected behavior. Practical value is provided by RM metric rankings reflect both performance and stability. In this way, reproducibility and the credibility of empirical conclusions are strengthened
