Which leakage types actually matter?

Which Leakage Types Matter?

A Quantitative Landscape Across 2,047 Benchmark Datasets

Roth, S. (2026). EPAGOGY. · arXiv:2604.04199

The question

Everyone agrees data leakage is bad. But the severity had not been comparatively measured — not across classes, not at scale, not with pre-registered predictions. Two caveats: predictions were registered internally, not on a public registry; and d_z is standardised on paired differences, so cross-class comparisons should be read as directional, not exact. Textbooks emphasize fitting scalers on the full dataset. Competition winners worry about seed selection. Reviewers flag target encoding. Are these the same severity? The same mechanism? The answers are not what the conventional emphasis implies.

This paper tests 29 experiments (28 core + 1 boundary study on 129 temporal datasets) across 2,047 tabular benchmark datasets from OpenML and PMLB, covering 17 scientific fields. The design is within-subject counterfactual: each dataset and model serves as its own control. Every finding replicates on a held-out confirmation split.

Four classes

Leakage effects vary by two orders of magnitude. The variation is not random — it is organized by causal mechanism. Four classes emerge:

Effect sizes at a glance

Cohen’s d_z for each leakage type, measured across 2,047 datasets. Hover for raw ΔAUC.

The uncomfortable finding

The leakage that textbooks emphasize most — fitting a scaler on the full dataset — has negligible effect. The leakage that competition platforms structurally incentivize — peeking at test performance and selecting the best seed — produces the largest and most persistent inflation.

Peeking at a 10-fold cross-validated test set inflates AUC by +0.040 (d_z = 0.93). This does not shrink with more data. The non-zero floor is ΔAUC∞ ≈ 0.046.

Seed cherry-picking (best of 10 random seeds) inflates AUC by +0.045 with a logarithmic dose-response curve (R² = 0.994 on 5 aggregated dose levels, not individual datasets). Try 100 seeds and you get +0.08.

Key experiments

Class IV (boundary leakage) is not in the table above — all rows measure effects under random CV, and random CV cannot detect boundary leakage by design. The boundary experiment covered 129 temporal datasets: 14 with verified genuine timestamps (effect +0.023 AUC), 92 FOREX null controls without real drift (effect near zero), and 23 with spurious time columns (excluded from effect estimates).

Cross-validation coverage gap

Across these 2,047 datasets, standard k-fold cross-validation confidence intervals achieve only 55.1% actual coverage at nominal 95%. The Student-t correction improves this to 70.4%. The best method tested (Conservative-Z) reaches 89%. None of the methods tested achieves the stated 95%.

Actual coverage at nominal 95% for six CI methods, measured on Logistic Regression (LR) and Decision Tree (DT). Bootstrap collapses for DT. Reference line at 95% nominal.

This means published error bars from cross-validation are systematically too narrow. Reported differences between models may not be differences at all.

Mechanism matters more than data

A Bayesian hierarchical meta-regression across 12,103 observations confirms: between-experiment variance (τ_exp = 0.013) exceeds between-dataset variance (τ_ds = 0.005) by 2.6×. All dataset moderators — log(n), log(p), class imbalance — classify as NULL after hierarchical correction.

The practical implication: leakage prevention should be unconditional. Dataset characteristics do not reliably predict which datasets are vulnerable. You either prevent the mechanism or you don’t.

Validation

Internal replication: All findings confirmed on held-out confirmation split. Prediction scorecard: 10/13 directional predictions confirmed (internal validation protocol, not formal pre-registration on an external platform); 3 failures documented in the paper. N-scaling: 493 datasets tested from n = 50 to 10,000 — Class I vanishes at n ≥ 200; Class II persists across all tested sample sizes.

Why this matters

If you teach ML, you should spend less time on scaler leakage and more time on peeking and seed selection. If you build ML tools, you should enforce constraints on test-set access, not just preprocessing order. If you review ML papers, you should ask how many seeds were tried and whether the test set was touched before the final report.

These findings are the empirical foundation for the grammar of machine learning workflows. The grammar’s four hard constraints — terminal assess, partition isolation, seed commitment, and CV-coverage correction — target the two classes that matter (II and III), not the one that doesn’t (I). Without this landscape, those constraints would be opinion. With it, they are calibrated to measured effect sizes.

Citation