A genomic classifier can score well on a held-out dataset and still fail when researchers apply it to genuinely new samples. The unanswered question is whether the evaluation data were independent of the information used to build the model. In machine learning in genomics, multiple specimens, related donors and laboratory batches make that question especially consequential. Data leakage occurs when information unavailable in the intended prediction setting influences training or evaluation. As we consider our genomics research interests, we regard a defensible validation design as essential to interpreting a reported score.
Decide what an independent observation means
A spreadsheet row is not necessarily an independent biological sample. A person may contribute several biopsies, a tissue specimen may have technical replicates, and related individuals may share genetic structure. Randomly assigning rows to training and test sets can place dependent observations on both sides. The model might learn donor or batch signatures.
Define the prediction target: another specimen from a known donor, a new patient or a different centre. For a new-patient claim, donor-level grouping may be appropriate; other applications may require grouping by family, site or collection episode. Record these relationships before splitting. Unknown dependencies should be treated as a limitation, since simply setting a random seed does not establish independence.
Keep the final test data outside model development
The order of operations determines what a performance estimate means. Researchers should define the prediction target and eligible observations first, then reserve an appropriate test partition before making data-dependent modelling choices. Fit scaling parameters, imputation rules and feature-selection procedures using only the training data available in the relevant fold. Apply the learned transformations to validation or test observations afterwards. Otherwise information from those observations can influence the model prematurely.
Tune on training-only validation folds, retaining relevant groups. Nested cross-validation can separate model selection from performance estimation. The final hold-out set should remain untouched until the pipeline and analysis choices are fixed. EMBL-EBI's guidance on data splitting and model evaluation covers these general modelling decisions, including cross-validation, bias and overfitting.
Genomic assays may require specialised normalisation. Authors should document which data each step uses. If a step relies on pooled data by design, explain why that is compatible with the intended deployment setting.
Check whether the model learned a laboratory shortcut
In a hypothetical dataset, positive and negative samples come mainly from different centres. The classifier could learn technical signatures associated with the centre. A randomly mixed test set might preserve that relationship and produce impressive discrimination. Performance on a new centre, where label and laboratory conditions differ, could be much weaker. This example illustrates a possible failure mode; it is not a finding from a Foundation study.
Examine how labels align with processing dates, instrument platforms, sequencing runs and sample preparation. Compare performance and label balance across batches where feasible. Batch correction itself needs scrutiny: estimating correction parameters from all observations before splitting may leak test information. The right approach depends on the data-generating process and the actual prediction setting.
Choose validation that matches the claim
Internal cross-validation asks how a specified modelling procedure behaves across partitions of the study dataset. A fully independent external cohort asks an additional question about transport to another collection, population or setting. Neither result should be described without its sampling design, inclusion criteria and relationship to the target population.
A published analysis of data leakage and scientific reproducibility identifies non-independent train/test observations and preprocessing performed before the split among routes to overoptimistic results. Its conclusions span machine-learning research fields; they are not a new performance estimate for a particular genomic task. Even external cohorts may share donors or incompatible outcome definitions.
For imbalanced classes, accuracy alone can conceal poor results on the rarer class. Sensitivity, specificity or precision may help, depending on the intended use. Uncertainty estimates should reflect sample dependence. No single metric resolves a flawed split.
Make the methods reviewable
A methods section should allow another investigator to reconstruct the evaluation logic. We would look for the prediction target, sample inclusion rules, identifiers used for grouping, handling of duplicates and the allocation of donors or batches to partitions. Authors should name the preprocessing steps fitted within each fold, model-selection procedure, final evaluation set and an appropriate reference model.
Disclose the number of independent units alongside the number of rows. Describe missing data, possible label leakage, the source of each feature and any tests of performance across cohorts. Where permitted, versioned code and analysis configurations help others reproduce decisions. Genomic data may carry access, consent and privacy restrictions, so reproducibility does not require indiscriminate public release of individual-level records.
A brief validation trail can be written as a sequence: define the task; identify dependencies; partition by the valid unit; train and tune without consulting the hold-out set; evaluate and report limitations. Each choice needs a reason tied to the scientific question. Otherwise even technically correct code can deliver an answer to the wrong task.
Trust the design before the headline score
We support evidence-led enquiry that can withstand questions about provenance and independence. For genomic machine-learning research, trust begins with an explicit account of what the model did and did not see. A performance figure becomes meaningful when its test samples genuinely represent the claim, its preparation steps are transparent and its remaining sources of uncertainty are visible to reviewers.