Version Compatibility

Reference examples tested with: pandas 2.2+, scikit-learn 1.4+, xgboost 2.0+, imbalanced-learn 0.12+.

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures

Two high-risk drifts: XGBoost moved early_stopping_rounds from fit() to the constructor (deprecated 1.6, removed from fit() in 2.1); scikit-learn deprecated LogisticRegression(penalty=) in 1.8 (use l1_ratio+C) and CalibratedClassifierCV(cv='prefit') in 1.6 (use FrozenEstimator). If code throws TypeError/FutureWarning, switch to the constructor / l1_ratio / FrozenEstimator form.

Classification Models for Omics Data

"Build a classifier from my expression data" -> Start with a regularized linear model (often the ceiling in p>>n), check for batch shortcuts, and treat the probability -- not the label -- as the product.

• Linear (often best): LogisticRegression(penalty='elasticnet', solver='saga')
• Trees when nonlinear/interaction signal: RandomForestClassifier, xgboost.XGBClassifier
• Imbalance: class_weight='balanced' or threshold tuning, NOT SMOTE for risk models

The Single Most Important Modern Insight -- In p>>n, Simple Often Wins and the Probability, Not the Label, Is the Product

Omics classification almost always lives in p>>n (thousands of features, tens-to-hundreds of samples). Two counterintuitive consequences follow. First, more flexible is not better: with n in the dozens the variance of a flexible learner dominates, the full covariance is singular so QDA/full-LDA are undefined, and simple diagonal/linear methods match or beat elaborate ones (Dudoit 2002). "Random forest is the obvious choice for expression data" is a myth -- SVM/regularized logistic frequently win on microarray-style problems (Statnikov 2008), and gradient-boosted trees beat deep nets on tabular/omics data (Grinsztajn 2022). Regularization is the load-bearing wall, not a tuning nicety.

Second, in diagnostic/prognostic use the probability is the product, not the label -- which makes calibration, not accuracy, the thing that breaks silently (Van Calster 2019). A model can rank perfectly (AUC 0.9) and still output dishonest risks. And the most common cause of a beautiful AUC is not skill but a batch artifact: if batch correlates with the outcome, the classifier learns the cleaner technical signal and the performance collapses on any independent cohort.

Algorithm Choice for p>>n

Model	Wins when	Overfits / fails when	Calibration	Scaling
L1 logistic (lasso)	Sparse signal, want a small signature	Correlated features -> unstable selection; >n true signals	Good (proper loss); shrinks toward base rate	Standardize
L2 / elastic-net logistic	Many small correlated effects; omics default	Needs C (and l1_ratio) tuning	Good; preferred when calibration matters	Standardize
DLDA / nearest-centroid	Tiny n, roughly linear (Dudoit 2002)	Strong interactions; non-Gaussian	Crude; recalibrate	Variance-scaled
Linear SVM	High-dim linear separability (Statnikov 2008)	Heavy overlap; needs C	No native probabilities -- Platt-scale decision_function	Critical
Random forest	Nonlinear/interaction signal; robust baseline	Sparse-linear signal; tiny n; OOB-as-test leakage	Bagged votes bounded away from 0 and 1	Scale-invariant
GBDT (XGBoost/LightGBM)	Best general tabular performer	Tiny n + deep/many rounds; needs early stopping	Log-loss overfitting tends to overconfident extremes	Scale-invariant
Tabular deep nets	Very large n; multimodal/transfer	Typical omics n -> loses to GBDT	Variable; often needs temperature scaling	Standardize

Tree ensembles are often miscalibrated and the direction depends on the learner and loss: classic boosted ensembles push probabilities toward 0.5 (sigmoid distortion; Niculescu-Mizil 2005), bagged forests are comparatively well-calibrated but bound their votes away from 0 and 1, and modern gradient boosting trained to log-loss for many rounds tends to overfit toward overconfident extremes. Check a reliability curve and recalibrate rather than assuming a direction.

Decision Tree by Scenario

Scenario	Recommended approach	Why
Default omics classifier, want a signature	Elastic-net logistic	Often the ceiling in p>>n; sparse + grouping; well-calibrated
Suspected nonlinear/interaction (epistasis, thresholds)	Random forest then XGBoost with early stopping	Trees capture interactions; benchmark vs the linear baseline
Probabilities will drive a clinical decision	Linear model + calibration check; recalibrate if needed	Probability is the product; AUC is blind to calibration
Class imbalance	class_weight='balanced' or threshold tuning; never SMOTE for risk	Resampling destroys calibration for no AUC gain
Mixed continuous + categorical features	ColumnTransformer (scale continuous, encode categorical)	Different feature types need different handling
Missing values, especially below-detection	XGBoost/LightGBM native NaN handling	Missingness is often informative (MNAR)
Considering a deep net	Only at very large n or multimodal/raw inputs	GBDT beats deep on engineered omics matrices (Grinsztajn 2022)
Need unbiased performance / nested CV / calibration metrics	-> machine-learning/model-validation	Evaluation is its own discipline
Time-to-event outcome	-> machine-learning/survival-analysis	Censoring needs survival models, not classifiers

Core Workflow: Regularized Logistic First

Goal: A calibrated, interpretable baseline that is often the best omics classifier.

Approach: Standardize inside a Pipeline and fit elastic-net logistic with cross-validated penalty; the L2 component keeps correlated genes together, the L1 component yields a sparse signature.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegressionCV

# saga supports elasticnet; C = 1/lambda (small C = strong shrinkage). Standardize: the penalty is scale-sensitive.
clf = LogisticRegressionCV(penalty='elasticnet', solver='saga', l1_ratios=[0.1, 0.5, 0.9],
                           Cs=20, cv=5, max_iter=10000, class_weight='balanced')
pipe = Pipeline([('scaler', StandardScaler()), ('clf', clf)])
pipe.fit(X_train, y_train)

Tree Ensembles

Goal: Capture nonlinear and interaction structure when the linear baseline leaves signal on the table.

Approach: Random forest needs no scaling and is a robust baseline; XGBoost needs a low learning rate, shallow depth, and early stopping (set in the constructor in 2.x) to avoid overfitting tiny n.

from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier

rf = RandomForestClassifier(n_estimators=500, max_features='sqrt', min_samples_leaf=3,
                            class_weight='balanced', n_jobs=-1, random_state=0)

# XGBoost 2.x: early_stopping_rounds and eval_metric go in the CONSTRUCTOR, not fit().
# scale_pos_weight is omitted on purpose: like resampling, it reweights the prior and
# distorts calibration -- use it only for hard-label problems, not risk models (see Class Imbalance).
xgb = XGBClassifier(n_estimators=2000, learning_rate=0.03, max_depth=4, subsample=0.8,
                    colsample_bytree=0.5, reg_lambda=1.0,
                    early_stopping_rounds=50, eval_metric='aucpr', n_jobs=-1, random_state=0)
xgb.fit(X_train, y_train, eval_set=[(X_val, y_val)])      # NaN handled natively (missing=np.nan)

Detecting Batch Shortcut Learning

Goal: Rule out that a high AUC is a batch artifact rather than biology.

Approach: Try to predict the batch from the features and use batch-aware splits; if batch is confounded with the outcome, no correction rescues the design (Soneson 2014) -- fix it at the design stage.

from sklearn.model_selection import cross_val_score, StratifiedGroupKFold
from scipy.stats import chi2_contingency
import pandas as pd

# 1. Can the classifier predict the BATCH? If yes, batch is a strong axis and the label model is suspect.
batch_auc = cross_val_score(pipe, X, batch_labels, cv=5, scoring='roc_auc')
print(f'Batch predictability AUC: {batch_auc.mean():.2f} (high = shortcut risk)')

# 2. Is the outcome associated with batch by design?
print('label vs batch p:', chi2_contingency(pd.crosstab(y, batch_labels))[1])

# 3. Leave-one-batch-out is the honest generalization estimate (usually << random-split CV).
gcv = StratifiedGroupKFold(n_splits=5)
honest = cross_val_score(pipe, X, y, cv=gcv, groups=batch_labels, scoring='roc_auc')
print(f'Batch-aware AUC: {honest.mean():.2f}')

Class Imbalance: What Works and What Fails

Goal: Handle a rare positive class without destroying the probabilities.

Approach: For a risk model, do not resample -- class-weight cautiously or tune the threshold on a validation fold; SMOTE/oversampling change the training prior, inflate minority probabilities, give no AUC gain, and the same sensitivity is recoverable by moving the threshold (van den Goorbergh 2022). When resampling is unavoidable (a hard-label problem), use an imblearn Pipeline so only training folds are resampled.

from imblearn.pipeline import Pipeline as ImbPipeline   # NOT sklearn's Pipeline
from imblearn.over_sampling import SMOTE
from sklearn.linear_model import LogisticRegression

# Correct placement: SMOTE's fit_resample runs only during fit on the train fold, no-op on transform.
imb = ImbPipeline([('smote', SMOTE(random_state=0)),
                   ('clf', LogisticRegression(max_iter=5000))])
# Prefer for risk models: no resampling, then pick the operating threshold by cost on a validation fold.

Probability Calibration

Goal: Ensure a "0.9" means a 90% risk, not just a high rank.

Approach: Tree ensembles are often miscalibrated -- bagged forests bound votes away from 0 and 1, classic boosting is sigmoid-distorted toward 0.5 (Niculescu-Mizil 2005), and log-loss GBDT can overfit to overconfident extremes -- so check a reliability curve and recalibrate on a disjoint fold. Logistic regression optimizes a proper scoring rule and is usually best-calibrated out of the box. See machine-learning/model-validation for reliability curves, Brier, and the full protocol.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator               # sklearn >=1.6; cv='prefit' deprecated
calibrated = CalibratedClassifierCV(FrozenEstimator(rf.fit(X_tr, y_tr)), method='isotonic')
calibrated.fit(X_cal, y_cal)                             # X_cal disjoint from train and test

Preprocessing, Scaling, Encoding, Missing Data

• Inside the CV fold (refit per fold via Pipeline): feature selection, scaling, log/VST/quantile normalization, ComBat batch correction, imputation, PCA. Fitting any of these on the full matrix is leakage (machine-learning/model-validation).
• Scale-sensitive: regularized logistic, SVM, kNN, PCA, neural nets -- standardize. Scale-invariant: trees/RF/GBDT.
• Variant features: additive 0/1/2 ordinal for trees/additive logistic; one-hot for non-additive (dominant/recessive) effects; CatBoost-style encoding for high-cardinality (HLA).
• Missing data: XGBoost/LightGBM learn a default split direction for NaN; in omics missingness is often MNAR (below detection limit) and a missingness indicator can carry signal -- naive zero-fill conflates "absent" with "not measured."

Hyperparameters That Matter

Model	Tune these	Leave default
Logistic	C (log-spaced), l1_ratio, class_weight	solver (saga for elasticnet)
Random forest	max_features, min_samples_leaf, max_depth (cap for tiny n)	n_estimators (more is safe; 500-1000)
XGBoost	learning_rate+n_estimators+early stopping, max_depth (3-6), subsample, colsample_bytree, reg_lambda	most others

Per-Method Failure Modes

Beautiful AUC that is a batch artifact

• Trigger: Cases and controls processed in different batches/sites/times.
• Mechanism: The classifier exploits the cleaner technical signal; even nested CV is optimistic, and ComBat cannot rescue a confounded design (Soneson 2014).
• Symptom: Near-perfect CV AUC; collapse on an independent cohort; the model predicts batch easily.
• Fix: Detect with the batch-prediction check; use leave-one-batch-out; fix at design (balance batches across outcome).

RF/boosting probabilities trusted as risks

• Trigger: Reading predict_proba from RF or XGBoost as a calibrated risk.
• Mechanism: RF bounds votes away from 0 and 1; classic boosting is sigmoid-distorted (Niculescu-Mizil 2005) and log-loss GBDT can overfit to overconfident extremes.
• Symptom: Good AUC, reliability curve far from diagonal.
• Fix: Recalibrate on a disjoint fold; or prefer logistic when the probability matters.

SMOTE-before-split / SMOTE for a risk model

• Trigger: Resampling before the CV split, or to "fix" imbalance for a probability model.
• Mechanism: Synthetic points derived from test samples leak; resampling inflates minority risk and wrecks calibration (van den Goorbergh 2022; Carriero 2025).
• Symptom: Inflated CV performance; predicted risks systematically too high.
• Fix: imblearn Pipeline (train-fold only); for risk models, do not resample -- tune the threshold.

OOB error read as an unbiased test estimate

• Trigger: Reporting RF out-of-bag error after selecting features on the full data.
• Mechanism: Selection leaked; OOB then reflects the contaminated feature set.
• Symptom: Optimistic OOB; external collapse.
• Fix: Selection inside CV; estimate performance by nested CV.

Quantitative Thresholds

Threshold	Source	Rationale
Try a regularized linear model first	Dudoit 2002; Statnikov 2008	Simple often beats complex in p>>n
GBDT over deep nets for tabular omics	Grinsztajn 2022	Trees handle uninformative features and non-rotational data
Do not resample for risk models	van den Goorbergh 2022; Carriero 2025	Resampling destroys calibration for no AUC gain
XGBoost: low LR + many rounds + early stopping	field standard	Prevents overfitting tiny n
Report AUPRC + MCC under imbalance	Saito 2015; Chicco 2020	Accuracy and ROC-AUC mislead when positives are rare

Common Errors

Error / symptom	Cause	Solution
XGBoost early_stopping_rounds TypeError in fit()	Moved to constructor in 2.x	Pass it (and eval_metric) in XGBClassifier(...)
penalty='l1' FutureWarning	Deprecated in sklearn 1.8	Use l1_ratio=1 + C (1.8+) or keep penalty on 1.4-1.7
elasticnet solver error	Only saga supports it	solver='saga' + l1_ratio
CalibratedClassifierCV(cv='prefit') deprecated	sklearn 1.6	Wrap in FrozenEstimator
95% accuracy but useless model	Imbalance + accuracy metric	Report AUPRC/MCC; check the confusion matrix

References

• Dudoit S, Fridlyand J, Speed TP. 2002. Comparison of discrimination methods for the classification of tumors using gene expression data. J Am Stat Assoc 97:77-87.
• Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. 2002. SMOTE: Synthetic Minority Over-sampling Technique. J Artif Intell Res 16:321-357.
• Zou H, Hastie T. 2005. Regularization and variable selection via the elastic net. J R Stat Soc B 67:301-320.
• Niculescu-Mizil A, Caruana R. 2005. Predicting good probabilities with supervised learning. Proc 22nd ICML 625-632.
• Diaz-Uriarte R, Alvarez de Andres S. 2006. Gene selection and classification of microarray data using random forest. BMC Bioinformatics 7:3.
• Statnikov A, Wang L, Aliferis CF. 2008. A comprehensive comparison of random forests and support vector machines for microarray-based cancer classification. BMC Bioinformatics 9:319.
• Soneson C, Gerster S, Delorenzi M. 2014. Batch effect confounding leads to strong bias in performance estimates obtained by cross-validation. PLoS ONE 9:e100335.
• Saito T, Rehmsmeier M. 2015. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10:e0118432.
• Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. 2019. Calibration: the Achilles heel of predictive analytics. BMC Med 17:230.
• Chicco D, Jurman G. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21:6.
• Shwartz-Ziv R, Armon A. 2022. Tabular data: deep learning is not all you need. Inf Fusion 81:84-90.
• Grinsztajn L, Oyallon E, Varoquaux G. 2022. Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS Datasets and Benchmarks.
• van den Goorbergh R, van Smeden M, Timmerman D, Van Calster B. 2022. The harm of class imbalance corrections for risk prediction models. J Am Med Inform Assoc 29:1525-1534.
• Carriero A, Luijken K, de Hond A, Moons KGM, van Calster B, van Smeden M. 2025. The harms of class imbalance corrections for machine learning based prediction models. Stat Med 44:e10320.

Related Skills

• machine-learning/model-validation - Nested CV, calibration, and net benefit for the trained classifier
• machine-learning/biomarker-discovery - Select features before modeling (inside the CV fold)
• machine-learning/prediction-explanation - Interpret the classifier and detect shortcuts with SHAP
• machine-learning/survival-analysis - Time-to-event outcomes that classifiers cannot handle
• differential-expression/batch-correction - Batch correction done design-aware, not across the split
• expression-matrix/normalization - Per-sample normalization that is safe outside the CV fold