Train-Test Split, Cross-Validation & Avoiding Data Leakage0%
Linear Regression: Coefficients, R-Squared & MSE

Train-Test Split, Cross-Validation & Avoiding Data Leakage

Beginner12 min read•Updated: Jul 11, 2026
Study Materials

Train-Test Split, Cross-Validation & Avoiding Data Leakage

The true measure of a machine learning model is its ability to generalize to unseen future data. Evaluating a model on the same data it was trained on produces overly optimistic performance estimates due to overfitting. Mastering holdout splitting, Stratified K-Fold Cross-Validation, and rigorous leakage prevention is a core competency for data analysts.

1. Train-Test Split: Holdout Validation

The simplest validation strategy splits historical data into two disjoint subsets:

  • Training Set (typically 70-80%): Used by the estimator to learn optimal weights and parameters.
  • Testing Set (typically 20-30%): Kept in a vault until training is complete to assess real-world out-of-sample generalization.
Python
from sklearn.model_selection import train_test_split
 
X = df.drop(columns=['target'])
y = df['target']
 
# 80% Train, 20% Test with fixed reproducibility seed
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.20,
random_state=42,
stratify=y # Essential for imbalanced classification!
)
Why stratify=y Matters: In customer churn where only 5% of users churn, a random split might accidentally assign 8% churners to the test set and only 3% to the train set. Stratification preserves identical class ratios across both splits!

2. K-Fold Cross-Validation

A single train-test split can yield misleading results if the random split happens to contain an unusually easy or hard test set. K-Fold Cross-Validation divides the data into $K$ equal folds:

  • The model trains on $K-1$ folds and tests on the remaining fold.
  • This process repeats $K$ times so every fold acts as the validation set once.
  • The final metric is the average performance across all $K$ iterations.
Python
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
 
model = LogisticRegression(max_iter=1000)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
 
# Compute 5-fold cross validation scores
scores = cross_val_score(model, X_train, y_train, cv=cv, scoring='roc_auc')
 
print(f"5-Fold ROC-AUC Scores: {scores.round(3)}")
print(f"Mean ROC-AUC: {scores.mean():.3f} (+/- {scores.std():.3f})")

3. Data Leakage: The #1 Killer of Production ML Models

Data leakage occurs when information from outside the training dataset (such as the target variable or future test set distributions) inadvertently influences model training:

Major Sources of Data Leakage:

  1. 1
    Global Imputation/Scaling: Computing the global mean of a column across the entire dataset before calling train_test_split.
  2. 2
    Lookahead Leakage: In time series, using future data to predict the past.
  3. 3
    Duplicate / Clustered Rows: Having multiple transactions from the exact same user present in both train and test splits.

4. Bulletproof Pipelines with Scikit-Learn Pipeline

To guarantee that transformers are fit exclusively on training folds during cross-validation, wrap steps in a Scikit-Learn Pipeline:

Python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
 
# Safe pipeline: Scaling is fit strictly inside each CV fold!
pipeline = Pipeline([
('scaler', StandardScaler()),
('classifier', LogisticRegression())
])
 
# Evaluates with zero data leakage
cv_scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring='accuracy')
Interactive Quiz

Multiple Choice Questions

Question 1
What is the primary purpose of setting stratify=y in train_test_split for classification?
Question 2
In 5-Fold Cross-Validation, how many times is the model trained and evaluated?
Question 3
Which of the following constitutes severe Data Leakage?
Question 4
What is the benefit of wrapping data preprocessing and model estimators inside a Scikit-Learn Pipeline?
Question 5
Why is a standard randomized K-Fold Cross Validation unsuitable for temporal (time series) forecasting?
Next Lesson

Linear Regression: Coefficients, R-Squared & MSE

Continue learning with hands-on practice, examples, and exercises in the upcoming topic.

Related Lessons

Previous LessonNext Lesson
Feature Engineering: One-Hot Encoding, StandardScaler & MinMaxLinear Regression: Coefficients, R-Squared & MSE

Practice Quiz

Test your understanding of this lesson with 5 questions. Each question has one correct answer.

PreviousNext
PrevNext