Lecture Overview

Today we explore Model Validation—the core framework for evaluating, selecting, and tuning machine learning models.

1. Data Splitting Strategy: Training, Validation, & Testing sets
2. Stratified Splitting: Handling class imbalance
3. Cross-Validation: \(K\)-Fold & Leave-One-Out (LOO)
4. Model Selection: Impact of data splits & selection workflow
5. Hyperparameter Tuning: Grid Search & Model Optimization
6. Validation Curves: Diagnosing Bias vs. Variance

How Do We Know If a Model Works?

We split the data into separate sets so that we can answer three different questions:

• Train — Can the model learn?
  Use the training data to estimate the model’s parameters.

• Validate — Which model should we use?
  Use the validation data to compare models, select among alternatives, and tune hyperparameters.

• Test — How well does the final model generalize?
  Use unseen test data to obtain an unbiased estimate of the final model’s performance.

Train → Choose → Evaluate

Why is Cross-Validation Important?

• Preventing Overfitting: A model that memorizes training data will fail on new, unseen data.
• Realistic Performance Estimation: Evaluating on training data gives overly optimistic accuracy.
• Data Efficiency: Single train/test splits can lead to high variance depending on which samples fall into which set.
• Robustness: Cross-validation tests performance across multiple subsets to ensure reliability.

Data Splitting: Train, Validation, and Test Sets

To properly train and evaluate models, we divide dataset \(\mathcal{D}\) into three distinct sets:

Golden Rule: Never use the Test Set for model selection or tuning! It must remain vault-locked until final evaluation.

Stratified Splits

A sampling method that ensures each fold or split contains approximately the same percentage of samples of each target class as the complete dataset.

When to Use?

• Imbalanced Datasets: e.g., Fraud detection (99% normal, 1% fraud).
• Small Datasets: Random splits might omit rare classes entirely from train or validation sets.
• Classification Tasks: Standard practice for single-label and multi-label classification.

Code: Creating Train, Validation, and Test Sets

In scikit-learn, we can perform a 3-way split by calling train_test_split twice:

from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris

# Load dataset
X, y = load_iris(return_X_y=True)

# Step 1: Split into Train+Val (80%) and Test (20%)
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42, stratify=y
)

# Step 2: Split Train+Val into Train (75% of 80% = 60%) and Validation (25% of 80% = 20%)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25, random_state=42, stratify=y_train_val
)

print(f"Train: {X_train.shape[0]} | Val: {X_val.shape[0]} | Test: {X_test.shape[0]}")

K-Fold Cross-Validation

Split the training dataset into \(K\) equal-sized subsets (folds).

1. Train the model on \(K-1\) folds.
2. Validate the model on the remaining \(1\) fold.
3. Repeat \(K\) times, using each fold as validation once.
4. Compute the final performance metric as the average across all \(K\) trials:

\[\text{CV Score} = \frac{1}{K} \sum_{i=1}^{K} \text{Score}_i\]

• Common Values for \(K\): \(K=5\) or \(K=10\) (good trade-off between computational cost and bias/variance).

Leave-One-Out Cross-Validation (LOO-CV)

An extreme case of \(K\)-Fold CV where \(K = N\) (\(N\) = total number of samples).

• For each iteration:

• Train on \(N-1\) samples.

• Test on the single \(1\) remaining sample.

• Advantages:

• Unbiased performance estimation (uses almost all data for training).

• Fully deterministic (no randomness in splitting).

• Disadvantages:

• High computational cost (fits \(N\) separate models).

• High variance in evaluation error.

Code: Leave-One-Out CV in Scikit-Learn

from sklearn.model_selection import LeaveOneOut, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris

# Load dataset
X, y = load_iris(return_X_y=True)

# Initialize LeaveOneOut iterator
loo = LeaveOneOut()
model = LogisticRegression(max_iter=1000)

# Evaluate using LOO (fits 150 times for Iris dataset)
scores = cross_val_score(model, X, y, cv=loo, scoring='accuracy')

print(f"Number of fits (N): {len(scores)}")
print(f"LOO-CV Mean Accuracy: {scores.mean():.4f}")

Code: Implementing Cross-Validation in Scikit-Learn

from sklearn.model_selection import cross_val_score, KFold, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(random_state=42)

# Standard 5-Fold CV
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print(f"5-Fold CV Accuracy: {scores.mean():.4f} (+/- {scores.std():.4f})")

# Custom Stratified K-Fold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
skf_scores = cross_val_score(model, X, y, cv=skf, scoring='accuracy')
print(f"Stratified 5-Fold Accuracy: {skf_scores.mean():.4f}")

Model Selection

Model Selection is the process of selecting the optimal machine learning algorithm (or architecture) among a set of candidate models for a given problem.

Examples of candidate models:

• Logistic Regression vs. Support Vector Machines vs. Decision Trees vs. Random Forests.

Goal

Identify the model architecture that generalizes best to unseen data.

What Makes a Good Model?

When comparing candidate models, consider:

• Strong performance — high accuracy, low mean squared error, etc.
• Consistent performance — similar results across cross-validation or different data splits.
• Reasonable assumptions — model assumptions are appropriate and not seriously violated.
• Interpretability — the model is understandable when interpretation is important.

Key Idea

There is often no single best model.

We want a model that generalizes well while balancing:

Performance + Consistency + Reasonable assumptions

Bias–Variance Tradeoff

A model’s ability to generalize depends on two sources of error:

• Bias — error from a model that is too simple to capture important patterns.
• Variance — error from a model that is too sensitive to the particular training data.

The Tradeoff

• High bias → model underfits the data.
• High variance → model overfits the data.
• Low bias + low variance → better potential to generalize to unseen data.

The goal is not to minimize training error alone.

But to Choose a model that captures meaningful patterns without fitting noise.

Impact of Splits on Performance & Parameter Estimates

Single train/test splits can lead to volatile conclusions:

• Metric Instability: A lucky or unlucky split can inflate or deflate accuracy by several percentage points.
• Parameter Sensitivity: Model coefficients (or tree structures) vary wildly depending on which samples end up in the training set.
• Cross-Validation Solution: Averaging metrics across \(K\) folds reduces estimate variance and provides a confidence interval (\(\text{mean} \pm \text{std}\)).

Steps for Model Selection with Cross-Validation

1. Split Data: Reserve a holdout Test Set (\(10\% - 20\%\)).
2. Define Candidate Models: List algorithm families to compare.
3. Evaluate via CV: Perform \(K\)-Fold CV on the training set for each candidate.
4. Compare Performance: Compute mean and standard deviation of validation metrics across folds.
5. Select Best Model: Choose the model family with the superior CV score.
6. Refit & Test: Re-train selected model on the entire training set and evaluate on the holdout Test Set.

Code: Implementing Model Selection with CV

from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000),
    "Support Vector Machine": SVC(),
    "Random Forest": RandomForestClassifier(random_state=42)
}

# Evaluate candidates on training portion
for name, clf in models.items():
    cv_scores = cross_val_score(clf, X_train_val, y_train_val, cv=5)
    print(f"{name}: Mean CV Accuracy = {cv_scores.mean():.4f} (std: {cv_scores.std():.4f})")

Hyperparameter vs. Parameter & Model Tuning

Parameters vs. Hyperparameters

• Parameters: Learned directly from data during training (e.g., weights \(w\), bias \(b\), split points in trees).
• Hyperparameters: Configured before training to control learning behavior (e.g., learning rate \(\alpha\), \(C\) in SVM, max_depth in Decision Trees).

Model Tuning

The process of searching for the combination of hyperparameters that yields optimal model validation performance.

sklearn methods

• sklearn provides two main methods for hyperparameter optimization
• Grid Search — systematically evaluates every combination in a predefined grid.
• Random Search — randomly samples configurations from specified distributions.

The choice of search strategy depends largely on the size and structure of the hyperparameter space.

Setting a Tuning Grid & Grid Search

A Tuning Grid defines a discrete search space over hyperparameter values.

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [3, 5, 10, None],
    'min_samples_split': [2, 5]
}

• GridSearchCV: Evaluates all \(3 \times 4 \times 2 = 24\) combinations using \(K\)-Fold cross-validation.
• Selection: Picks the combination that yields the highest mean CV score.

Grid Search

• Covers all possible combinations from the provided grid.
• Can be parallelized easily.
• Integrates cross-validation.

Grid search example

What Does GridSearchCV Do?

• Generates all hyperparameter combinations.
• Trains and evaluates each combination using cross-validation.
• Calculates the mean CV score for each configuration.
• Identifies the configuration with the highest mean CV score.

The best configuration can be accessed with:

#| eval: false

grid_search.best_params_

Random Search

• More efficient than grid search when dealing with large hyperparameter spaces.
• Samples a given number of parameter settings from distributions.

Random search example

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

pipe_svc = make_pipeline(CountVectorizer(), SVC())

param_dist = {
    "countvectorizer__max_features": randint(100, 2000), 
    "svc__C": uniform(0.1, 1e4),  # loguniform(1e-3, 1e3),
    "svc__gamma": loguniform(1e-5, 1e3),
}
random_search = RandomizedSearchCV(pipe_svm,                                    
                                  param_distributions = param_dist, 
                                  n_iter=10, 
                                  n_jobs=-1, 
                                  return_train_score=True)

# Carry out the search
random_search.fit(X_train, y_train)
random_search.best_score_

Code: Implementing Model Tuning in Scikit-Learn

#| eval: false

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

# Define candidate model and search space
rf = RandomForestClassifier(random_state=42)
param_grid = {
    'n_estimators': [25, 50, 100],
    'max_depth': [3, 5, None],
    'min_samples_split': [2, 5]
}

# Set up Grid Search with 5-Fold CV
grid_search = GridSearchCV(estimator=rf, param_grid=param_grid, cv=5, scoring='accuracy', n_jobs=-1)
grid_search.fit(X_train_val, y_train_val)

print("Best Parameters:", grid_search.best_params_)
print(f"Best CV Accuracy: {grid_search.best_score_:.4f}")

# Evaluate optimal model on holdout test set
best_model = grid_search.best_estimator_
test_accuracy = best_model.score(X_test, y_test)
print(f"Final Holdout Test Accuracy: {test_accuracy:.4f}")