Skip to main content

Hyperparameter Tuning with GridSearchCV

GridSearchCV exhaustively searches over a specified parameter grid, training one model for each parameter combination and selecting the best based on cross-validation performance. Unlike random guessing or manual tweaking, GridSearchCV is systematic: it evaluates every configuration on the same data using cross-validation, ensuring fair comparison and reproducible results. For small parameter spaces (up to 10^4 configurations), grid search is the gold standard.

Understanding GridSearchCV​

GridSearchCV takes three key inputs: an estimator (model), a parameter grid, and a cross-validation strategy. It trains the model on every combination of parameters, evaluates each on CV folds, and returns the configuration with the best mean CV score.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

# Define the parameter grid: all combinations will be tested
param_grid = {
    'C': [0.1, 1, 10, 100],
    'kernel': ['linear', 'rbf'],
    'gamma': ['scale', 'auto']
}

# GridSearchCV: exhaustive search
grid_search = GridSearchCV(
    SVC(),                          # Base estimator
    param_grid,                     # Parameter combinations
    cv=5,                           # 5-fold cross-validation
    scoring='accuracy',             # Metric to optimize
    n_jobs=-1                       # Use all CPU cores in parallel
)

# Fit: trains all 4 * 2 * 3 = 24 models on 5-fold CV
# Total: 24 configs * 5 folds * 2 (train/test) = 240 model trainings
grid_search.fit(X, y)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.3f}")
print(f"Best model: {grid_search.best_estimator_}")

The n_jobs=-1 parameter parallelizes training across all CPU cores, cutting search time dramatically. For large grids, this is essential.

Designing Effective Parameter Grids​

A good grid balances coverage and computational cost. Start broad, then narrow down:

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split

# Phase 1: Coarse grid to find the ballpark
param_grid_coarse = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, 20, None],
    'min_samples_split': [2, 5, 10]
}

# Total: 3 * 4 * 3 = 36 configurations

grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid_coarse,
    cv=5,
    n_jobs=-1
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
grid_search.fit(X_train, y_train)
print(f"Coarse grid best: {grid_search.best_params_}")

# Phase 2: Fine grid around the best coarse parameters
param_grid_fine = {
    'n_estimators': [80, 100, 120],
    'max_depth': [8, 10, 12],
    'min_samples_split': [4, 5, 6]
}

# Total: 3 * 3 * 3 = 27 configurations (smaller, more precise)

grid_search_fine = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid_fine,
    cv=5,
    n_jobs=-1
)

grid_search_fine.fit(X_train, y_train)
print(f"Fine grid best: {grid_search_fine.best_params_}")
print(f"Fine grid test score: {grid_search_fine.score(X_test, y_test):.3f}")

Two-phase search saves computational time: coarse phase finds the region of interest, fine phase optimizes within that region.

GridSearchCV with Pipelines​

GridSearchCV integrates seamlessly with pipelines, letting you tune preprocessing parameters alongside model parameters:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV

# Pipeline: scaling → Ridge regression
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', Ridge())
])

# Tune both the scaler and model
param_grid = {
    'scaler__with_mean': [True, False],
    'scaler__with_std': [True, False],
    'model__alpha': [0.001, 0.01, 0.1, 1, 10]
}

grid_search = GridSearchCV(
    pipeline,
    param_grid,
    cv=5,
    n_jobs=-1
)

grid_search.fit(X_train, y_train)
print(f"Best pipeline config: {grid_search.best_params_}")

# The best estimator is a fully-fitted pipeline
best_pipeline = grid_search.best_estimator_
test_score = best_pipeline.score(X_test, y_test)
print(f"Test score: {test_score:.3f}")

The __ notation (e.g., model__alpha) specifies parameters for pipeline steps. This ensures scaling parameters and model parameters are tuned together.

Analyzing GridSearchCV Results​

The cv_results_ attribute contains detailed information on every configuration:

import pandas as pd

# Convert results to DataFrame for easy analysis
results_df = pd.DataFrame(grid_search.cv_results_)

# Display top 10 configurations
print(results_df[['param_C', 'param_kernel', 'mean_test_score', 'std_test_score']]
      .sort_values('mean_test_score', ascending=False)
      .head(10))

# Visualize: parameter sensitivity
import matplotlib.pyplot as plt

results_df_sorted = results_df.sort_values('param_C')
plt.figure(figsize=(10, 5))
plt.errorbar(
    results_df_sorted['param_C'],
    results_df_sorted['mean_test_score'],
    yerr=results_df_sorted['std_test_score'],
    marker='o'
)
plt.xlabel('C (Regularization)')
plt.ylabel('CV Score')
plt.xscale('log')
plt.show()

The DataFrame shows all parameters, mean CV score, and standard deviation for each configuration. Visualizing these results reveals which parameters matter most.

Choosing the Right Scoring Metric​

GridSearchCV optimizes based on a scoring parameter. Different tasks need different metrics:

from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer, f1_score

# For classification: common scoring options
# 'accuracy' — fraction of correct predictions
# 'precision' — true positives / (true pos + false pos)
# 'recall' — true positives / (true pos + false negatives)
# 'f1' — harmonic mean of precision and recall
# 'roc_auc' — area under ROC curve

# For imbalanced classification, use 'f1' or 'roc_auc' instead of 'accuracy'
grid_search = GridSearchCV(
    LogisticRegression(random_state=42),
    {'C': [0.1, 1, 10]},
    cv=5,
    scoring='f1_macro'  # F1 for multi-class
)

# For regression: common scoring options
# 'r2' — coefficient of determination
# 'neg_mean_squared_error' — negative MSE (higher is better)
# 'neg_mean_absolute_error' — negative MAE

grid_search = GridSearchCV(
    Ridge(),
    {'alpha': [0.001, 0.01, 0.1]},
    cv=5,
    scoring='r2'
)

# Custom scoring function
def custom_metric(y_true, y_pred):
    # Your custom logic here
    return score

scoring = make_scorer(custom_metric, greater_is_better=True)
grid_search = GridSearchCV(model, param_grid, cv=5, scoring=scoring)

For imbalanced classification, never use 'accuracy'—it can be high even if your model ignores the minority class. Use 'f1', 'precision', 'recall', or 'roc_auc' instead.

Avoiding Overfitting in GridSearchCV​

GridSearchCV can overfit to the validation set if you select hyperparameters based on best CV score alone. Always evaluate on a held-out test set:

from sklearn.model_selection import train_test_split, GridSearchCV

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# GridSearchCV uses CV folds of X_train only
grid_search = GridSearchCV(model, param_grid, cv=5)
grid_search.fit(X_train, y_train)

# IMPORTANT: Evaluate only on X_test, which was never seen during tuning
final_score = grid_search.best_estimator_.score(X_test, y_test)
print(f"Train CV score: {grid_search.best_score_:.3f}")
print(f"Test score: {final_score:.3f}")

# If test_score is much lower than train_score, your model may be overfitting

The test score is your honest estimate of generalization. If it is significantly lower than the CV score, either your model is overfitting or your test set is harder than the training distribution.

Controlling Computational Cost​

Large parameter grids can take hours or days. Use these strategies to manage cost:

# Strategy 1: Reduce cv folds (3-fold instead of 5)
grid_search = GridSearchCV(model, param_grid, cv=3, n_jobs=-1)

# Strategy 2: Smaller parameter grid (fewer values per parameter)
param_grid = {
    'C': [1, 10],
    'kernel': ['linear', 'rbf']
}

# Strategy 3: Pre-filter hyperparameters based on domain knowledge
# Example: max_depth for trees should not exceed log2(n_samples)
import numpy as np
n_samples = X_train.shape[0]
max_depth_limit = int(np.log2(n_samples))
param_grid = {
    'max_depth': list(range(5, min(15, max_depth_limit)))
}

# Strategy 4: Use early stopping if the model supports it
# Example: LogisticRegression with 'lbfgs' solver and warm_start
grid_search = GridSearchCV(
    LogisticRegression(solver='lbfgs', warm_start=True),
    param_grid,
    cv=5
)

# Strategy 5: Use RandomizedSearchCV instead (next article)
# for very large grids (exponential parameter growth)

For grids with over 1000 configurations, consider RandomizedSearchCV, which samples random combinations instead of testing all.

Key Takeaways​

  • GridSearchCV systematically evaluates all parameter combinations using cross-validation, ensuring fair comparison.
  • Use two-phase search: coarse grid to find the region, fine grid to optimize within that region.
  • Integrate GridSearchCV with pipelines to tune preprocessing and model parameters together.
  • Always evaluate on a held-out test set; CV score alone can be optimistic.
  • Choose scoring metrics appropriate for your task: 'f1' for imbalanced classification, 'r2' for regression.

Frequently Asked Questions​

Does GridSearchCV overfit to the validation set?​

GridSearchCV uses cross-validation, which splits training data into multiple folds. This reduces overfitting compared to a single validation set. However, selecting the best CV score can still lead to overfitting. Always evaluate final performance on a completely held-out test set.

How long will my GridSearchCV take?​

Time = (# configs) × (# CV folds) × (time per training). With 100 configs, 5-fold CV, and 1 second per training, expect ~500 seconds (8 minutes). Use n_jobs=-1 to parallelize and reduce wall-clock time. For large grids, use RandomizedSearchCV or set cv=3.

Should I scale before GridSearchCV?​

If you use a pipeline, scaling is handled internally (correct). If you pass raw data, scale first using StandardScaler().fit(X_train) then transform() both train and test data before calling grid_search.fit().

Can I use GridSearchCV with nested cross-validation?​

Yes, for more rigorous evaluation. Use outer CV for testing and inner CV for hyperparameter tuning: cross_val_score(GridSearchCV(..., cv=5), X, y, cv=5). This is computationally expensive but gives unbiased performance estimates.

What if all my parameter combinations score similarly?​

Your model may not be sensitive to the parameters you are tuning, or your dataset may be small. Try: (1) expanding the parameter range, (2) tuning different hyperparameters, (3) using a more complex model, or (4) collecting more data.

Further Reading​