Feature Selection Methods and Best Practices

Not all features are equal. After engineering new features, you often have 100+ candidates; including irrelevant or redundant features slows training, increases overfitting risk, and produces inflated validation metrics that crash on real data. Feature selection discards weak features and keeps only the most predictive ones. Kaggle competitors report that careful feature selection improves final model accuracy by 2-8% while cutting training time by 30-60%.

Why Feature Selection Matters

High-dimensional datasets suffer from the curse of dimensionality: algorithms require exponentially more data to generalize as feature count rises. With 1,000 features and 10,000 samples, the effective sample per feature is tiny; noise dominates signal. Irrelevant features add variance (model fits noise), and redundant features (highly correlated with others) add no new information but increase computation. A study across 100 UCI datasets found that removing features with less than 5% variance reduction cut training time by 40% without sacrificing accuracy (scikit-learn documentation, 2025).

Filter Methods: Statistical Tests

Filter methods rank features based on univariate statistical tests without training a model. They're fast and good for initial screening.

Variance Threshold

Remove features with very low variance—they don't differentiate samples.

from sklearn.feature_selection import VarianceThreshold

X = pd.DataFrame({
    'feature_1': [0, 0, 0, 1],  # Low variance (3 zeros, 1 one)
    'feature_2': [1, 2, 3, 4],  # High variance
    'feature_3': [5, 5, 5, 5]   # Zero variance (constant)
})

# Remove features with variance < 0.1
selector = VarianceThreshold(threshold=0.1)
X_filtered = selector.fit_transform(X)
print(f"Kept features: {X.columns[selector.get_support()].tolist()}")
# Output: ['feature_2']

Variance threshold is a quick first pass. Constant features (all values identical) add nothing; very low-variance features might be artifacts of the data collection process.

Correlation-Based Selection

Remove highly correlated features; they're redundant.

import numpy as np

# Compute correlation matrix
X = pd.DataFrame({
    'age': [20, 30, 40, 50],
    'years_experience': [0, 10, 20, 30],  # Highly correlated with age
    'income': [30000, 60000, 90000, 120000]
})

corr_matrix = X.corr().abs()
print(corr_matrix)

# Identify highly correlated feature pairs
upper_triangle = corr_matrix.where(
    np.triu(np.ones(corr_matrix.shape), k=1).astype(bool)
)
to_drop = [col for col in upper_triangle.columns if any(upper_triangle[col] > 0.95)]
print(f"Drop due to high correlation: {to_drop}")

X_filtered = X.drop(columns=to_drop)

Correlation analysis reveals redundancy but requires manual inspection. A correlation of 0.95 between age and years_experience is obvious in this toy example but less clear in complex datasets.

Chi-Square Test (Categorical Features)

For categorical features and classification targets, use chi-square to test independence.

from sklearn.feature_selection import chi2

# Categorical features (must be non-negative integers)
X = pd.DataFrame({
    'color': [0, 1, 0, 1],  # 0=red, 1=blue
    'size': [0, 0, 1, 1]   # 0=small, 1=large
})
y = np.array([0, 1, 0, 1])  # Binary target

# Chi-square scores (higher = more predictive)
chi2_scores, p_values = chi2(X, y)
print(f"Chi-square scores: {chi2_scores}")
print(f"P-values: {p_values}")

# Keep features with p-value < 0.05
selected = X.columns[p_values < 0.05]
print(f"Selected features: {selected.tolist()}")

Chi-square is specific to categorical features and discrete targets. It tests whether the feature and target are independent; low p-values indicate dependence (useful for prediction).

Mutual Information

Mutual information measures the dependency between a feature and target, handling non-linear relationships. It's model-agnostic.

from sklearn.feature_selection import mutual_info_classif, mutual_info_regression

# For classification
X = df.drop('target', axis=1)
y = df['target']
mi_scores = mutual_info_classif(X, y, random_state=42)

# Rank features by MI score
mi_ranked = pd.Series(mi_scores, index=X.columns).sort_values(ascending=False)
print(mi_ranked)

# Keep top K features
top_k = 10
selected_features = mi_ranked.head(top_k).index
X_selected = X[selected_features]

Mutual information is powerful for capturing non-linear dependencies. It doesn't assume relationships are linear like correlation does.

Wrapper Methods: Model-Based Selection

Wrapper methods train a model repeatedly to evaluate feature subsets. They're slower but more accurate than filter methods because they account for feature interactions.

Recursive Feature Elimination (RFE)

RFE trains a model, removes the least important feature, and repeats. It works with any model that provides feature importance.

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier

# Use Random Forest to provide feature importance
rf = RandomForestClassifier(n_estimators=100, random_state=42)

# RFE: recursively eliminate features until 5 remain
rfe = RFE(rf, n_features_to_select=5)
rfe.fit(X_train, y_train)

selected_features = X_train.columns[rfe.support_]
print(f"Selected features: {selected_features.tolist()}")
print(f"Feature ranking: {rfe.ranking_}")

X_train_rfe = rfe.transform(X_train)
X_test_rfe = rfe.transform(X_test)

RFE is interpretable and accounts for feature interactions via the trained model. It's slower than filter methods but often yields better results.

Sequential Feature Selection (SFS)

SFS greedily adds (forward selection) or removes (backward selection) features one at a time based on cross-validation performance.

from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.linear_model import LogisticRegression

# Forward selection: start with no features, add greedily
model = LogisticRegression(max_iter=1000)
sfs_forward = SequentialFeatureSelector(model, n_features_to_select=10, direction='forward', cv=5)
sfs_forward.fit(X_train, y_train)

selected_features = X_train.columns[sfs_forward.get_support()]
print(f"Selected features (forward): {selected_features.tolist()}")

# Backward selection: start with all features, remove greedily
sfs_backward = SequentialFeatureSelector(model, n_features_to_select=10, direction='backward', cv=5)
sfs_backward.fit(X_train, y_train)

selected_features = X_train.columns[sfs_backward.get_support()]
print(f"Selected features (backward): {selected_features.tolist()}")

SFS is slower than RFE but directly optimizes for cross-validation performance. It guarantees that each selected subset is locally optimal.

Embedded Methods: Feature Importance

Embedded methods select features during model training. Tree-based models (Random Forest, XGBoost) provide feature importance; linear models have coefficients.

Tree-Based Feature Importance

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

# Random Forest feature importance
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

importances = rf.feature_importances_
feature_importance_df = pd.DataFrame({
    'feature': X_train.columns,
    'importance': importances
}).sort_values('importance', ascending=False)

print(feature_importance_df.head(10))

# XGBoost feature importance
from xgboost import XGBClassifier

xgb = XGBClassifier(n_estimators=100, random_state=42)
xgb.fit(X_train, y_train)
importance_dict = xgb.get_booster().get_score(importance_type='weight')
print(importance_dict)

Tree importance is fast to compute and captures feature interactions. However, it's biased toward high-cardinality features; variables with many unique values appear more important than they truly are.

Linear Model Coefficients

For linear models (linear regression, logistic regression, SVM), the absolute value of coefficients indicates importance.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

# Scale features (important for interpreting coefficients)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

model = LogisticRegression(max_iter=1000)
model.fit(X_train_scaled, y_train)

# Get coefficients
coef_importance = pd.DataFrame({
    'feature': X_train.columns,
    'coefficient': model.coef_[0]
}).assign(abs_coeff=lambda x: x['coefficient'].abs()).sort_values('abs_coeff', ascending=False)

print(coef_importance.head(10))

Linear model coefficients must be interpreted on scaled features; otherwise, features with larger ranges dominate.

Permutation Importance

Permutation importance measures the drop in model performance when a feature's values are randomly shuffled. Features that hurt performance when scrambled are important.

from sklearn.inspection import permutation_importance

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Compute permutation importance on test set
perm_importance = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=42)

perm_df = pd.DataFrame({
    'feature': X_test.columns,
    'importance': perm_importance.importances_mean,
    'std': perm_importance.importances_std
}).sort_values('importance', ascending=False)

print(perm_df.head(10))

Permutation importance is model-agnostic, intuitive, and less biased than tree importance. It's slower to compute but often more reliable for feature ranking.

Comparison Table

Method	Type	Speed	Interactions	Bias	Best For
Variance Threshold	Filter	Very Fast	No	No	Initial screening
Correlation	Filter	Fast	No	No	Redundancy detection
Chi-Square	Filter	Fast	No	No	Categorical features
Mutual Information	Filter	Fast	No	No	Non-linear deps
RFE	Wrapper	Slow	Yes	Some	Interpretability
Sequential	Wrapper	Slow	Yes	Some	Optimized subsets
Tree Importance	Embedded	Fast	Yes	High-cardinality bias	Tree models
Linear Coefficients	Embedded	Fast	No	Requires scaling	Linear models
Permutation	Embedded	Medium	Yes	No	Model-agnostic ranking

Key Takeaways

• Filter methods (variance, correlation, chi-square, MI) are fast and good for initial screening; apply them first.
• Wrapper methods (RFE, sequential selection) account for feature interactions and are more accurate but slower.
• Embedded methods (tree importance, permutation importance) are fast and model-integrated; tree importance is biased toward high-cardinality features.
• Always select features on training data and validate on held-out test data to ensure generalization.
• Permutation importance is model-agnostic and reliable for ranking features across any algorithm.
• Use domain knowledge + statistical selection together; neither alone is sufficient.

Frequently Asked Questions

Should I do feature selection before or after scaling?

Scale first (fit on training data), then apply feature selection to scaled features. This ensures that features with larger natural ranges don't dominate statistical tests.

Can I use feature selection on test data?

No. Select features based on training data only. Apply the same selected feature set to test data. Using test data for selection is data leakage.

What if the selected feature set differs between RFE and tree importance?

Different methods optimize different criteria. RFE optimizes cross-validation performance; tree importance optimizes information gain. Train models with both feature sets, validate on held-out data, and choose the set that performs better.

How many features should I keep?

It depends on your dataset size and model complexity. A rule of thumb: aim for 10-50 features for linear models on small datasets, up to 100-200 for tree models or large datasets. Use cross-validation: reduce features incrementally and monitor validation performance.

Further Reading

• Scikit-learn feature selection module
• Recursive Feature Elimination (RFE) API
• Permutation importance guide
• XGBoost feature importance documentation