Bias-Variance Tradeoff
The bias-variance tradeoff is the central challenge in supervised machine learning. Understanding it explains why models overfit or underfit, and guides the choice of model complexity, regularization, and training data size.
The Fundamental Problem
Every machine learning model makes errors on unseen data. These errors come from three sources:
Total Error = Bias^2 + Variance + Irreducible Noise
- Bias - Error from wrong assumptions in the model (underfitting) - Variance - Error from sensitivity to small fluctuations in training data (overfitting) - Irreducible Noise - Inherent randomness in the data that no model can eliminate
The goal is to find the sweet spot that minimizes total error.
High Bias: Underfitting
A model with high bias makes strong, oversimplified assumptions about the data:
- Training error is high - Validation error is also high (and similar to training error) - The model fails to capture the true patterns in the data
Examples: - Fitting a linear model to non-linear data - Using a decision tree with max_depth=1 for complex data - Using too few features
Fix: Use a more complex model, add more features, reduce regularization.
High Variance: Overfitting
A model with high variance is overly sensitive to the training data:
- Training error is very low - Validation/test error is much higher than training error - The model memorizes noise and specific training examples
Examples: - A deep decision tree with no depth limit - A neural network trained too long on small data - A polynomial regression with very high degree
Fix: Simplify the model, add regularization, get more training data, use dropout/early stopping.
Diagnosing with Learning Curves
Learning curves plot training and validation error as a function of training set size. They are the most powerful diagnostic tool for bias-variance issues:
- High Bias pattern - Both train and validation errors converge to a high value. Adding more data does not help much.
- High Variance pattern - Large gap between low training error and high validation error. Adding more data gradually closes the gap.
- Good fit pattern - Both errors converge to a low value with a small gap.
- If you see high bias, try a more complex model. If you see high variance, try regularization or more data.
Diagnosing and Fixing Bias-Variance Issues
Practical implementation of learning curves and regularization:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.datasets import make_regression
# Generate non-linear data
X, y = make_regression(n_samples=300, n_features=1, noise=20, random_state=42)
y = y + 0.5 * X.ravel()**2 # Add non-linearity
# ---- Compare underfitting vs overfitting vs good fit ----
configs = [
('Underfit (degree=1)', 1, 0.01),
('Good fit (degree=3)', 3, 1.0),
('Overfit (degree=15)', 15, 0.0001),
]
for name, degree, alpha in configs:
pipe = Pipeline([
('poly', PolynomialFeatures(degree=degree)),
('scaler', StandardScaler()),
('ridge', Ridge(alpha=alpha)),
])
train_sizes, train_scores, val_scores = learning_curve(
pipe, X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 10),
scoring='neg_mean_squared_error', n_jobs=-1
)
train_mse = -train_scores.mean(axis=1)
val_mse = -val_scores.mean(axis=1)
print(f"\n{name}")
print(f" Final Train MSE: {train_mse[-1]:.1f}")
print(f" Final Val MSE: {val_mse[-1]:.1f}")
gap = val_mse[-1] - train_mse[-1]
if train_mse[-1] > 500:
print(" Diagnosis: HIGH BIAS (underfitting)")
elif gap > 200:
print(" Diagnosis: HIGH VARIANCE (overfitting)")
else:
print(" Diagnosis: GOOD FIT")Regularization as the Solution
Regularization is the primary tool for controlling the bias-variance tradeoff. It adds a penalty to the loss function that discourages model complexity:
- L2 (Ridge) - Penalizes sum of squared weights. Shrinks all weights toward zero. Good default choice. - L1 (Lasso) - Penalizes sum of absolute weights. Drives some weights to exactly zero (feature selection). - Dropout - Randomly deactivates neurons during training (neural networks). Acts as ensemble averaging. - Early Stopping - Stop training when validation error starts increasing. Prevents overfitting in iterative algorithms.
The regularization strength (lambda or alpha) is a hyperparameter tuned via cross-validation.
Key Takeaways
- Total prediction error = Bias^2 + Variance + Irreducible Noise. The goal is to minimize the first two.
- High bias (underfitting): both train and val errors are high. Fix: more complex model, more features.
- High variance (overfitting): low train error, high val error. Fix: regularization, more data, simpler model.
- Learning curves are the best diagnostic tool - they reveal whether you have a bias or variance problem.
- Regularization (L1/L2, dropout, early stopping) is the primary technique for controlling variance.
Contact Us
Have a question or feedback? Fill out the form below or reach us directly at support@nvaitraining.com