T2-L04 — Gradient Boosting

GBM regression step-by-step · Learning rate · Residual shrinkage · XGB vs LGBM

GBM Regression — Animate Iterations

F_m(x) = F_{m-1}(x) + η · hₘ(x)  |  hₘ fit to residuals rᵢ = yᵢ − F_{m-1}(xᵢ)
0 0.3

Iteration Log

Learning Rate η — Effect on Convergence

So sánh Train MSE và Val MSE cho 3 learning rates theo số trees. Optimal: η nhỏ + nhiều trees + early stopping.

η=0.1: chậm nhưng smooth convergence, ít overfit. η=0.5: nhanh đạt minimum nhưng volatile. η=1.0: overfit ngay sau vài iterations.

XGBoost vs LightGBM vs sklearn GBM

Simulated benchmark: accuracy và training time theo dataset size.