Gradient Boosting

Definition

Gradient Boosted Decision Trees build an ensemble by repeatedly adding new models that predict the errors of the current ensemble. XGBoost is the dominant implementation for conventional (tabular/structured) data.

---

Core Ideas

The boosting cycle

1. Start from a naive base prediction (even a wildly inaccurate one).
2. Compute the errors of the current ensemble for each observation.
3. Train a new model to predict those errors.
4. Add it to the ensemble.
5. Repeat — a prediction is the sum of all models’ predictions.

Each round chips away at the previous round’s errors, so early inaccuracy is corrected over time.

Tuning parameters (XGBoost)

• n_estimators (100–1000) — number of boosting rounds; too low underfits, too high overfits.
• early_stopping_rounds — set n_estimators high and let this stop when validation stops improving (e.g. 5 straight non-improving rounds). Guards against overfitting and wasted iterations.
• learning_rate — a small rate with many estimators generally yields more accurate models but trains slower.
• n_jobs — parallelism (≈ number of CPU cores).

---

Relationships

• Machine Learning — a supervised ensemble method
• Neural Network — the main alternative; boosting usually wins on tabular data
• Data Leakage — high accuracy that’s “too good” often signals leakage, not skill
• AI & Machine Learning