Gradient Boosting Libraries: XGBoost & LightGBM
Why boosted trees usually win on table data, how XGBoost, LightGBM and scikit-learn compare, the parameters that matter, early stopping, categorical features, and explaining the model.
Why boosted trees
On table data (rows and columns like a spreadsheet), gradient-boosted decision trees are usually the strongest model with the least tuning: they handle mixed feature types, missing values, non-linear effects and interactions, and need no feature scaling. The idea, adding small trees that each fix the previous ones' errors, is in Ensembles. This topic is about the libraries.
Always start with a simple baseline (for example logistic regression) so you know how much the boosted model actually adds.
Three libraries, one idea
| scikit-learn HistGradientBoosting | XGBoost | LightGBM | |
|---|---|---|---|
| Number of trees | max_iter | n_estimators | n_estimators |
| Step size | learning_rate | learning_rate (eta) | learning_rate |
| Tree size | max_leaf_nodes, max_depth | max_depth (grows level by level) | num_leaves (grows leaf by leaf) |
| Minimum data per leaf | min_samples_leaf | min_child_weight | min_child_samples |
| Row / column sampling | (not built in) | subsample, colsample_bytree | subsample (+ subsample_freq), colsample_bytree |
| Regularisation | l2_regularization | reg_lambda, reg_alpha, gamma | reg_lambda, reg_alpha |
| Categorical features | categorical_features | enable_categorical=True + pandas category | pandas category columns |
| Early stopping | early_stopping=True | early_stopping_rounds + eval_set | callbacks=[lgb.early_stopping(50)] + eval_set |
All three use fast histogram-based splitting and handle missing values natively. Results are usually close; LightGBM is often fastest on large data, XGBoost is very widely used, and scikit-learn's version needs no extra install (and runs in this browser).
The parameters that matter
- Learning rate and number of trees go together: a smaller learning rate needs more trees but usually generalises a little better. Fix a learning rate (0.03–0.1) and let early stopping on validation data choose the number of trees.
- Tree size (
num_leaves15–127, ormax_depth3–8): bigger trees capture more interactions and overfit sooner. - Minimum data per leaf and sampling (
subsample,colsample_bytreearound 0.7–0.9) regularise.
Tune a small grid or a random search on validation data (or cross-validation), then refit on train + validation and evaluate on the test set once. Bigger searches rarely add much; better features usually do.
Try it: learning rate, tree size and early stopping
Watch training and validation loss as trees are added. Find where early stopping would stop, and how the learning rate and tree size change that point.
Interactive: this part runs in your browser.
Code example: boosting with early stopping, tuned properly
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Explaining a boosted model
- Built-in importance: "split" importance counts how often a feature is used and is biased toward features with many values; "gain" importance is better but still global and can mislead with correlated features.
- Permutation importance (
sklearn.inspection.permutation_importance) on validation data: how much the score drops when a feature is shuffled. - SHAP values (
shap.TreeExplainer) explain each prediction as contributions from each feature; summary plots show global patterns. Notebook 15 shows both. - Importance shows what the model uses, not what causes the outcome.
Hands-on: XGBoost and LightGBM in Colab (notebook 15)
The Adult income dataset: a baseline, the three libraries side by side with early stopping, a tuning search, a final test, feature importance and SHAP.
Interactive: this part runs in your browser.
Cheat sheet: XGBoost & LightGBM
The commands you will use most, on one screen. Come back to it while you work.
Interactive: this part runs in your browser.
Quiz
1. On a table of customer records with mixed feature types, a strong default model is…
2. Best way to choose the number of trees?
3. LightGBM's main tree-size parameter is…
4. Do boosted trees need standardised features?
5. High SHAP importance for a feature means…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- XGBoost documentation ↗ parameters, early stopping, categorical data
- LightGBM documentation ↗ parameters and tuning guide
- scikit-learn: Histogram-based gradient boosting ↗ the browser-friendly equivalent
- SHAP documentation ↗ explaining tree models