What is gradient boosting?

Gradient boosting is a machine learning (ML) algorithm used for regression and classification tasks. Gradient boosting has become popular due to its ability to handle complex relationships in data and protect against overfitting. Using this technique, data scientists can improve the predictive accuracy and speed of their ML models. In this article, learn about gradient boosting, how to share the benefits of using this technique and three common use cases.

Gradient boosting is an ensemble ML technique that combines a collection of weak models into a single, more accurate and efficient predictive model. These weak models are typically decision trees, which is why the algorithms are commonly referred to as gradient boosted decision trees (GBDTs). Gradient boosting algorithms work iteratively by adding new models sequentially, with each new addition aiming to resolve the errors made by the previous ones. The final prediction of the aggregate represents the sum of the individual predictions of all the models. Gradient boosting combines the gradient descent algorithm and boosting method, with a nod to each component included in its name.

This training process leverages a strength-in-numbers approach, allowing data scientists to optimize arbitrary differentiable loss functions. Gradient boosting is used to solve complex regression and classification problems. With regression, the final result represents the average of all weak learners. When working with classification problems, the model’s final result can be computed as the class with the majority of votes from weak learner models.

Boosting and bagging are the two primary types of ensemble learning. Ensemble learning methods are distinguished by their collective approach, aggregating a group of base learners to generate more accurate predictions than any of the component parts could on its own. With boosting methods, the weak learner models are trained successively, with each individual model having made its contribution to the collective whole before the next one is brought in. Bagging techniques train the base learners in tandem.

Gradient boosting provides a good balance of accuracy, efficiency and scalability that can be widely applied to: 

• Classification: Predicting categories or classes (e.g., spam detection, fraud detection)
• Regression: Predicting numerical values (e.g., stock price prediction, sales forecasting). 
• Ranking: Ranking items based on their relevance or importance (e.g., search results, recommendations).

Other boosting techniques, such as AdaBoost and XGBoost, are also popular ensemble learning methods. Here’s how they work.

XGBoost is a turbocharged version of gradient boosting designed for optimal computational speed and scalability. XGBoosting uses multiple cores in the CPU to enable parallel learning during model training.

AdaBoost, or adaptive boosting, fits a succession of weak learners to the data. These weak learners are usually decision stumps, a decision tree with a single split and two terminal nodes. This technique works recursively,  identifying misclassified data points and automatically adjusting them to reduce training errors. AdaBoost repeats this process until it generates the strongest predictor.