86 Imbalanced Overview

86.1 Imbalanced Overview

Whether to keep this chapter in this book or not was considered, as the methods in this section are borderline feature engineering methods.

The potential problem with imbalanced data can most easily be seen in a classification setting. Suppose we want to predict if an incoming email is spam or not. Furthermore, we assume that the spam rate is low and around 1% of the incoming emails. If you are not careful, you can easily end up with a model that predicts “not spam” all the time, since it will be correct 99% of the time. This is a common scenario, and it happens all the time. One culprit could be that there isn’t enough information in the minority class to be able to distinguish it from the majority class. And that is okay, not all modeling problems are easy. But your model will do its best anyway.

There are several different ways to handle imbalanced data, we will list all the ways in broad strokes, and then cover the methods that we could count as feature engineering.

using the right performance metrics
weights
loss functions
Calibrating the predictions
Sampling the data, can be done naively and with more advanced methods

The example above showcases why accuracy as a metric isn’t a good choice when the classes are not uniformly represented. So you can look at other metrics, such as precision, recall, ROC-AUC, or Brier score. You will need to know what metric works best for your project Harrell (2017).

Another way we can handle this is by adding weights, either to the observations directly, or in the modeling framework as class weights. Giving your minority class high enough weight forces your model to consider them.

Related to the last point, some methods allow you, the user, to pass in custom objective functions, this can also be beneficial.

Even if your model performs badly by default, your classification model might still have good separation, just not around the 50% cut-off point. Changing the threshold is another way you can overcome an imbalanced data set.

Lastly, and the way that will be covered in this book, is sampling of the data. There are several different methods that we will cover in this book. These methods cluster somehow, so for some groups we only explain the general idea.

We can split these methods into two groups, under-sampling methods and over-sampling methods. In the under-sampling method, we are removing observations and in the over-sampling we are adding observations. Adding observations is usually done by basing the new observations on the existing observations, exactly or by interpolation.

Over-sampling methods we will cover are:

Under-sampling methods we will cover are:

Some methods do these methods together. We won’t consider those methods by themselves and instead let you know that you can do over-sampling followed by under-sampling if you choose. One-sided selection Kubat and Matwin (1997) is one of those methods, that combines Tomek link removal with condensed nearest neighbors.

TODO: I think we would be able to add some kind of list, regarding what the under sampling methods tries to do. Some tries to remove hard to classify observations, other try to remove easy to classify observations.

86.2 Extending to Multiple Classes

Most of the methods in this section were first introduced to solve the problem with imbalanced data in binary classification settings. While binary classification comes up a lot, it would be a shame if we had to limit ourselves to binary classification settings to use these methods.

This section will go over each of the ways methods in this section are being extended or modified to handle multiple outcome classes. There are some methods that are naturally created to work with multiple classes, and will thus not need to be expanded. One of the ways they do this is by framing the questions based on “same class” and “not same class”, without resorting to minority and majority classes. The question “how many points are in a different class than me” doesn’t restrict to 2 classes. The other way to handle multiple classes is to treat each class as its own unit. Observations are added or removed based on the observations only within that class. This naturally extends itself to multiple classes.

We can expand 2 class methods in two ways:

The first way is the one-vs-rest strategy. For each class we treat it as the minority/majority class and then lump all other classes together to form the other class. Turning a multi class setting into a series of two class method applications, being applied once per class.

We can also apply the one-vs-rest idea, but only applying it once, using the smallest class as the minority and letting all the other ones act as a pooled majority. This gives us a single application of the method, which is why it isn’t a true one-vs-rest.

Each chapter in this section will list which of these scenarios apply.

86.3 Changes to the Data Distribution

The main reason why you need to be careful of these methods, is that they change the distribution of the data set you are working with. By adding or removing observations to the data set in a systematic way, you are changing the modeling problem you have. The rationale behind some of these methods, that if you remove hard-to-classify and/or noisy observations, then you can get a more efficient decision boundary, is technically correct. But that decision boundary will still struggle on the hard-to-classify and noisy observations Goorbergh et al. (2022).