23  Target Encoding

Target encoding (also called mean encoding, likelihood encoding, or impact encoding) is a method that maps the categorical levels to probabilities of your target variable (Micci-Barreca 2001). This method is in some ways quite similar to frequency encoding. We are taking a single categorical variable, and turning it into a single numeric categorical variable.

This is a trained and supervised method since we are using the outcome of our modeling problem to guide the way this method is estimated. In the most simple formulation, target encoding is done by replacing each level of a categorical variable with the mean of the target variable within said level. The target variable will typically be the outcome, but that is not necessarily a requirement.

TODO

show motivated example of what happens when this method is applied to constant predictor. It would take full information of the outcome and become a perfect predictor.

Consider the following example data set

cuteness animal
1 dog
5 cat
9 cat
3 cat
2 dog
4 horse

If we were to calculate target encoding on animal using cuteness as the target, we would first need to calculate the mean of cuteness within each

animal math mean
dog (1 + 2) / 2 1.500000
cat (5 + 9 + 3) / 3 5.666667
horse 4 / 1 4.000000

Taking these means we can now use them as an encoding

cuteness animal
1 1.500000
5 5.666667
9 5.666667
3 5.666667
2 1.500000
4 4.000000

From the above example, we notice 3 things. Firstly, once the calculations have been done, applying the encoding to new data is a fairly easy procedure as it amounts to a left join.

Lastly, how will this method handle unseen levels?

Let us think about the unseen levels first. If we have no information about a given class. This could happen in at least two different ways. Because the level is truly unseen because the company was just started and wasn’t known in the training data set. Or because the known level wasn’t observed, e.i. no Sundays in the training data set. Regardless of the reason, we will want to give these levels a baseline number. For this, we can use the mean value of the target, across all of the training data set. So for our toy example, we have a mean cuteness of 4, which we will assign to any new animal.

This value is by no means a good value, but it is an educated guess that can be calculated with ease. This also means that regardless of the distribution of the target, these values can be calculated.

We have so far talked about target encoding, from the perspective of a regression task. But target encoding is not limited to numeric outcomes, but can be used in classification settings as well. In the classification setting, where we have a categorical outcome, instead of calculating the mean of the target variable, we need to figure something else out. It could be calculating the probability of the first level, or we could try to go more linear by converting them to log odds. Now everything works as before.

TODO

find good example

A type of data that is sometimes seen is hierarchical categorical variables. Typical examples are city-state and region-subregion. With hierarchical categorical variables you often end up with many levels, and target encoding can be used on such data, and can even use the hierarchical structure.

The encoding is calculated like normal, but the smoothing is done on a lower level than at the top. So instead of adjusting by the global mean, you smooth by the level below it.

23.2 Low count groups

Target encoding as described in this chapter, doesn’t apply smoothing to the statistics that are calculated. Depending on the data it won’t have too bad of a negative effect. This subsection goes over what happens if one or more of the groups have few counts. It will at the same time act as the main motivation, for why you shouldn’t be using the base form of target encoding as described in this chapter, and instead use one with smoothing as seen in Chapter 27.

Suppose we have the following data.

target predictor
A 0.4
A 0.6
A 0.5
A 0.2
A 0.5
B 0.2
B 0.3
B 0.2
B 0.4
B 0.6
C 0.1

Calculating the means gives us the following encoding.

target predictor
A 0.44
B 0.34
C 0.10

However, how much can we trust this encoding? C only has a single value and A and C both have 5. Maybe 0.1 is a good estimate of the target means under C, but it could also be an outlier of the distribution itself. Taking a single value of A would give a value between 0.2 and 0.6.

We are overfitting to the training data set, by having a single observation for C alone determine the encoding. Smoothing would take the global mean (here 0.36) into account by having the mean encoding deviate from the global mean. Target levels with many counts would use a mean close to the class mean, and target levels with few counts would use a mean close to the global mean.

We see this in sports and online reviews. An athlete that scored 1 out of 1 goal isn’t a reliable statistic, as the athlete that scores 80 out of 100. Like-wise, 10 five-star reviews aren’t a good as 100 five-star reviews.

23.3 Pros and Cons

23.3.1 Pros

  • Can deal with categorical variables with many levels
  • Can deal with unseen levels in a sensible way

23.3.2 Cons

  • Can be prone to overfitting

23.4 R Examples

The embed package comes with a couple of functions to do target encoding, we will look at step_lencode_glm() as it applied the method best described by this chapter. These functions are named such because they likelihood encode variables, and because the encodings can be calculated using no intercept models.

TODO

find a better data set

library(recipes)
library(embed)

data(ames, package = "modeldata")

rec_target <- recipe(Sale_Price ~ Neighborhood, data = ames) |>
  step_lencode_glm(Neighborhood, outcome = vars(Sale_Price)) |>
  prep()

rec_target |>
  bake(new_data = NULL)
# A tibble: 2,930 Γ— 2
   Neighborhood Sale_Price
          <dbl>      <int>
 1      145097.     215000
 2      145097.     105000
 3      145097.     172000
 4      145097.     244000
 5      190647.     189900
 6      190647.     195500
 7      324229.     213500
 8      324229.     191500
 9      324229.     236500
10      190647.     189000
# β„Ή 2,920 more rows

And we see that it works as intended, we can pull out the exact levels using the tidy() method

rec_target |>
  tidy(1)
# A tibble: 29 Γ— 4
   level                value terms        id               
   <chr>                <dbl> <chr>        <chr>            
 1 North_Ames         145097. Neighborhood lencode_glm_Bp5vK
 2 College_Creek      201803. Neighborhood lencode_glm_Bp5vK
 3 Old_Town           123992. Neighborhood lencode_glm_Bp5vK
 4 Edwards            130843. Neighborhood lencode_glm_Bp5vK
 5 Somerset           229707. Neighborhood lencode_glm_Bp5vK
 6 Northridge_Heights 322018. Neighborhood lencode_glm_Bp5vK
 7 Gilbert            190647. Neighborhood lencode_glm_Bp5vK
 8 Sawyer             136751. Neighborhood lencode_glm_Bp5vK
 9 Northwest_Ames     188407. Neighborhood lencode_glm_Bp5vK
10 Sawyer_West        184070. Neighborhood lencode_glm_Bp5vK
# β„Ή 19 more rows

23.5 Python Examples

We are using the ames data set for examples. {sklearn} provided the TargetEncoder() method we can use. For this to work, we need to remember to specify an outcome when we fit().

from feazdata import ames
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import TargetEncoder

ct = ColumnTransformer(
    [('target', TargetEncoder(target_type="continuous"), ['Neighborhood'])], 
    remainder="passthrough")

ct.fit(ames, y=ames[["Sale_Price"]].values.flatten())
ColumnTransformer(remainder='passthrough',
                  transformers=[('target',
                                 TargetEncoder(target_type='continuous'),
                                 ['Neighborhood'])])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
ct.transform(ames).filter(regex="target.*")
      target__Neighborhood
0               145110.156
1               145110.156
2               145110.156
3               145110.156
4               190636.427
...                    ...
2925            162269.818
2926            162269.818
2927            162269.818
2928            162269.818
2929            162269.818

[2930 rows x 1 columns]