Data Transformation & Feature Engineering
Reshaping clean data into the form models learn from best.
Why transform data at all?
Once data are clean, they are correct, but not necessarily in the best form for a model. Every algorithm makes assumptions about its inputs. Distance-based methods such as k-NN and SVMs assume that a difference of one unit means the same thing in every feature. Linear models assume that each feature affects the prediction in a straight-line way. Gradient-based training works best when features have similar scales. And almost every algorithm needs numbers, not text or dates. Data transformation reshapes the clean data so that these assumptions hold, and feature engineering goes further by creating new, more informative features.
The transformations in this topic fall into four groups, shown in the diagram. Scaling changes the range of a numeric feature without changing its shape. Shape transforms, such as taking logarithms, change the shape of a feature's distribution, typically to tame a long tail. Encoding turns categories, dates and text into numbers. And feature creation builds new features from existing ones: ratios, interactions, date parts, counts and aggregates.
One principle applies to every transformation, and it is worth stating before anything else. A transformation that learns something from the data (the mean used for scaling, the categories seen by an encoder, the vocabulary of a text vectoriser, the best power for a power transform) must learn it from the training data only. In scikit-learn's language, you call fit on the training set and then transform on the training, validation and test sets alike. Learning these quantities from all the data, before splitting, is one of the most common causes of data leakage.
Scaling numerical features
Scaling puts numeric features onto comparable ranges. Let's compare the main methods on one small example: a feature with the five values 2, 4, 6, 8 and 100. Its mean is 24, its standard deviation is about 38.1 (dividing by , as scikit-learn does), its median is 6, and its interquartile range is .
Standardisation (the z-score) subtracts the mean and divides by the standard deviation, . The value 2 becomes , 8 becomes , and 100 becomes . Every standardised feature has mean 0 and standard deviation 1. It is the default choice for linear models, SVMs, PCA and neural networks. Notice, though, that the four ordinary values have been squeezed into a narrow band between −0.58 and −0.42, because the outlier inflated the standard deviation.
Min–max scaling maps the smallest value to 0 and the largest to 1, . Here 2 becomes 0, 100 becomes 1, and 8 becomes only . A single extreme value has pushed every ordinary value into the bottom 6% of the range. Min–max scaling is natural for bounded data such as pixel intensities, but fragile when outliers are present.
Robust scaling uses the median and the interquartile range instead, . Here 2 becomes , 8 becomes , and 100 becomes . The ordinary values now have a sensible spread, and the outlier is simply left far away rather than distorting everyone else.
Two other scalers are occasionally useful. Max-abs scaling divides by the largest absolute value, keeping zeros at zero, which matters for sparse data such as word counts. And normalisation in the strict sense rescales each row (each example) to have length 1, which is useful when only the direction of a vector matters, as with text represented by word counts compared using cosine similarity.
| Method | Formula | Value 2 | Value 8 | Value 100 | Good for |
|---|---|---|---|---|---|
| Standardise | −0.58 | −0.42 | 2.00 | default for most models | |
| Min–max | 0 | 0.06 | 1 | bounded data (pixels) | |
| Robust | −1 | 0.5 | 23.5 | data with outliers |
Which models need scaling? Anything that uses distances (k-NN, k-means, SVMs with RBF kernels), anything trained by gradient descent (neural networks, logistic regression), anything with a penalty on weight size (ridge, lasso), and PCA. Tree-based models (decision trees, random forests, gradient boosting) do not need it, because they split on one feature at a time using thresholds, and rescaling a feature doesn't change which threshold separates the data best.
| income | standard | min–max | robust |
|---|---|---|---|
| 48,000 | -0.52 | 0.000 | -1.00 |
| 52,000 | -0.51 | 0.004 | -0.56 |
| 57,000 | -0.49 | 0.010 | 0.00 |
| 61,000 | -0.48 | 0.014 | 0.44 |
| 950,000 | 2.00 | 1.000 | 99.22 |
The outlier (red) drags the mean and std, so standard scaling squashes the four ordinary incomes together (−0.52 to −0.48). Min–max is even worse: they all land within 0.014 of zero. Robust scaling uses the median and IQR, which ignore the outlier, so the ordinary incomes stay spread out (−1.00 to 0.44), and only the outlier gets an extreme value (99.22).
Try it: scalers vs shape transforms
Pick a feature and a transform and compare the histograms. Watch the difference between transforms that change only the range (standardise, min–max, robust) and those that change the shape (log, Box-Cox).
Interactive: this part runs in your browser.
Fixing skewed distributions: log, power transforms and binning
Many real quantities, such as incomes, house prices, city populations, website visits and reaction times, are right-skewed: most values are modest, but a long tail stretches out to very large values. Skew causes several problems. The extreme values dominate squared-error losses, linear models struggle because the relationship with the target is often closer to multiplicative than additive, and scaling alone can't help, because scaling never changes a distribution's shape.
The classic fix is the log transform. Because , the logarithm turns multiplication into addition: equal ratios become equal distances. Incomes of $20,000, $200,000 and $2,000,000 are wildly unequal on the original scale, but on a log scale (base 10) they become 4.3, 5.3 and 6.3, evenly spaced. The log also has a natural interpretation in models: a one-unit change in corresponds to multiplying by a constant factor, so effects become "per percentage change" rather than "per dollar". Because is undefined, data containing zeros (such as counts) are usually transformed with , written log1p. A gentler alternative, the square root, is often used for counts.
Power transforms generalise this idea by searching for the best power automatically. The Box-Cox transform, for strictly positive data, uses a parameter : leaves the data unchanged (apart from a shift), is like a square root, is the logarithm, and negative values transform even more strongly. The best is chosen by maximum likelihood to make the result as close to normally distributed as possible. The Yeo-Johnson transform extends the same idea to data that include zeros and negative numbers. A third option, the quantile transform, maps each value to its rank and then onto a target distribution (uniform or normal), which forces any distribution into the desired shape, at the cost of distorting distances between values.
Binning (discretisation) takes the opposite approach: it replaces a numeric value with the interval it falls into, such as turning age into "18–29", "30–44", "45–64" and "65+". Equal-width bins divide the range into intervals of the same size, while equal-frequency (quantile) bins put the same number of examples in each. Binning can capture non-linear effects in a linear model, reduce the influence of outliers, and make results easier to explain, but it throws information away (a 30-year-old and a 44-year-old become identical), so it should be used deliberately. Tree-based models effectively learn their own bins and rarely need it.
A final caution about transforming the target. If you train a regression model to predict instead of , its predictions must be converted back with , and the result estimates the median of rather than its mean, which is systematically a little low for skewed data. Libraries such as scikit-learn's TransformedTargetRegressor handle the back-transformation for you.
Interactive: this part runs in your browser.
Encoding categorical features
Most algorithms need numbers, so categorical features such as colour, city or product type must be encoded. The right encoding depends on whether the categories have a natural order and on how many categories there are. Let's use a small example throughout: a "size" column with values S, M and L, and a "colour" column with values red, green and blue.
Ordinal encoding replaces each category with an integer. For size, S = 0, M = 1, L = 2 is a good encoding, because the numbers respect the real order. For colour, however, red = 0, green = 1, blue = 2 would be a mistake for most models, because it invents an order that doesn't exist: a linear model would conclude that blue is "twice as green as green" and that green lies halfway between red and blue. Use ordinal encoding only for ordinal categories, or for tree-based models, which tolerate arbitrary orders reasonably well.
One-hot encoding is the standard choice for nominal categories (those without an order). It creates one new yes/no column for each category, with a 1 in the column of the row's category and 0 in the others: red becomes (1, 0, 0), green (0, 1, 0) and blue (0, 0, 1). No false order is implied. For linear models, one of the columns is usually dropped to avoid the dummy-variable trap, so that a row of all zeros represents the dropped "reference" category. The weakness of one-hot encoding is high cardinality: a postcode column with 3,000 values would create 3,000 mostly-zero columns.
For high-cardinality features there are several alternatives. Frequency (count) encoding replaces each category with how often it occurs, a single column that uses no labels and so can't leak, though different categories with the same count become indistinguishable. Target encoding replaces each category with a smoothed average of the target for that category; it is very powerful, but because it uses the labels, it must be computed out-of-fold (from data other than the row being encoded), or the model will simply learn to read the answer off the encoding. Feature hashing maps categories into a fixed number of columns using a hash function, accepting occasional collisions in exchange for bounded memory. And in neural networks, each category can be given a learned embedding: a short vector of numbers that the network adjusts during training, so that similar categories end up with similar vectors.
Two practical details cause many bugs. First, new categories will appear in real use that were never seen during training, so encoders should be told what to do with them (scikit-learn's handle_unknown="ignore" encodes an unseen category as all zeros). Second, very rare categories make unreliable features, so it is common to group categories that appear fewer than a set number of times into a single "other" category.
| city | bought |
|---|---|
| Brisbane | 1 |
| Brisbane | 0 |
| Brisbane | 1 |
| Sydney | 0 |
| Sydney | 0 |
| Melbourne | 1 |
(mean of bought per city)→
| citycity_te | bought |
|---|---|
| Brisbane0.67 | 1 |
| Brisbane0.67 | 0 |
| Brisbane0.67 | 1 |
| Sydney0.00 | 0 |
| Sydney0.00 | 0 |
| Melbourne1.00 | 1 |
Warning: Melbourne's value comes from a single row and simply copies that row's answer. This is target leakage. Real target encoding uses smoothing towards the overall mean and cross-fitting, so a row never sees its own label.
| city |
|---|
| Brisbane |
| Brisbane |
| Brisbane |
| Sydney |
| Sydney |
| Melbourne |
(share of rows)→
| citycity_freq |
|---|
| Brisbane0.50 |
| Brisbane0.50 |
| Brisbane0.50 |
| Sydney0.33 |
| Sydney0.33 |
| Melbourne0.17 |
| Encoding | Columns created | Best for | Watch out for |
|---|---|---|---|
| Ordinal | 1 | ordered categories; trees | invents an order for nominal data |
| One-hot | one per category | nominal, few categories | too many columns at high cardinality |
| Frequency | 1 | high cardinality, no leakage | ties between categories |
| Target | 1 | high cardinality, strong signal | leakage: compute out-of-fold |
| Hashing / embeddings | fixed | very high cardinality; deep learning | collisions; less interpretable |
Try it: encode the same column five ways
Choose a column and an encoding to see the resulting numbers. Try ordinal encoding on colour (no natural order) versus size (ordered), and target encoding on city with the smoothing set to 0.
Interactive: this part runs in your browser.
Creating new features: dates, text, interactions and aggregates
Feature engineering means creating new inputs that make the pattern easier for a model to see. It is where domain knowledge pays off most, and a single well-chosen feature can improve a model more than any amount of tuning.
Dates and times. A raw timestamp is almost useless to most models, but the information hidden inside it is often very valuable. You can extract the year, month, day of the week, hour, whether it is a weekend or public holiday, and the time elapsed since some event, such as days since a customer's last purchase. Cyclical quantities need special treatment. If hour of the day is stored as a number from 0 to 23, the model believes 23:00 and 00:00 are 23 units apart, when they are really one hour apart. The fix is to place the hour on a circle using two features:
On the circle, 23:00 and 00:00 are neighbours, exactly as they should be. The same trick works for day of the week (period 7), month (period 12) and compass directions (period 360°).
Text. The simplest way to turn documents into numbers is the bag of words: build a vocabulary of all the words seen in the training data, and represent each document by how many times each word appears in it. Common words such as "the" appear everywhere and carry little information, so the TF-IDF weighting multiplies each count by a factor that is large for rare words and small for common ones (the formula is in the Intermediate section). For example, in a collection of support emails, the word "refund" might be rare overall but very frequent in a particular email, giving it a high TF-IDF score in that email. Modern systems often replace these counts with embeddings produced by a pretrained language model, which capture meaning rather than exact words.
Interactions and polynomial features. Sometimes the effect of one feature depends on another: a discount might boost sales much more on weekends. A linear model can't discover this by itself, but you can give it a product feature, discount × weekend. Polynomial features add all such products and powers up to some degree automatically, letting a linear model fit curves and interactions, at the cost of many more features.
Ratios and domain features. Often the most useful features combine raw columns in a meaningful way: debt divided by income, price per square metre, body-mass index from height and weight, or the change from last month. Such features express relationships that experts already know matter.
Aggregates. When data have a group structure, summaries of the group are powerful: a customer's average order value, the number of transactions on a card in the last hour, or the average price of houses in the same suburb. As always, these must be computed using only information that would be available at prediction time, and only from training data, or they will leak.
Try it: why cyclical features need sine and cosine
Pick two hours. Compare the distance between them as raw numbers (left) with their distance on the circle (right). Try 23:00 and 01:00.
Interactive: this part runs in your browser.
The mathematics of transformations
Scalers only slide and stretch a feature, so they can't change its shape. To change the shape, you need a curved function such as a log or a power, and maximum likelihood can pick the best power for you. Logs also turn "multiplies by" into "adds", and placing time on a circle makes midnight sit next to 11 pm.
🔤 Symbols in this section
| affine (scaling) transform | |
| Box-Cox power parameter | |
| variance after transforming with λ | |
| true time gap (the short way round) | |
| term frequency, document frequency | |
| number of polynomial features |
Affine transformations preserve shape. Standardisation, min–max and robust scaling all have the form for constants and . Such an affine map shifts and stretches every value in the same way, so it preserves the order of the values and the relative distances between them; in particular, the skewness is unchanged. That is why no scaler can fix a skewed distribution. It also explains why tree models don't care about scaling: if is split at threshold , the scaled feature is split at with exactly the same result. For standardisation, and , and the rules for expectation and variance confirm that and .
The Box-Cox transform for positive is
The two cases join smoothly, because (write for small ). The parameter is chosen by maximum likelihood, assuming the transformed data are normal. After accounting for the stretching the transformation applies to the data (its "Jacobian"), the log-likelihood to maximise is
where is the variance of the transformed data. In practice is found by a simple search, which is what the demo does.
Why logs make relationships additive. If a target behaves multiplicatively, (for example, sales proportional to price raised to some elasticity), taking logs gives , which is exactly a linear model in the logged features. The coefficients and then read as elasticities: a 1% increase in changes by about %.
Cyclical encoding preserves the true distance. Placing hour at the point on a unit circle makes the straight-line distance between two hours a smooth function of the true time gap (measured the short way round the clock):
So 23:00 and 01:00 () are at distance , the same as 10:00 and 12:00, exactly as it should be.
TF-IDF. For a term in a document , within a collection of documents, the term frequency counts how often appears in , and the document frequency counts how many documents contain . Then
A word that appears in every document has and is ignored, while a word in only a few documents gets a large weight. Worked example: in emails, "refund" appears in 10 of them, so its IDF is ; if it appears 3 times in one email, its TF-IDF there is . The word "the", appearing in all 1000, gets . (Libraries add small smoothing constants to avoid division by zero.)
How many polynomial features? With original features, the number of polynomial features up to degree (including the constant) is . With and that is ; with and it is . This explosive growth is why polynomial features are usually combined with regularisation or used only for a handful of carefully chosen features.
Advanced: leakage-safe pipelines and feature selection
Every transformation that learns from data must learn only from the training part, and the safest way to guarantee that is to put everything inside one pipeline that cross-validation refits for each fold. The same rule applies when choosing which features to keep.
🔤 Symbols in this section
| number of folds for out-of-fold encoding | |
| count and target mean of category c | |
| smoothing strength | |
| mutual information between a feature and the target |
The fit/transform contract. Every scikit-learn transformer has two phases. fit learns parameters from data (means, categories, vocabularies, λ values), and transform applies the learned parameters to any data. A Pipeline chains transformers with a final model so that calling fit on the pipeline fits each step on the training data in turn, and a ColumnTransformer applies different transformers to different columns (scaling to numbers, one-hot encoding to categories, TF-IDF to text) and joins the results. When the pipeline is passed to cross_val_score or GridSearchCV, every fold refits every step on that fold's training portion, which makes the whole process leakage-free by construction. Transformer settings can then be tuned like model hyperparameters, for example choosing between scalers or between encodings by cross-validation.
Target encoding done properly. Encoding a category by the mean target of its training rows leaks: each row's own label contributes to its own feature. The standard remedy is out-of-fold encoding: split the training data into folds and encode each fold using statistics computed on the other folds, combined with the smoothing formula from the Data topic. The validation and test data are encoded with statistics from the full training set. scikit-learn's TargetEncoder implements exactly this "cross-fitting".
Feature selection. After engineering many features, it is often worth keeping only the useful ones, which reduces overfitting, speeds up training and makes models easier to interpret. Methods fall into three families. Filter methods score each feature on its own, before any model is trained: removing features with near-zero variance, ranking features by their correlation with the target, or by their mutual information with it, , which measures how much knowing the feature reduces uncertainty about the target and also captures non-linear relationships. Wrapper methods search over subsets of features using a model's cross-validated performance; recursive feature elimination repeatedly trains a model and removes the least important feature. They capture interactions between features but are expensive. Embedded methods select features as part of training: the lasso drives the weights of unhelpful features to exactly zero, and tree ensembles provide importance scores (preferably permutation-based). Whatever the method, feature selection must happen inside cross-validation; selecting features on the full dataset and then cross-validating gives optimistically biased scores, because the selection has already seen the validation folds.
Training/serving skew. In deployed systems, a classic failure is that features are computed one way during training (in a notebook, from a historical table) and another way in production (in application code, in real time), so the model receives subtly different inputs from the ones it learned on. Keeping the transformations inside the saved pipeline, or computing features once in a shared feature store, ensures that training and serving use identical logic, and point-in-time correct joins ensure that historical features only use information that was available at that moment.
Code: one pipeline for numbers, skew, categories, dates and text
A single ColumnTransformer handles every feature type from this topic, and the whole thing is cross-validated without leakage.
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import (StandardScaler, RobustScaler, PowerTransformer, OneHotEncoder,
OrdinalEncoder, FunctionTransformer, TargetEncoder)
from sklearn.impute import SimpleImputer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold
rng = np.random.default_rng(0)
n = 1200
df = pd.DataFrame({
"income": np.exp(rng.normal(10.8, 0.6, n)), # right-skewed
"age": rng.normal(40, 12, n).clip(18, 85),
"size": rng.choice(["S", "M", "L"], n), # ordinal
"colour": rng.choice(["red", "green", "blue"], n), # nominal
"suburb": rng.choice([f"suburb_{i}" for i in range(150)], n), # high cardinality
"hour": rng.integers(0, 24, n), # cyclical
"note": rng.choice(["fast delivery great", "refund please late", "great product",
"late again refund", "love it great"], n), # text
})
logit = (0.8 * (np.log(df["income"]) - 10.8) + 0.03 * (df["age"] - 40)
+ np.where(df["note"].str.contains("refund"), -1.2, 0.4) + np.cos(2 * np.pi * df["hour"] / 24))
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
def cyclical(X, period=24):
a = 2 * np.pi * np.asarray(X, dtype=float) / period
return np.c_[np.sin(a), np.cos(a)]
prep = ColumnTransformer([
("skewed", make_pipeline(SimpleImputer(strategy="median"), PowerTransformer(method="yeo-johnson")), ["income"]),
("numeric", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), ["age"]),
("ordinal", OrdinalEncoder(categories=[["S", "M", "L"]]), ["size"]),
("nominal", OneHotEncoder(handle_unknown="ignore", drop="first"), ["colour"]),
("target", TargetEncoder(random_state=0), ["suburb"]), # out-of-fold internally
("cyclic", FunctionTransformer(cyclical, feature_names_out=lambda t, f: ["hour_sin", "hour_cos"]), ["hour"]),
("text", TfidfVectorizer(), "note"), # a single text column
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=2000))])
cv = StratifiedKFold(5, shuffle=True, random_state=0)
print("CV ROC-AUC: %.3f" % cross_val_score(model, df, y, cv=cv, scoring="roc_auc").mean())
model.fit(df, y)
names = model.named_steps["prep"].get_feature_names_out()
print(len(names), "features, e.g.", list(names[:4]), "...", list(names[-3:]))
print("learned Yeo-Johnson lambda for income:",
model.named_steps["prep"].named_transformers_["skewed"][-1].lambdas_.round(3))
# Scaling comparison on the worked example
x = np.array([[2.0], [4.0], [6.0], [8.0], [100.0]])
print("standard:", StandardScaler().fit_transform(x).ravel().round(2))
print("robust: ", RobustScaler().fit_transform(x).ravel().round(2))Quiz
1. Which transformation can make a right-skewed feature more symmetric?
2. Colour (red/green/blue) for a linear model should usually be…
3. Why encode hour of day with sine and cosine?
4. Which models generally do NOT need feature scaling?
5. A word appears in every document. Its IDF is…
6. Target encoding must be computed…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- scikit-learn: Preprocessing data ↗ Official guide
- scikit-learn: Common pitfalls ↗ Leakage and other mistakes