Machine Learning
Topic 34 · Beyond the basics · ⏱ about 3 h

The ML Project Workflow

Putting it all together, from problem to deployed model.

Show:
In this topic
Beginner

End-to-end steps

Knowing individual algorithms is only part of machine learning. Real projects succeed or fail on how the whole process is run. This section walks through the steps of a typical project, in order.

frame theproblemexploredatasplitbaselinesclean &transformmodels +tuningerroranalysistest once,deploy, monitoriterate: improve data & features
A typical machine-learning project. Error analysis usually sends you back to improve the data and features.

The first step is to frame the problem clearly before touching any data. What decision will the model support, and who will use it? Is it a regression problem (predicting a number) or a classification problem (predicting a category)? And, very importantly, which metric reflects the real cost of mistakes? A fraud model judged on accuracy will look excellent while catching nothing, so the choice of metric should come from the business or scientific goal.

Next, collect and explore the data. Exploratory data analysis means looking at the distribution of each feature, the amount and pattern of missing values, the correlations between features, and the balance of the classes. This is also the time to think about possible sources of leakage and about how the data were collected, since biases in collection become biases in the model.

Then split the data into training, validation and test sets, before doing any preprocessing or modelling, so that nothing you do can be influenced by the test set.

Before building anything sophisticated, build baselines. Start with a trivial one, such as always predicting the average value or the most common class, and then a simple model, such as linear or logistic regression. Baselines tell you what "good" means for this problem: if your complex model barely beats the trivial baseline, something is wrong.

Next comes preprocessing and feature engineering: handling missing values, scaling, encoding categories and creating informative features, all inside a pipeline so that it is applied consistently and without leakage. Then try stronger models, such as random forests, gradient boosting or neural networks, and tune their hyperparameters with cross-validation.

A step that is often skipped but is extremely valuable is error analysis. Look at the examples the model gets wrong. Are they concentrated in particular groups, ranges or situations? This usually points to the most effective next improvement, which is often better data or better features rather than a fancier algorithm.

Finally, evaluate the chosen model once on the test set, then deploy it. Deployment is not the end: the world changes, and the data the model sees in use gradually drift away from its training data. Production models must be monitored for this data drift and for declining performance, and retrained when necessary.

Beginner

Responsible ML

Machine learning models increasingly make or influence decisions about people's lives, such as who gets a loan, which job applicants are interviewed, or how a medical condition is treated. Building them responsibly is part of building them well, and involves several considerations.

Fairness. A model learns from historical data, and if that data reflect past biases, the model will reproduce them, often invisibly. The first step is to measure: evaluate the model's performance separately for different groups, such as by age, gender, ethnicity or location, and look for large differences in error rates or outcomes. When problems appear, the remedies range from collecting more representative data to adjusting the training process or the decision thresholds.

Explainability. In high-stakes areas such as health, finance and hiring, people affected by a decision, and the professionals responsible for it, often need to understand why the model decided as it did. Global tools such as feature importance show what drives the model overall, and local tools such as SHAP and LIME explain individual predictions by showing how much each feature pushed that prediction up or down.

Privacy. Training data often contain personal information. Good practice is to collect and keep only the data you need, and to remove or protect identifying details. More advanced techniques include federated learning, which trains a model across many devices without the raw data ever leaving them, and differential privacy, which adds carefully calibrated noise so that the model cannot reveal information about any single individual.

Robustness and security. Models can fail in unexpected ways. Small, deliberately crafted changes to an input, called adversarial examples, can fool image classifiers. Attackers may try to corrupt training data (data poisoning). And ordinary changes in the world over time, distribution shift, can silently degrade performance. Testing under realistic variations and monitoring after deployment help guard against all three.

Documentation. Writing down what a model is for, what data it was trained on, how it was evaluated and where it is known to perform poorly, in a model card or datasheet, helps others use it appropriately and makes problems easier to find.

As a final practical guide, here is where to start for common kinds of data. For tabular data, try gradient boosting alongside a simple linear or logistic baseline. For images, fine-tune a pretrained CNN or vision transformer. For text, fine-tune or prompt a pretrained transformer. And when you have very little data, prefer simple, well-regularised models, which are less likely to overfit.

Intermediate · Maths

Fairness and explainability, formally

📐 Maths you'll need: m-prob
💡 The idea in plain words

"Fair" can be defined in several reasonable ways (equal selection rates, equal true-positive rates, scores meaning the same thing for everyone), and when groups differ in their base rates these definitions can't all be satisfied at once. Shapley values explain a single prediction by fairly sharing it out among the features.

🔤 Symbols in this section
AAsensitive attribute (e.g. a demographic group)
Y^, Y\hat Y,\ Yprediction and true outcome
FFset of all features
SSa subset of features
v(S)v(S)model output using only features in S
ϕj\phi_jShapley value of feature j

This section makes two ideas from responsible ML precise: how fairness can be defined mathematically, and how Shapley values divide a prediction among the features.

Defining fairness. Let AA be a sensitive attribute (for example, a demographic group), YY the true outcome and Y^\hat Y the model's prediction. Several reasonable definitions of group fairness have been proposed, and each captures a different intuition.

Demographic parity requires that the model predicts the positive outcome at the same rate for every group, P(Y^=1∣A=a)=P(Y^=1∣A=b)P(\hat Y = 1\mid A = a) = P(\hat Y = 1\mid A = b): for example, that loans are approved at the same rate for each group, regardless of other differences between the groups. Equal opportunity requires equal true-positive rates, P(Y^=1∣Y=1,A=a)=P(Y^=1∣Y=1,A=b)P(\hat Y = 1\mid Y = 1, A = a) = P(\hat Y = 1\mid Y = 1, A = b): among people who really would repay a loan, each group is equally likely to be approved. Equalised odds is stricter, requiring equal true-positive rates and equal false-positive rates across groups, so that the model's errors have the same profile for everyone. Calibration within groups requires that a given score means the same thing for every group: among all people given a 70% score, about 70% should turn out positive, in every group, P(Y=1∣p^=q,A=a)=qP(Y = 1\mid \hat p = q, A = a) = q.

A crucial result is that these definitions can conflict. Kleinberg and colleagues (2016) and Chouldechova (2017) proved that when the base rates P(Y=1∣A)P(Y = 1\mid A) differ between groups, no classifier (other than a perfect one) can be calibrated within groups and satisfy equalised odds at the same time. This means fairness cannot be achieved simply by choosing the "correct" formula: deciding which definition matters most in a given setting is a value judgement that should involve the people affected, not just a technical decision.

Shapley values. To explain an individual prediction, we would like to say how much each feature contributed to it. Shapley values, borrowed from cooperative game theory, do this by imagining the features joining a "team" one at a time and measuring how much the prediction changes when each one joins. Because the change can depend on which features have already joined, the contribution of feature jj is averaged over every possible joining order:

ϕj=∑S⊆F∖{j}∣S∣! (∣F∣−∣S∣−1)!∣F∣![v(S∪{j})−v(S)]. \phi_j = \sum_{S\subseteq F\setminus\{j\}}\frac{|S|!\,(|F| - |S| - 1)!}{|F|!}\Big[v(S\cup\{j\}) - v(S)\Big].

Here FF is the set of all features, SS is a subset that has joined before feature jj, v(S)v(S) is the model's prediction using only the features in SS (with the others averaged out), and the bracket is feature jj's marginal contribution when it joins SS. The fraction is the proportion of joining orders in which exactly the features in SS come before jj. Shapley values are the only way of sharing out a prediction that satisfies four natural fairness properties. The most useful is efficiency: the contributions add up exactly to the difference between this prediction and the average prediction, ∑jϕj=f(x)−E[f(x)]\sum_j\phi_j = f(x) - \mathbb E[f(x)]. The others are symmetry (features that contribute identically get equal credit), the dummy property (a feature that never changes the prediction gets zero) and additivity. Computing the formula exactly requires considering every subset of features, which is exponential in their number, but for tree ensembles the TreeSHAP algorithm computes exact Shapley values efficiently, and model-agnostic approximations exist for other models.

Key takeaway. Pick a fairness criterion deliberately (they can conflict), measure it per group, and use SHAP or permutation importance to explain predictions.
Code

Code: an end-to-end project template

A complete, reusable skeleton: baseline → pipeline → randomised search → test evaluation → per-group check → permutation importance → saving the model.

Python · scikit-learn
import numpy as np
import pandas as pd
import joblib
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split, RandomizedSearchCV, StratifiedKFold
from sklearn.compose import ColumnTransformer, make_column_selector as cols
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score, classification_report
from sklearn.inspection import permutation_importance

# 1. Data (adult census income; needs internet the first time)
df = fetch_openml("adult", version=2, as_frame=True).frame
y = (df.pop("class") == ">50K").astype(int)
X = df
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

# 2. Preprocessing shared by all models
prep = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")), ("sc", StandardScaler())]),
     cols(dtype_include="number")),
    ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                      ("oh", OneHotEncoder(handle_unknown="ignore", min_frequency=20))]),
     cols(dtype_exclude="number")),
])
cv = StratifiedKFold(5, shuffle=True, random_state=0)

# 3. Baselines: always know what "good" means
for name, m in [("majority", DummyClassifier()), ("logistic", LogisticRegression(max_iter=2000))]:
    pipe = Pipeline([("prep", prep), ("m", m)]).fit(Xtr, ytr)
    print(name, "test AUC:", round(roc_auc_score(yte, pipe.predict_proba(Xte)[:, 1]), 3))

# 4. Stronger model + randomised hyperparameter search
search = RandomizedSearchCV(
    Pipeline([("prep", prep), ("m", HistGradientBoostingClassifier(random_state=0))]),
    {"m__learning_rate": np.logspace(-2, -0.5, 10), "m__max_leaf_nodes": [15, 31, 63],
     "m__min_samples_leaf": [10, 20, 50], "m__l2_regularization": [0, 0.1, 1.0]},
    n_iter=15, cv=cv, scoring="roc_auc", n_jobs=-1, random_state=0).fit(Xtr, ytr)
best = search.best_estimator_
p = best.predict_proba(Xte)[:, 1]
print("best params:", search.best_params_)
print("final TEST AUC:", round(roc_auc_score(yte, p), 3))       # touch the test set once
print(classification_report(yte, (p >= 0.5).astype(int)))

# 5. Per-group check (fairness): compare TPR and positive rate across groups
res = pd.DataFrame({"sex": Xte["sex"], "y": yte, "pred": (p >= 0.5).astype(int)})
print(res.groupby("sex").apply(lambda g: pd.Series({
    "positive_rate": g.pred.mean(), "TPR": g.pred[g.y == 1].mean()})).round(3))

# 6. Explain: permutation importance on held-out data
imp = permutation_importance(best, Xte.iloc[:2000], yte.iloc[:2000], scoring="roc_auc",
                             n_repeats=5, random_state=0)
print(pd.Series(imp.importances_mean, index=X.columns).sort_values(ascending=False).head(8).round(4))

# 7. Save for deployment
joblib.dump(best, "income_model.joblib")
This code downloads a dataset from the internet with fetch_openml, which the in-page Python cannot do. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. Why build a trivial baseline first?

2. Data drift after deployment means…

3. A good first model for tabular data is usually…

4. Equal opportunity requires equal ___ across groups.

5. When base rates differ, which pair cannot both hold for an imperfect classifier?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready