Logistic Regression
Despite the name, it's a classification model that outputs probabilities.
From a line to a probability
Despite its name, logistic regression is a method for classification, not regression. It answers yes/no questions, such as whether an email is spam or whether a tumour is malignant, and instead of a bare yes or no it gives a probability.
We start exactly as in linear regression, by computing a weighted score from the features, . This score can be any number from minus infinity to plus infinity, which is no good as a probability. So we pass it through the sigmoid function, which squashes any number smoothly into the range between 0 and 1:
Large positive scores give probabilities close to 1, large negative scores give probabilities close to 0, and a score of exactly 0 gives a probability of exactly one half. To make a decision, we usually predict class 1 when the probability is at least 0.5, which happens exactly when the score is at least 0.
The set of points where the score is exactly zero, , separates the inputs predicted as class 1 from those predicted as class 0, and it is called the decision boundary. Because it is defined by a linear equation, it is a straight line when there are two features, a flat plane with three, and a flat "hyperplane" in higher dimensions. For this reason, logistic regression is called a linear classifier: it can only separate classes that lie on either side of a flat boundary (unless you engineer extra features).
The weights also have a useful interpretation. The score turns out to equal the log-odds of class 1, , where the odds compare the chance of "yes" with the chance of "no". Increasing feature by one unit adds to the log-odds, which multiplies the odds by . A weight of 0.7, for instance, multiplies the odds by about 2, doubling them.
Interactive: this part runs in your browser.
Watch it: Logistic regression: from a score to a probability
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Log-loss (cross-entropy)
To train logistic regression we need a loss function that measures how good the predicted probabilities are. Squared error turns out to be a poor choice here: combined with the sigmoid, it produces a bumpy loss surface with flat regions where learning stalls. Instead we use log-loss, also called binary cross-entropy:
Although it looks complicated, only one of the two terms is active for each example. If the true label is , the second term vanishes and the loss for that example is . If , the first term vanishes and the loss is . In both cases, the loss is minus the log of the probability the model assigned to the correct answer.
Let's see how this behaves. Suppose the true label is 1. If the model says , the loss is , almost nothing. If it says , the loss is . If it confidently says , the loss is , and it grows without limit as the probability approaches zero. So log-loss gently rewards correct confidence and severely punishes confident mistakes. As the likelihood topic shows, it is also exactly the negative log-likelihood of the data, so minimising it is maximum likelihood estimation.
Unlike linear regression, there is no formula that gives the best weights directly, so we use gradient descent. Two facts make this easy. First, the loss is convex, a single bowl with no false minima, so gradient descent is guaranteed to reach the best solution. Second, the gradient has a strikingly simple form, derived in the next section:
which reads "prediction minus label, times the input", exactly the same shape as for linear regression.
For problems with more than two classes, logistic regression generalises to softmax regression. The model computes one score per class and converts the scores into probabilities with the softmax function, which exponentiates each score (making it positive) and divides by the total (making them add up to 1). The loss becomes the categorical cross-entropy, again minus the log of the probability assigned to the correct class.
Interactive: this part runs in your browser.
Try it: fit a sigmoid
Blue dots are class 0, orange dots are class 1. Adjust w and b to minimise log-loss, then let gradient descent do it.
Interactive: this part runs in your browser.
Deriving the gradient and proving convexity
We need the slope of the log-loss to run gradient descent. The chain rule does the work, and almost everything cancels, leaving a beautifully simple result: (prediction − label) × input. A second derivative then shows the loss is a bowl, so there's only one minimum to find.
🔤 Symbols in this section
| predicted probability for example i | |
| linear score (log-odds) | |
| true label | |
| derivative of the sigmoid | |
| diagonal weighting matrix | |
| Hessian of the loss |
This section derives the gradient of the log-loss step by step and then proves that the loss has no false minima. The chain rule from the derivatives topic does most of the work.
To keep the notation light, we fold the bias into the weight vector (by adding a constant feature of 1), so the score is and the predicted probability is . The total loss is
Each example's loss depends on through a chain: determines the score , which determines the probability , which determines the loss . We'll differentiate one link at a time.
All of the complexity of the logarithms and the sigmoid has cancelled away, leaving "prediction minus label, times input". If the model predicts 0.9 for an example whose label is 1, the error is −0.1 and the weights move a little to increase that prediction; if it predicts 0.9 for a label of 0, the error is +0.9 and the weights move a lot to decrease it.
Convexity. To show that gradient descent can't get stuck, we look at the second derivative. Differentiating the gradient once more, using , gives the Hessian
Every diagonal entry of is a product of a probability and its complement, so it lies between 0 and . For any vector , we therefore have , since it is a sum of non-negative numbers times squares. So the Hessian is positive semi-definite everywhere, the loss is convex, and every local minimum is a global minimum.
Newton's method. Because we know the Hessian explicitly, we can also use Newton's method: . Each Newton step turns out to be a weighted least-squares problem with weights , so this algorithm is known as iteratively re-weighted least squares (IRLS). It usually converges in fewer than ten steps, and it is what traditional statistics packages such as R's glm and Python's statsmodels use.
Softmax, separability and calibration
Three practical extensions. Softmax handles many classes by turning scores into probabilities that sum to 1. If the classes can be split perfectly, the model keeps growing its weights to become ever more confident, so you need regularisation. And because the model is trained by maximum likelihood, its probabilities tend to be trustworthy (calibrated).
🔤 Symbols in this section
| number of classes | |
| score for class k | |
| softmax probability of class k | |
| 1 if example i is class k, else 0 (one-hot) | |
| odds ratio for feature j |
This section covers three things you need when using logistic regression in practice: extending it to many classes, a failure that occurs on perfectly separable data, and why its probabilities can usually be trusted.
Many classes: softmax regression. With classes, the model keeps one weight vector per class and computes one score per class, . The softmax function turns these scores into probabilities:
Exponentiating makes every score positive, and dividing by the total makes the probabilities add up to 1; the class with the highest score gets the highest probability. Writing the true class as a one-hot vector (1 in the correct position, 0 elsewhere), the loss is the categorical cross-entropy, and its gradient has the same pleasing form as before:
One implementation detail matters a lot: exponentials of large scores overflow. Because adding the same constant to every score doesn't change the softmax, software subtracts the largest score first, computing as with . This is called the log-sum-exp trick.
Perfectly separable data. Suppose some line separates the two classes perfectly. Then we can take any weights that achieve that separation and multiply them by a large number . The decision boundary stays in the same place, but every score gets times larger, so every probability moves closer to 0 or 1 on the correct side and the loss gets smaller. Making larger always helps, so the loss approaches zero only as the weights grow to infinity: maximum likelihood has no finite solution, and in practice the weights keep growing and the model becomes absurdly overconfident. Adding regularisation, a penalty on large weights, restores a sensible finite answer. Interestingly, research has shown that plain gradient descent on separable data converges in direction to the same boundary as a hard-margin support vector machine, even though the weights themselves grow forever.
Interpreting the weights. Because the score is the log-odds, , the number is an odds ratio: increasing feature by one unit, with everything else fixed, multiplies the odds of class 1 by . This is why logistic regression is popular in medicine and the social sciences, where interpretable effects matter.
Calibration. A model is well calibrated if its probabilities can be taken at face value: among all the cases where it says 70%, about 70% really are positive. Log-loss is what statisticians call a proper scoring rule, meaning it is minimised by reporting the true probabilities, so logistic regression usually produces well-calibrated probabilities. Many more powerful models, including support vector machines, boosted trees and deep networks, do not, and their scores need to be corrected after training. Two common fixes are Platt scaling, which fits a small logistic regression to the model's scores, and isotonic regression, which fits a flexible increasing curve.
Code: logistic regression from scratch and in scikit-learn
Gradient descent with the gradient we derived, checked against scikit-learn.
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=500, n_features=4, n_informative=3,
n_redundant=0, random_state=0)
X = (X - X.mean(0)) / X.std(0)
Xb = np.c_[np.ones(len(X)), X]
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def nll(w, lam=0.0):
p = np.clip(sigmoid(Xb @ w), 1e-12, 1 - 1e-12)
return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)) + lam / 2 * w[1:] @ w[1:]
w, lr, lam = np.zeros(Xb.shape[1]), 0.5, 0.01
for it in range(2000):
p = sigmoid(Xb @ w)
g = Xb.T @ (p - y) / len(y) # gradient = X^T (p - y) / n
g[1:] += lam * w[1:] # L2 penalty (not on the bias)
w -= lr * g
print("weights:", w.round(3), " loss:", round(nll(w, lam), 4))
print("accuracy:", np.mean((sigmoid(Xb @ w) >= 0.5) == y))
print("odds ratios:", np.exp(w[1:]).round(2))from sklearn.linear_model import LogisticRegression
# C = 1 / (lambda * n) roughly; smaller C = stronger regularisation
clf = LogisticRegression(C=1 / (lam * len(y)), max_iter=1000).fit(X, y)
print("sklearn:", clf.intercept_.round(3), clf.coef_.round(3))
print("P(y=1) for first 3 rows:", clf.predict_proba(X[:3])[:, 1].round(3))
# Multiclass: the same class uses softmax (multinomial) automatically
from sklearn.datasets import load_iris
Xi, yi = load_iris(return_X_y=True)
soft = LogisticRegression(max_iter=1000).fit(Xi, yi)
print(soft.predict_proba(Xi[:2]).round(3)) # each row sums to 1Quiz
1. The sigmoid function outputs values in…
2. Logistic regression's decision boundary is…
3. What loss does logistic regression minimise?
4. The gradient of logistic regression's log-loss is…
5. On perfectly separable data, unregularised logistic regression weights…
6. e^(wⱼ) in logistic regression is…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- scikit-learn: Linear models ↗ Official guide
- An Introduction to Statistical Learning (free book) ↗ James, Witten, Hastie, Tibshirani (Python edition available)