Neural Networks
Layers of simple units that together learn complex functions.
The artificial neuron
Neural networks are built from a very simple unit, the artificial neuron, and you have already met it in disguise. A neuron takes several inputs, multiplies each by a weight, adds them up together with a bias, and then passes the result through a function called the activation function:
The part inside the brackets is exactly the weighted sum used in linear regression. The activation function is what makes the neuron more than a linear model. The earliest version, Frank Rosenblatt's perceptron from 1958, used a step function: output 1 if the weighted sum is positive, and 0 otherwise. If is the sigmoid, a single neuron is exactly logistic regression. So a single neuron draws a straight-line boundary, and everything that makes neural networks powerful comes from connecting many of them together, as the following sections explain.
Watch it: How a neural network learns: forward pass, error, backpropagation
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Try it: one neuron, three logic gates
Adjust the weights and bias so the single neuron computes AND, then OR. Then try XOR, and discover you can't.
Interactive: this part runs in your browser.
Multi-layer networks
A single neuron can only draw a straight boundary, as the XOR demo showed. To learn more complicated patterns, we connect many neurons into layers.
The input layer simply holds the features. Each neuron in the first hidden layer takes all the inputs, computes its own weighted sum and applies its activation function. The neurons in the next hidden layer take the outputs of the first hidden layer as their inputs, and so on, until the output layer produces the prediction. They are called "hidden" because their values are internal: they are neither inputs nor outputs, and the network decides for itself what they should represent. A network built this way is called a multi-layer perceptron (MLP), and a network with many hidden layers is what we mean by deep learning.
In matrix form, each layer computes : a matrix of weights (one row per neuron), a vector of biases , and the activation applied to every element. A whole network is a chain of these operations.
The activation functions are not an optional detail; they are essential. If you removed them, each layer would be a linear transformation, and a chain of linear transformations is itself just one linear transformation, because multiplying several matrices together gives a single matrix. A hundred-layer network without activation functions would be no more powerful than a single linear model. The non-linear activations are what allow each layer to bend and reshape the data in ways a single linear map cannot.
How powerful are such networks? The universal approximation theorem says that a network with just one hidden layer, given enough neurons, can approximate any continuous function as closely as you like. In practice, though, deep networks (many layers of moderate size) learn far more efficiently than shallow, extremely wide ones, because each layer can build on the features discovered by the layer before. In image recognition, for example, early layers learn to detect edges, middle layers combine edges into textures and shapes, and later layers combine those into whole objects such as faces or cars.
Finally, the output layer is chosen to match the task. For regression, the output neuron has no activation (a plain linear output), so it can produce any number. For binary classification, a single output neuron with a sigmoid produces a probability. For multi-class classification, one output neuron per class is followed by a softmax, which turns the outputs into probabilities that add up to 1.
Interactive: this part runs in your browser.
Interactive: this part runs in your browser.
Layers as matrix operations, and solving XOR by hand
A layer is a matrix multiplication followed by a simple non-linear function. Stacking layers lets the network re-describe the data: the XOR example shows a hidden layer transforming four points that no line can separate into a new representation where a line can.
🔤 Symbols in this section
| activations of layer ℓ (one row per example) | |
| weights and biases of layer ℓ | |
| activation function (ReLU, sigmoid, …) | |
| number of units in layer ℓ | |
| batch size |
This section writes a network in precise matrix form, counts its parameters, and then solves the XOR problem by hand to show exactly how a hidden layer helps.
For efficiency, networks process a whole mini-batch of examples at once, stored as a matrix with one row per example. Layer takes the previous layer's activations (with ) and computes
Here is a weight matrix of shape (inputs by outputs), the term simply adds the bias vector to every row, holds the weighted sums (the "pre-activations"), and is applied to every entry separately. Each layer therefore has weights plus biases. For example, a network for 28-by-28-pixel images (784 inputs) with one hidden layer of 256 neurons and 10 output classes has parameters.
The output layer and the loss are chosen together, in matched pairs: a linear output with mean squared error for regression, a sigmoid output with binary cross-entropy for two classes, and a softmax output with categorical cross-entropy for many classes. Each pairing is chosen because it makes the gradient of the loss with respect to the output's pre-activation simply , prediction minus target, which keeps training simple and numerically stable.
Solving XOR by hand. The XOR problem asks for output 1 when exactly one of two binary inputs is 1, and output 0 otherwise. As the demo showed, no single neuron can do this, because no straight line separates the points and from and . But two hidden ReLU neurons are enough. Define
| (0, 0) | 0 | 0 | 0 |
| (0, 1) | 1 | 0 | 1 |
| (1, 0) | 1 | 0 | 1 |
| (1, 1) | 2 | 1 | 0 |
Let's check the last row carefully. With both inputs equal to 1, and , so . Every row gives the correct XOR output. What happened is that the hidden layer mapped the four input points into a new two-dimensional space, , in which the classes are separable by a straight line. Learning useful new representations of the data like this, layer by layer, is the central idea of deep learning, and it is why it is often called representation learning.
The same example shows why the activation is essential. Without the ReLUs, and would be linear in the inputs, so would be linear too, and no linear function can produce XOR. In general, without activations, , and the product of all those matrices is a single matrix: the whole network collapses into one linear map.
Expressive power: universal approximation and depth
Theory says a single wide hidden layer can approximate any reasonable function, but it might need absurdly many units, and it says nothing about whether training will find them. Depth is far more efficient: each extra layer can multiply the number of "pieces" the network can represent.
🔤 Symbols in this section
| a bounded region of input space | |
| approximation tolerance | |
| number of hidden units | |
| the worst case over the region |
This section looks at what theory says about the power of neural networks, and why, in practice, depth matters so much.
Universal approximation. A series of results from the late 1980s and early 1990s, by Cybenko, Hornik and Leshno and colleagues, established the following. Take any continuous function on a bounded region of input space, any tolerance , and any activation function that is not a polynomial (which includes sigmoid, tanh and ReLU). Then there is a network with a single hidden layer of some size whose output stays within of everywhere in the region:
(The symbol means "the largest value over the region", so the statement says the worst-case error is below .) It is important to read this theorem carefully, because it is often over-interpreted. It guarantees that a suitable network exists. It does not say that gradient descent will find it, it says nothing about how well the network will generalise from limited data, and it gives no reasonable bound on : for some functions the number of neurons needed grows exponentially with the number of input dimensions.
Why depth helps. A network with ReLU activations computes a piecewise-linear function: the input space is divided into regions, and inside each region the network's output is linear. The number of these regions measures how complicated a function the network can represent. Montúfar and colleagues showed in 2014 that the maximum number of linear regions can grow exponentially with the number of layers, but only polynomially with the number of neurons per layer. Intuitively, each layer can "fold" the space produced by the previous layer, and repeated folding multiplies the number of pieces. There are known functions that a deep network can represent with a modest number of neurons but that any shallow network would need exponentially many neurons to match (Telgarsky, 2016). Depth also matches the compositional structure of real-world data: images are built from parts, which are built from shapes, which are built from edges.
Inductive biases of architectures. Different network architectures build in different assumptions about the data. A plain MLP assumes very little. A convolutional network assumes that nearby inputs are related and that a pattern means the same thing wherever it appears. A recurrent network assumes that the data come in a sequence. A transformer treats its input as a set of items and learns how strongly each item should interact with each other one. A graph neural network assumes the data form a network of connected nodes. Choosing the architecture whose assumptions match your data is often more important than making the network bigger.
The puzzle of over-parameterisation. Modern networks often have far more parameters than training examples. Zhang and colleagues (2017) showed that such networks can perfectly memorise training data with completely random labels, so their capacity is clearly enormous, yet on real data they generalise well. Classical theory, which says the generalisation gap grows with capacity, cannot explain this on its own. Current explanations focus on the implicit regularisation of stochastic gradient descent (it tends to find simple, small-weight solutions), on the tendency to settle in wide, flat minima that are robust to small changes, and on the double-descent phenomenon.
Code: an MLP forward pass in NumPy and a PyTorch model
The NumPy code verifies the hand-built XOR network and counts parameters; the PyTorch class is the standard way to define a network.
import numpy as np
relu = lambda z: np.maximum(0, z)
# Hand-built XOR network from the maths section
W1 = np.array([[1.0, 1.0],
[1.0, 1.0]]); b1 = np.array([0.0, -1.0])
W2 = np.array([[1.0], [-2.0]]); b2 = np.array([0.0])
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
H = relu(X @ W1 + b1) # hidden layer, shape (4, 2)
y_hat = H @ W2 + b2 # output layer, shape (4, 1)
print(np.c_[X, H, y_hat]) # columns: x1 x2 h1 h2 y_hat
def n_params(sizes):
return sum(a * b + b for a, b in zip(sizes[:-1], sizes[1:]))
print("784-256-10 params:", n_params([784, 256, 10]))import torch
from torch import nn
class MLP(nn.Module):
def __init__(self, d_in, hidden=(256, 128), d_out=10, p_drop=0.2):
super().__init__()
layers, d = [], d_in
for h in hidden:
layers += [nn.Linear(d, h), nn.ReLU(), nn.Dropout(p_drop)]
d = h
layers.append(nn.Linear(d, d_out)) # logits: no softmax here
self.net = nn.Sequential(*layers)
def forward(self, x):
return self.net(x)
model = MLP(784)
print(model)
print("parameters:", sum(p.numel() for p in model.parameters()))
logits = model(torch.randn(32, 784)) # batch of 32
probs = logits.softmax(dim=1) # nn.CrossEntropyLoss takes logits directlyQuiz
1. A single neuron with a sigmoid activation is equivalent to…
2. Why can't one neuron learn XOR?
3. Without non-linear activations, a 10-layer network is…
4. How many parameters does a Linear(100 → 50) layer have?
5. The universal approximation theorem guarantees…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Dive into Deep Learning (free book with code) ↗ Interactive textbook in PyTorch, by Zhang, Lipton, Li & Smola
- Deep Learning (Goodfellow, Bengio & Courville) ↗ The classic textbook, free online