Generative Models: Autoencoders, VAEs, GANs & Diffusion
Models that learn to produce data: autoencoders for compression and anomaly detection, VAEs and latent spaces, GANs, diffusion models, and the risks of synthetic data.
Discriminative vs generative
Everything so far has been discriminative: predict a label y from an input x, . A generative model learns the distribution of the data itself, , well enough to produce new examples: images, text, molecules, synthetic patient records. Large language models are generative (they model text one token at a time); so are image generators. Generative models are also used for anomaly detection (inputs the model finds unlikely), denoising, compression and data augmentation.
Watch it: Diffusion models: learn to remove noise
A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.
Interactive: this part runs in your browser.
Autoencoders
An autoencoder is trained to reproduce its input through a narrow bottleneck: an encoder compresses x to a small code z, and a decoder reconstructs from z, minimising the reconstruction error . The bottleneck forces it to keep only the essential structure. Uses:
- Anomaly detection: trained on normal data (normal ECG beats, normal machine sounds), it reconstructs normal inputs well and unusual ones badly, so a high reconstruction error flags an anomaly.
- Denoising autoencoders are trained to reconstruct clean inputs from corrupted ones.
- The codes z are compact features for other models.
A plain autoencoder is not a good generator: its code space has gaps, and decoding a random z usually gives nonsense.
Variational autoencoders and latent spaces
A VAE makes the code space smooth and fillable. The encoder outputs a distribution for each input, a mean μ and a standard deviation σ, and the code is sampled from it. Sampling is not differentiable, so the reparameterisation trick writes it as with : the randomness is an input, and gradients flow through μ and σ.
The loss (the negative ELBO) has two parts: reconstruction error, plus a KL divergence that pulls each code distribution towards a standard normal:
The KL term packs codes together without gaps, so you can sample and decode it into a new, plausible example, or interpolate between two codes and get a smooth transition. Directions in this latent space often correspond to meaningful factors.
Try it: walk through a latent space
A decoder maps each point of a 2-D latent space to a heartbeat waveform (a stand-in for a trained VAE decoder). Click anywhere to decode it, or interpolate between two beats.
Interactive: this part runs in your browser.
GANs
A generative adversarial network trains two networks against each other: a generator turns random noise into fake examples, and a discriminator tries to tell real from fake. The generator improves by fooling the discriminator. GANs produced the first strikingly realistic faces (StyleGAN), but training is unstable: it can oscillate, or suffer mode collapse (the generator produces only a few kinds of output). For most image generation they have been overtaken by diffusion models, though they remain fast at generation time.
Diffusion models
A diffusion model learns to reverse a gradual noising process. The forward process is fixed: over T steps (e.g. 1,000), add a little Gaussian noise each step until the data becomes pure noise. In closed form, , where falls from 1 to almost 0.
A network (a U-Net or a transformer) is trained on a very simple task: given a noisy and t, predict the noise ε that was added (mean squared error). To generate, start from pure noise and repeatedly subtract a little of the predicted noise, step by step, until a clean sample emerges. Conditioning the network on text (via cross-attention to a text encoder) gives text-to-image systems. Diffusion training is stable and the samples are diverse and high quality; the cost is many network evaluations per sample, which newer samplers reduce from about 1,000 to 20–50 steps.
Try it: noise and denoise
Drag the time slider to noise the data step by step. Then run the reverse process: 300 points start as pure noise and are denoised into the shape. (The denoiser here uses the exact score of this small dataset instead of a trained network.)
Interactive: this part runs in your browser.
Using generative models responsibly
- Evaluation is hard: likelihood, FID (for images) and human ratings each capture only part of quality and diversity. Always look at many samples.
- Memorisation and privacy: generators can reproduce training examples. "Synthetic" patient data is not automatically de-identified; it needs a privacy assessment (and differential privacy where required).
- Synthetic data for training can help with rare classes, but test on real data only, and check that the synthetic data does not encode artefacts the model learns instead of the real signal.
- Misuse: deepfakes and impersonation. Many organisations watermark or label generated content.
Code example: the VAE's key pieces
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Quiz
1. An autoencoder trained only on normal ECG beats is used for anomaly detection by…
2. Why does a VAE need the reparameterisation trick?
3. What does the KL term in a VAE do?
4. What is a diffusion model's network trained to predict?
5. Mode collapse is a failure of…
6. A vendor offers "synthetic patient records, so privacy rules don't apply". You should…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Auto-Encoding Variational Bayes (Kingma & Welling) ↗ The VAE paper
- Denoising Diffusion Probabilistic Models (Ho et al., 2020) ↗ The DDPM paper
- Understanding Deep Learning (Prince) ↗ Modern, clear and free textbook