Deep Learning
Topic 15 · Deep Learning · ⏱ about 8 h

Generative Models: Autoencoders, VAEs, GANs & Diffusion

Models that learn to produce data: autoencoders for compression and anomaly detection, VAEs and latent spaces, GANs, diffusion models, and the risks of synthetic data.

Show:
In this topic
Beginner

Discriminative vs generative

Everything so far has been discriminative: predict a label y from an input x, p(y∣x)p(y\mid x). A generative model learns the distribution of the data itself, p(x)p(x), well enough to produce new examples: images, text, molecules, synthetic patient records. Large language models are generative (they model text one token at a time); so are image generators. Generative models are also used for anomaly detection (inputs the model finds unlikely), denoising, compression and data augmentation.

Beginner

Watch it: Diffusion models: learn to remove noise

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

Autoencoders

An autoencoder is trained to reproduce its input through a narrow bottleneck: an encoder compresses x to a small code z, and a decoder reconstructs x^\hat x from z, minimising the reconstruction error ∥x−x^∥2\lVert x-\hat x\rVert^2. The bottleneck forces it to keep only the essential structure. Uses:

  • Anomaly detection: trained on normal data (normal ECG beats, normal machine sounds), it reconstructs normal inputs well and unusual ones badly, so a high reconstruction error flags an anomaly.
  • Denoising autoencoders are trained to reconstruct clean inputs from corrupted ones.
  • The codes z are compact features for other models.

A plain autoencoder is not a good generator: its code space has gaps, and decoding a random z usually gives nonsense.

Beginner

Variational autoencoders and latent spaces

A VAE makes the code space smooth and fillable. The encoder outputs a distribution for each input, a mean μ and a standard deviation σ, and the code is sampled from it. Sampling is not differentiable, so the reparameterisation trick writes it as z=μ+σ⊙εz=\mu+\sigma\odot\varepsilon with ε∼N(0,I)\varepsilon\sim\mathcal N(0,I): the randomness is an input, and gradients flow through μ and σ.

The loss (the negative ELBO) has two parts: reconstruction error, plus a KL divergence that pulls each code distribution towards a standard normal:

L=∥x−x^∥2+β⋅12∑j(μj2+σj2−log⁡σj2−1).\mathcal L=\lVert x-\hat x\rVert^2+\beta\cdot\tfrac12\sum_j\big(\mu_j^2+\sigma_j^2-\log\sigma_j^2-1\big).

The KL term packs codes together without gaps, so you can sample z∼N(0,I)z\sim\mathcal N(0,I) and decode it into a new, plausible example, or interpolate between two codes and get a smooth transition. Directions in this latent space often correspond to meaningful factors.

Try it

Try it: walk through a latent space

A decoder maps each point of a 2-D latent space to a heartbeat waveform (a stand-in for a trained VAE decoder). Click anywhere to decode it, or interpolate between two beats.

Interactive: this part runs in your browser.

Beginner

GANs

A generative adversarial network trains two networks against each other: a generator turns random noise into fake examples, and a discriminator tries to tell real from fake. The generator improves by fooling the discriminator. GANs produced the first strikingly realistic faces (StyleGAN), but training is unstable: it can oscillate, or suffer mode collapse (the generator produces only a few kinds of output). For most image generation they have been overtaken by diffusion models, though they remain fast at generation time.

Beginner

Diffusion models

A diffusion model learns to reverse a gradual noising process. The forward process is fixed: over T steps (e.g. 1,000), add a little Gaussian noise each step until the data becomes pure noise. In closed form, xt=αˉt x0+1−αˉt εx_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon, where αˉt\bar\alpha_t falls from 1 to almost 0.

A network (a U-Net or a transformer) is trained on a very simple task: given a noisy xtx_t and t, predict the noise ε that was added (mean squared error). To generate, start from pure noise and repeatedly subtract a little of the predicted noise, step by step, until a clean sample emerges. Conditioning the network on text (via cross-attention to a text encoder) gives text-to-image systems. Diffusion training is stable and the samples are diverse and high quality; the cost is many network evaluations per sample, which newer samplers reduce from about 1,000 to 20–50 steps.

Try it

Try it: noise and denoise

Drag the time slider to noise the data step by step. Then run the reverse process: 300 points start as pure noise and are denoised into the shape. (The denoiser here uses the exact score of this small dataset instead of a trained network.)

Interactive: this part runs in your browser.

Beginner

Using generative models responsibly

  • Evaluation is hard: likelihood, FID (for images) and human ratings each capture only part of quality and diversity. Always look at many samples.
  • Memorisation and privacy: generators can reproduce training examples. "Synthetic" patient data is not automatically de-identified; it needs a privacy assessment (and differential privacy where required).
  • Synthetic data for training can help with rare classes, but test on real data only, and check that the synthetic data does not encode artefacts the model learns instead of the real signal.
  • Misuse: deepfakes and impersonation. Many organisations watermark or label generated content.
Try it

Code example: the VAE's key pieces

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Check yourself

Quiz

1. An autoencoder trained only on normal ECG beats is used for anomaly detection by…

2. Why does a VAE need the reparameterisation trick?

3. What does the KL term in a VAE do?

4. What is a diffusion model's network trained to predict?

5. Mode collapse is a failure of…

6. A vendor offers "synthetic patient records, so privacy rules don't apply". You should…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready