Lesson 16.5 · 27 min

GANs: A Generator and Discriminator in Constant Competition

What happens if we train one network to forge images and a second network to catch the forgeries — and let them compete until the forger wins?

In short: A Generative Adversarial Network (GAN) trains two networks against each other: a generator that turns random noise into fake samples, and a discriminator that tries to tell real samples from fakes. Each improves by exploiting the other's weaknesses, and at the ideal end point the fakes are indistinguishable from real data. GANs generate sharp images in a single fast pass, but training is a delicate balancing act that can oscillate or collapse to a few repeated outputs.

What is a Generative Adversarial Network (GAN)?

A generative model learns what a dataset looks like so it can produce new examples that could plausibly belong to it. In 2014, Ian Goodfellow and colleagues proposed a striking way to train one: instead of writing down a formula for 'how good is this fake image?', let a second neural network learn to judge. That pair of networks, trained as opponents, is a Generative Adversarial Network (GAN). 'Adversarial' simply means 'set against each other'.

Our running example: an appliance shop wants thousands of realistic product photos of kettles in different colours and settings to train a damage detector, without photographing every combination. A GAN trained on real kettle photos can produce new ones on demand, each from a different random input.

For several years (roughly 2014–2021), GANs produced the most photorealistic generated images, including the famous synthetic faces. Diffusion models have since taken over most text-to-image work, but GANs remain important wherever we need one-pass, real-time generation, and their adversarial loss lives on inside many modern systems.

The two players: generator vs discriminator

The noise vector z lives in the latent space. Each point in it maps to one output image, and nearby points usually map to similar images. Sampling a new z gives a new kettle; sliding smoothly between two z's often morphs one kettle smoothly into another.

The counterfeiter vs police analogy

Forger and detective The generator is a counterfeiter printing fake banknotes; the discriminator is a police detective examining notes. At first the fakes are crude, and the detective spots them easily by obvious tells: wrong colour, no watermark. Each time a fake is caught, the counterfeiter learns which feature gave it away and fixes it. The detective then has to look for subtler clues. Round after round, both get better. The game ends, in theory, when the fakes are so good that the detective can do no better than guessing: 50/50.

There is one twist the analogy hides. Our counterfeiter does not just hear 'caught' or 'not caught'. Because both players are differentiable networks, the generator receives the gradient of the detective's judgement: an exact signal saying 'make these pixels a bit brighter and the detective will be more convinced'. That rich feedback is what makes GAN training possible at all.

Pause and think: At the ideal end of training, what does the discriminator output for every input, and why?

About 0.5. If the generator's samples come from exactly the same distribution as the real data, no classifier can tell them apart, so the best possible guess for any input is 50% real.

The adversarial training loop

The same loop in plain words

1. Fix G, improve D: D sees a mix of real and fake samples and takes one gradient step to classify them better.
2. Fix D, improve G: G takes one gradient step to make its outputs look more real to the current D.
3. Alternate: Repeat thousands of times. Some recipes give D several steps per G step, or use different learning rates for the two.
4. Watch samples, not just losses: Unlike normal training, the losses do not steadily fall: when one player improves, the other's loss goes up. Practitioners judge progress by looking at samples and by metrics such as FID (Fréchet Inception Distance), which compares statistics of real and generated images.

The loss function and the minimax game in simple words

The original GAN objective is a single value V(D, G) that D tries to make large and G tries to make small. That is why it is called a minimax game:

• First term, log D(x): on real data, D wants D(x) near 1, so log D(x) near 0 (its maximum). If D wrongly says 0.1 on a real image, log 0.1 ≈ −2.3: a big penalty.
• Second term, log(1 − D(G(z))): on fakes, D wants D(G(z)) near 0, so log(1 − 0) = 0. The generator controls only this term and wants D(G(z)) near 1, which drives it toward log(0) = −∞.
• Together: D maximising V is exactly ordinary binary cross-entropy training. G minimising V is 'fool D'.

The non-saturating trick used in practice Early in training, fakes are terrible, D(G(z)) ≈ 0, and log(1 − D(G(z))) is almost flat there, so G gets tiny gradients and learns nothing. The original paper therefore suggested that G instead maximise log D(G(z)). It has the same goal (fool D) but gives strong gradients exactly when G is losing. Almost all implementations use this version.

A tiny PyTorch-style code sketch (and a runnable numpy version)

Here is what a minimal GAN for 28×28 images looks like in PyTorch. It is a sketch to read, not run here (PyTorch is not installed in this course environment, and loader stands for a data loader).

gan_sketch.py (PyTorch, for reading)

import torch, torch.nn as nn
G = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 784), nn.Tanh())
D = nn.Sequential(nn.Linear(784, 256), nn.LeakyReLU(0.2), nn.Linear(256, 1))  # logits
opt_G = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss()
for real in loader:                          # real: (batch, 784) images in [-1, 1]
n = real.size(0)
ones, zeros = torch.ones(n, 1), torch.zeros(n, 1)
# 1) Train D: real -> 1, fake -> 0. detach() so G is not updated here.
fake = G(torch.randn(n, 64))
loss_D = bce(D(real), ones) + bce(D(fake.detach()), zeros)
opt_D.zero_grad(); loss_D.backward(); opt_D.step()
# 2) Train G: make D output 1 on fakes (non-saturating loss).
loss_G = bce(D(fake), ones)
opt_G.zero_grad(); loss_G.backward(); opt_G.step()

To see real training dynamics with real output, here is the same loop in numpy on 1-D data. Real data are numbers from a bell curve with mean 4 and standard deviation 0.5. The generator is G(z) = a·z + b, and the discriminator is a logistic classifier on x and x². We run it twice with different learning rates.

tiny_gan.py

import numpy as np
rng = np.random.default_rng(0)
sig = lambda u: 1 / (1 + np.exp(-u))
real = lambda n: rng.normal(4.0, 0.5, n)       # real data: mean 4, std 0.5
feat = lambda x: np.stack([x, x * x, np.ones_like(x)])
def train(lr_d, lr_g, steps=6000, n=64):
a, b = 1.0, 0.0              # generator  G(z) = a*z + b,  z ~ N(0, 1)
w = np.zeros(3)              # discriminator D(x) = sigmoid(w1*x + w2*x^2 + c)
for step in range(steps + 1):
# 1) Discriminator step: push D(real) -> 1 and D(fake) -> 0
xr, xf = real(n), a * rng.normal(size=n) + b
dr, df = sig(w @ feat(xr)), sig(w @ feat(xf))
w += lr_d * (feat(xr) @ (1 - dr) - feat(xf) @ df) / n
# 2) Generator step: move fakes so D calls them real
z = rng.normal(size=n)
xf = a * z + b
df = sig(w @ feat(xf))
g = (1 - df) * (w[0] + 2 * w[1] * xf)   # d log D(G(z)) / d x
a += lr_g * np.mean(g * z)
b += lr_g * np.mean(g)
if step % 2000 == 0:
f = a * rng.normal(size=5000) + b
print(f"  step {step:4d}: fake mean={f.mean():.2f}  fake std={f.std():.2f}")
print("run A (lr_d=0.1,  lr_g=0.005)")
train(0.1, 0.005)
print("run B (lr_d=0.02, lr_g=0.02)")
train(0.02, 0.02)
print("target: mean=4.00  std=0.50")

Output:

run A (lr_d=0.1,  lr_g=0.005)
step    0: fake mean=-0.00  fake std=1.00
step 2000: fake mean=3.95  fake std=0.60
step 4000: fake mean=4.01  fake std=0.55
step 6000: fake mean=3.97  fake std=0.55
run B (lr_d=0.02, lr_g=0.02)
step    0: fake mean=-0.01  fake std=1.01
step 2000: fake mean=4.02  fake std=0.02
step 4000: fake mean=4.02  fake std=0.02
step 6000: fake mean=4.01  fake std=0.01
target: mean=4.00  std=0.50

The mode collapse problem

Real data usually has many modes: distinct clusters such as kettles that are red, steel, white, or black. Mode collapse happens when the generator discovers a few outputs that currently fool the discriminator and produces only those, ignoring the rest of the data. In the worst case it outputs nearly the same image for every z, as run B did.

Why does it happen? The generator is rewarded only for fooling the current discriminator, not for covering the whole dataset. If D is temporarily weak on 'steel kettles', G can pile everything there. D then learns to reject steel kettles, G jumps to another mode, and the two can chase each other around without ever covering all modes at once.

Why it is easy to miss A collapsed GAN can produce individually stunning images. If we only inspect a handful of samples, everything looks great. Always look at many samples side by side and measure diversity (FID captures some of this; precision and recall metrics for generative models separate quality from coverage).

• Minibatch discrimination / minibatch statistics: let D look at a whole batch, so a batch of identical fakes is easy to spot.
• Better objectives: Wasserstein loss with a gradient penalty, or other losses with smoother gradients.
• Balancing the players: tune learning rates and update ratios (as our run A vs run B shows).
• Conditioning: asking for a specific class (e.g. 'red') forces coverage of each class.

Training stability

Normal training minimises one loss downhill. A GAN searches for a balance point (an equilibrium) between two players whose goals conflict. Gradient steps on such games can circle around the balance point or fly away from it instead of settling. Typical symptoms:

• Oscillation: samples cycle between styles; losses swing back and forth.
• Vanishing generator gradients: D becomes near-perfect, D(G(z)) ≈ 0, and G stops learning.
• Divergence: outputs degrade into noise or saturated colours after looking promising.
• Hyperparameter sensitivity: a small change in learning rate can change success into collapse, as our two runs show.

Types of GANs (DCGAN, Conditional GAN, StyleGAN, CycleGAN)

Key GAN variants

1. Original GAN: Goodfellow et al. introduce the generator-discriminator game, with fully connected networks.
2. Conditional GAN (cGAN): Mirza and Osindero feed a label y to both G and D, so we can ask for a specific class: G(z, y). 'Generate a red kettle.'
3. DCGAN: Radford et al. give architecture rules for convolutional GANs (strided convolutions, batch norm, no pooling) that made image GANs train far more reliably, and show meaningful arithmetic in latent space.
4. CycleGAN: Zhu et al. translate between two image domains without paired examples (horses ↔ zebras, summer ↔ winter) using two generators and a cycle-consistency loss: translating there and back should return the original.
5. StyleGAN: Karras et al. at NVIDIA map z to an intermediate 'style' vector that controls each resolution level of the generator, giving very high-quality faces and control over coarse (pose) vs fine (hair texture) attributes.

Other notable members include pix2pix (paired image-to-image translation, such as sketch to photo), Progressive GAN (grow the resolution during training), BigGAN (large-scale class-conditional generation) and SRGAN/ESRGAN (super-resolution).

Pause and think: We have photos of kettles in daylight and, separately, photos of other kettles under warm evening light, but no pairs of the same kettle in both. Which GAN type fits a 'daylight → evening' converter?

CycleGAN. It is designed for unpaired image-to-image translation, using the cycle-consistency loss in place of matched pairs. pix2pix would need paired examples.

Real-world applications of GANs

Responsible use GANs made convincing fake faces and 'deepfake' videos easy to produce. Synthetic media can be used for fraud, harassment and misinformation. Label generated content, respect consent and likeness rights, and do not use synthetic people where real identities are expected.

Worked example, step by step

GAN losses are hard to read because there are two of them and neither simply falls. Let us compute both by hand for one tiny batch, then learn what the numbers mean. Our batch has two real kettle photos and two fakes. The discriminator outputs D = 0.9 and 0.6 on the real ones, and 0.2 and 0.4 on the fakes. We use natural logs and average over each pair, as the PyTorch sketch does.

One batch, both losses

1. D on real photos: D wants 1. Loss = −(ln 0.9 + ln 0.6) / 2 = (0.105 + 0.511) / 2 = 0.308.
2. D on fakes: D wants 0, so we score 1 − D. Loss = −(ln 0.8 + ln 0.6) / 2 = (0.223 + 0.511) / 2 = 0.367.
3. Discriminator loss: 0.308 + 0.367 = 0.675.
4. Generator loss (non-saturating): G wants D to say 1 on fakes. Loss = −(ln 0.2 + ln 0.4) / 2 = (1.609 + 0.916) / 2 = 1.263.
5. The balance point for reference: If D said 0.5 for everything, its loss would be ln 2 + ln 2 = 1.386 and G's would be ln 2 = 0.693.
6. Read the result: D's 0.675 is well below 1.386, and G's 1.263 is well above 0.693. So D is ahead right now. That is healthy as long as G keeps getting a useful signal.

Practice: try it yourself

We will not train anything this time. Instead we freeze four different generators and, for each, compute the best possible discriminator and the score of the game. This shows what a perfect critic would say about an untrained, a nearly right, a collapsed and a perfect generator.

practice_best_discriminator.py

import numpy as np
# For a FIXED generator, the best possible discriminator is known:
#   D*(x) = p_data(x) / (p_data(x) + p_g(x))
# We compute it on a grid for several generators and score the game.
x = np.linspace(-6, 10, 16001)
dx = x[1] - x[0]
def bell(mean, std):
"""Density of a normal distribution on the grid."""
return np.exp(-0.5 * ((x - mean) / std) ** 2) / (std * np.sqrt(2 * np.pi))
p_data = bell(4.0, 0.5)                          # real data: mean 4, std 0.5
tiny = 1e-300                                    # avoids log(0) where both are 0
def play(name, mean, std):
p_g = bell(mean, std)
d_star = p_data / (p_data + p_g + tiny)      # best D for this generator
# V = E_data[log D] + E_fake[log(1 - D)], as integrals over the grid
v = np.sum(p_data * np.log(d_star + tiny)) * dx \
+ np.sum(p_g * np.log(1 - d_star + tiny)) * dx
js = max(0.0, (v + np.log(4)) / 2)           # Jensen-Shannon divergence
at4 = d_star[np.argmin(np.abs(x - 4.0))]     # D*'s verdict at x = 4
print(f"{name:22s} V={v:+.3f}  JS={js:.3f}  D*(4)={at4:.2f}")
play("untrained  N(0, 1)", 0.0, 1.0)
play("close      N(3.5, 0.5)", 3.5, 0.5)
play("collapsed  N(4, 0.05)", 4.0, 0.05)
play("perfect    N(4, 0.5)", 4.0, 0.5)
print(f"reference: -log 4 = {-np.log(4):.3f},  log 2 = {np.log(2):.3f}")

Output:

untrained  N(0, 1)     V=-0.023  JS=0.682  D*(4)=1.00
close      N(3.5, 0.5) V=-1.163  JS=0.111  D*(4)=0.62
collapsed  N(4, 0.05)  V=-0.520  JS=0.433  D*(4)=0.09
perfect    N(4, 0.5)   V=-1.386  JS=0.000  D*(4)=0.50
reference: -log 4 = -1.386,  log 2 = 0.693

Now change it:

• Add play("too wide N(4, 1.0)", 4.0, 1.0). Predict first: is a generator that is too spread out punished more or less than the collapsed one (JS = 0.433)?
• Move the 'close' generator's mean from 3.5 to 3.9. Predict whether D*(4) moves toward 0.5 or away from it, and what happens to JS.
• Make the real data two clusters: p_data = (bell(2.0, 0.5) + bell(6.0, 0.5)) / 2, and test a generator N(2, 0.5) that covers only one of them. Predict whether JS lands near 0, near log 2, or in between.

Pause and think: The collapsed generator N(4, 0.05) has exactly the right mean, yet the best discriminator gives D*(4) = 0.09. Why would a perfect critic call a sample at x = 4 'probably fake'?

Because the collapsed generator puts almost all its samples in a tiny range around 4, while the real data spread theirs over a much wider range. At x = 4 there are about ten fake samples for every real one, so 'fake' is the smart guess there. Away from 4 the opposite holds: D* is close to 1, since the generator never produces those values. A good D notices the missing spread, not just the mean.

Pause and think: We log D loss ≈ 1.386 and G loss ≈ 0.693 at step 0 and again at step 5,000. Does that prove training has reached the ideal end point?

No. Those values only say that D outputs about 0.5 on everything. At step 0 that is because D knows nothing. Later it could be the true balance point, or D could have become too weak to tell anything apart. The losses look the same in all three cases, so we have to inspect many samples (and a metric such as FID) to know which one we are in.

Key takeaways

• A GAN pits a generator (noise → fake) against a discriminator (real or fake?).
• The minimax objective trains D as a classifier and G to fool it; in practice G uses the non-saturating loss.
• At the ideal equilibrium, fakes match the data and D outputs 0.5 everywhere.
• Mode collapse and unstable, oscillating training are the main failure modes; balance and better losses help.
• DCGAN, cGAN, CycleGAN and StyleGAN extended GANs to reliable, controllable and unpaired image generation.

Key terms

• Generator: The network that maps random noise to synthetic samples.
• Discriminator: The network that outputs the probability that a sample is real.
• Minimax game: A two-player objective that one player maximises and the other minimises.
• Non-saturating loss: The generator objective 'maximise log D(G(z))', which avoids vanishing gradients early on.
• Mode collapse: When a generator covers only a few modes of the data, producing low-variety outputs.
• Latent space: The space of noise vectors z that the generator maps to outputs.
• Cycle-consistency loss: A loss requiring that translating A → B → A returns the original input.

← 16.4 Diffusion Models: Iterative Denoising to Generate Images · 16.6 Variational Autoencoders: Learning a Compressed Latent Space →