Lesson 18.1 · 29 min

JEPA: LeCun's Vision for World Model AI

If you see half of a dog behind a fence, you instantly “know” the rest is there, without imagining every hair. Can a machine learn to predict like that?

In short: JEPA (Joint Embedding Predictive Architecture) is a self-supervised learning design proposed by Yann LeCun. Instead of predicting the missing part of an input pixel by pixel, it encodes both the visible part and the hidden part into embeddings and trains a predictor to guess the hidden part's embedding from the visible part's. This lets the model ignore unpredictable detail and focus on meaning, avoids the need for negative examples, and is a key building block in LeCun's vision of world models. I-JEPA applies it to images and V-JEPA to video.

Learning by observing the world

Babies learn an astonishing amount before anyone teaches them words: that objects keep existing when hidden, that unsupported things fall, that a rolling ball continues rolling. They learn mostly by watching and by predicting what happens next. Animals do the same. Nobody labels these experiences; the world itself is the teacher.

Machine learning has a name for learning without human labels: self-supervised learning. We hide part of the data and train the model to predict it from the rest. Large language models do this with text (predict the next token). The open question for images and video is: what exactly should the model predict?

Everyday intuition: the car behind a truck When a car disappears behind a truck on the highway, we predict that it will reappear on the other side, at about the right time, still a car. We do not predict the exact reflections on its windscreen or the pattern of dust on its doors. We predict the important, predictable things and ignore the rest. JEPA is designed to learn this kind of prediction.

In 2022 Yann LeCun (then Meta's chief AI scientist) published a position paper, A Path Towards Autonomous Machine Intelligence, describing an architecture for machines that learn models of the world by observation and use them to plan. Its proposed modules include perception, a world model, a cost module, short-term memory, an actor and a configurator. JEPA was proposed as the way to train the world-model part: learn to predict, but in an abstract representation space rather than in raw sensory detail.

What does JEPA mean? Embeddings and representation space

Read the name in pieces. Joint Embedding: both inputs, the visible part x and the hidden part y, are turned into embeddings by encoders. Predictive: a predictor tries to guess y's embedding from x's embedding. Architecture: it is a design pattern, not one specific model.

An embedding (or representation) is a list of numbers that summarises an input. A good image encoder might turn a 224×224 photo (about 150,000 pixel values) into, say, a few hundred numbers per image patch, where similar content gives similar numbers. The space of all such vectors is the representation space or latent space. Predicting in that space means predicting summaries, not pixels.

The problems with predicting pixels and with contrastive methods

Problem 1: predicting raw pixels. Generative self-supervised methods, such as masked autoencoders (MAE), hide image patches and reconstruct them pixel by pixel. The trouble is that much of a picture is unpredictable detail: the exact leaves on a tree, grain in a carpet, noise. A pixel loss forces the model to spend capacity on those details, and when it cannot know them it predicts a blurry average. The representations still turn out useful, but a lot of effort goes into things that do not matter for understanding.

Problem 2: contrastive methods. Joint-embedding methods like SimCLR avoid pixels: they make embeddings of two augmented views of the same image close (positives) and embeddings of different images far apart (negatives). Without the negatives, the encoders could cheat by outputting the same vector for everything, a failure called collapse. But negatives bring costs: many are needed (large batches or memory banks), and the method relies on hand-crafted augmentations (crops, colour jitter) that encode human assumptions about what should not matter.

Pause and think: Why would a pixel-prediction model produce blurry images when the hidden region contains random texture?

With a squared-error pixel loss, the best guess under uncertainty is the average of all possible textures, which looks like a smooth blur. The model is penalised for not knowing details that were never predictable from the visible part.

The core idea and building blocks of JEPA

The core idea in one sentence: predict the representation of the missing part, not the missing part itself. Because the target encoder may drop unpredictable details from its embedding, the predictor is only asked to predict what is predictable.

The energy-based view in simple words

LeCun often describes JEPA using energy-based models (EBMs). An energy function E(x, y) gives a single number for a pair: low energy when y is a compatible continuation of x, high energy when it is not. There are no probabilities that must sum to 1, just a score of compatibility.

Training must do two things: make energy low on real (x, y) pairs and make sure it is not low everywhere. The second part is the collapse problem in different clothes. If the encoders output a constant vector, energy is zero for every pair, which is useless. There are two families of fixes: contrastive methods push energy up on wrong pairs (negatives), while regularised or architectural methods limit how much of the space can have low energy, for example by keeping embeddings' variance high (as in VICReg) or by the EMA target with stop-gradient used in I-JEPA. JEPA favours the non-contrastive route.

Why the latent variable z If the car behind the truck might exit in the left or right lane, one deterministic prediction would average the two. A latent variable z lets the predictor represent “which of several valid futures”, so each can have low energy without blurring them together.

Seeing the idea in numbers

Here is a small numpy experiment. Each “scene” has 2 hidden facts. The visible view shows those facts clearly; the hidden view shows the same facts plus lots of unpredictable detail. We compare predicting the hidden view's pixels with predicting its 2-number embedding, then show collapse and the EMA update.

jepa_toy.py

import numpy as np
rng = np.random.default_rng(0)
N, P = 2000, 50                       # 2000 samples, 50 "pixels" per view
z = rng.normal(size=(N, 2))           # the hidden facts of each scene (2 numbers)
M_ctx, M_tgt = rng.normal(size=(2, P)), rng.normal(size=(2, P))
x_ctx = z @ M_ctx + 0.1 * rng.normal(size=(N, P))   # visible part (context)
x_tgt = z @ M_tgt + 2.0 * rng.normal(size=(N, P))   # hidden part: same facts + lots of
# unpredictable detail (leaves, noise)
def fit(X, Y):                        # least-squares linear predictor
W, *_ = np.linalg.lstsq(X, Y, rcond=None); return W
# (a) Generative: predict every target pixel from the context pixels
err_pix = np.mean((x_ctx @ fit(x_ctx, x_tgt) - x_tgt) ** 2)
var_pix = x_tgt.var()
# (b) JEPA-style: encode both views to 2-D, predict the target *embedding*
U, S, Vt = np.linalg.svd(x_tgt - x_tgt.mean(0), full_matrices=False)
enc_tgt = Vt[:2].T                    # target encoder keeps the 2 main directions
s_tgt = x_tgt @ enc_tgt
s_ctx = x_ctx @ fit(x_ctx, z)         # context encoder (2-D)
err_lat = np.mean((s_ctx @ fit(s_ctx, s_tgt) - s_tgt) ** 2)
print(f"pixel space : error {err_pix:.2f} of variance {var_pix:.2f} -> {err_pix/var_pix:.0%} unexplained")
print(f"latent space: error {err_lat:.2f} of variance {s_tgt.var():.2f} -> {err_lat/s_tgt.var():.0%} unexplained")
# (c) Collapse: an encoder that outputs a constant gets a perfect zero loss
collapsed = np.zeros((N, 2))
print("collapsed encoder loss:", np.mean((collapsed - collapsed) ** 2), "| embedding variance:", collapsed.var())
# (d) EMA target encoder update used by I-JEPA-style training
theta_ctx, theta_tgt, m = 1.0, 0.0, 0.996
for _ in range(1000): theta_tgt = m * theta_tgt + (1 - m) * theta_ctx
print(f"target weights after 1000 EMA steps: {theta_tgt:.3f} (slowly follows 1.0)")

Output:

pixel space : error 3.92 of variance 5.94 -> 66% unexplained
latent space: error 4.11 of variance 52.22 -> 8% unexplained
collapsed encoder loss: 0.0 | embedding variance: 0.0
target weights after 1000 EMA steps: 0.982 (slowly follows 1.0)

In this toy the encoders are hand-made, so it shows the why (predictable vs unpredictable content, and collapse) rather than real JEPA training, where both encoders and the predictor are deep networks learned together.

Pause and think: In output line 3, the collapsed encoder has a perfect loss of 0.0. Why is that a bad sign rather than a good one?

Because the embedding variance is also 0: every input maps to the same vector, so the representation carries no information. A loss that only measures prediction agreement can be minimised trivially, which is why JEPA adds anti-collapse mechanisms.

How I-JEPA works (for images)

I-JEPA (Image-JEPA) was published by Meta researchers (Assran et al.) in 2023. It applies the recipe to images using Vision Transformers, which split an image into a grid of patches and produce one embedding per patch.

One I-JEPA training step

1. Pick targets: Sample several (four in the paper) rectangular target blocks, each covering a modest fraction of the image. These are what we will predict.
2. Pick the context: Sample one large context block covering most of the image, then remove any patches that overlap the targets, so the answer is not visible.
3. Encode: The context encoder embeds only the context patches. The target encoder (EMA weights) embeds the full image, and we take the embeddings at the target patch positions.
4. Predict: For each target block, the predictor receives the context embeddings plus mask tokens carrying the target's positions, and outputs predicted embeddings for those patches.
5. Learn: The loss is the average squared distance between predicted and target patch embeddings. Backprop updates the context encoder and predictor; the target encoder follows by EMA.

Two design choices matter. Targets are large, semantic blocks (not single tiny patches), so predicting them requires understanding object parts and layout rather than copying neighbouring texture. And there are no hand-crafted augmentations: the masking itself provides the learning signal. The paper reported strong results on image classification and lower-level tasks with good compute efficiency compared with earlier methods.

V-JEPA and the world-model vision

Video adds time. V-JEPA (2024) masks regions across space and time in video clips and predicts their representations, learning about motion and how scenes evolve purely from unlabelled video. V-JEPA 2 (2025) scaled this up to over a million hours of video and then added an action-conditioned version trained on a comparatively small amount of robot data, so the predictor learns “what will the scene look like, in embedding space, if the robot does this action?”. Meta showed it being used to plan simple robot manipulation by searching for actions whose predicted outcome is close to a goal image's embedding.

From position paper to video world models

1. A Path Towards Autonomous Machine Intelligence: LeCun's position paper lays out the modular architecture and proposes JEPA for learning world models.
2. I-JEPA: Image JEPA: predicts embeddings of masked image blocks with ViTs and an EMA target encoder.
3. V-JEPA: Extends feature prediction to video, masking regions in space and time.
4. V-JEPA 2: Much larger video pre-training plus an action-conditioned predictor used for robot planning.

This is the bridge to the next lesson. A world model predicts how the world will change, especially in response to actions. JEPA supplies a way to learn such a predictor in an abstract space, where planning is cheaper and not distracted by pixel detail.

Comparison, and when and why JEPA matters

Why it matters: if future AI systems need to understand and plan in the physical world (robots, video understanding, autonomous agents), they need to predict consequences efficiently. Predicting everything at the pixel level is expensive and unnecessary; predicting abstract states is closer to how we seem to reason. JEPA is one leading proposal for learning those abstract predictive models from observation.

Common misunderstandings JEPA is not a generative model: it does not produce images or video by itself (a separate decoder would be needed). It is not yet a replacement for LLMs on language tasks. And “no negatives” does not mean “no collapse risk”: the EMA target, stop-gradient or variance regularisation are essential. Whether JEPA-style world models will be the path to more general intelligence is an open research debate, not a settled fact.

Going one level deeper

We said collapse means every input gets the same embedding. In practice the more common failure is quieter. The embedding keeps a few useful dimensions and the rest go flat. This is called dimensional collapse (or partial collapse). The prediction loss looks healthy, because a flat dimension is trivially easy to predict. So we cannot find it by watching the loss. We have to look at the embeddings themselves.

A simple check: take a batch of embeddings and compute the standard deviation of each dimension across the batch. A healthy dimension varies from input to input. A dead one barely moves. Methods such as VICReg turn this check into a training penalty: for each dimension, add max(0, 1 − std) to the loss. Here is that penalty worked by hand for a made-up 3-dimension embedding.

The average penalty is (0 + 0.10 + 0.98) / 3 = 0.36, and nearly all of it comes from dimension 3. The gradient of this term pushes the encoder to spread that dimension out again. I-JEPA does not use this penalty; it relies on the EMA target and stop-gradient instead. But the per-dimension std is still the right thing to monitor in any JEPA-style run.

The EMA momentum m is the other dial worth understanding in numbers. After k steps, the target encoder has absorbed a share 1 − mᵏ of a change in the context encoder. A handy rule: the target averages over roughly 1 / (1 − m) recent steps.

Reading the momentum dial

1. Too low: With m = 0.9 the target is almost a copy of the context encoder. Both sides can drift together towards a constant output, so the protection against collapse is weak.
2. Too high: With m very close to 1 the target hardly moves. It is stable, but it keeps offering stale targets, so learning is slow.
3. In between: Values near 0.996, often raised towards 1 as training goes on, give a target that is steady from one step to the next but still improves over thousands of steps.
4. What to watch: Whatever m we choose, we log the per-dimension std of the target embeddings. If it sinks towards zero, the run is collapsing, however good the loss looks.

Practice: try it yourself

We will build a small collapse monitor. Three hand-made encoders embed the same scenes: a healthy one, one that silently drops a dimension, and one that outputs a constant. For each we print the JEPA prediction loss, the std of each embedding dimension, the variance penalty, and how much of the true scene facts a linear readout can recover. Watch which number tells the truth.

practice_collapse_monitor.py

import numpy as np
rng = np.random.default_rng(0)
N = 500
facts = rng.normal(size=(N, 2))                  # 2 true facts per scene
ctx = facts + 0.1 * rng.normal(size=(N, 2))      # visible view (a little noise)
tgt = facts + 0.1 * rng.normal(size=(N, 2))      # hidden view (different noise)
encoders = {                                     # three candidate 2 -> 2 encoders
"healthy":   np.array([[1.0, 0.0], [0.0, 1.0]]),   # keeps both facts
"one dim":   np.array([[1.0, 0.0], [0.0, 0.0]]),   # silently drops fact 2
"collapsed": np.zeros((2, 2)),                     # outputs a constant
}
def fit(X, Y):                                   # least-squares linear map X -> Y
return np.linalg.lstsq(X, Y, rcond=None)[0]
print("encoder    pred loss  std per dim   var penalty  facts recovered")
for name, E in encoders.items():
s_ctx, s_tgt = ctx @ E, tgt @ E              # embeddings of both views
pred = s_ctx @ fit(s_ctx, s_tgt)             # predictor: context -> target
loss = ((pred - s_tgt) ** 2).mean()          # the JEPA prediction loss
std = s_tgt.std(axis=0)
penalty = np.maximum(0, 1 - std).mean()      # hinge: wants std >= 1 per dim
# Probe: how much of the true facts can a linear readout get back?
resid = facts - s_tgt @ fit(s_tgt, facts)
r2 = 1 - resid.var() / facts.var()
print(f"{name:9s}  {loss:9.4f}  [{std[0]:.2f}, {std[1]:.2f}]  {penalty:11.2f}  {r2:14.0%}")

Output:

encoder    pred loss  std per dim   var penalty  facts recovered
healthy       0.0195  [0.98, 0.97]         0.02             99%
one dim       0.0099  [0.98, 0.00]         0.51             50%
collapsed     0.0000  [0.00, 0.00]         1.00              0%

Now change it:

• Raise the view noise from 0.1 to 0.5 in both views. Predict which encoder's prediction loss rises the most, and whether the std and penalty columns change much.
• Add an encoder that shrinks instead of dropping: "tiny dim": np.array([[1.0, 0.0], [0.0, 0.01]]). Predict its penalty and its “facts recovered”. Is a tiny dimension the same as a dead one for a linear probe?
• Change the penalty target from 1 to 0.5 in the penalty line. Predict which rows change, and what a lower target means for how spread out the embeddings must be.

Pause and think: Rank the three encoders by prediction loss, then by “facts recovered”. What do the two rankings tell us?

By loss, collapsed is best (0.0000), then one dim (0.0099), then healthy (0.0195). By facts recovered the order is exactly reversed: 99%, 50%, 0%. The prediction loss rewards having less to predict, so on its own it prefers the least informative encoder. That is why a JEPA needs an anti-collapse mechanism, and why we monitor variance and not only the loss.

Pause and think: The “one dim” encoder has about half the prediction loss of the healthy one. Where did the other half go?

Each embedding dimension carries the view noise of its own fact, and that noise cannot be predicted from the other view. The healthy encoder pays this unpredictable cost on two dimensions. The one-dim encoder outputs a constant 0 on the second dimension, which is predicted perfectly, so it pays the cost only once. It lowered its loss by throwing information away, and the std column [0.98, 0.00] is what gives it away.

Key takeaways

• Self-supervised learning learns from unlabelled data by hiding part of it and predicting it.
• JEPA predicts the embedding of the hidden part, not its pixels, so it can ignore unpredictable detail.
• Building blocks: context encoder, EMA target encoder, predictor, and optional latent variable.
• Collapse (constant embeddings) is the central risk; EMA, stop-gradient or variance regularisers prevent it.
• I-JEPA works on image blocks; V-JEPA on video; V-JEPA 2 adds action conditioning for planning.
• JEPA is a leading proposal for learning world models from observation, still an active research area.

Key terms

• Self-supervised learning: Learning from unlabelled data by predicting hidden parts of the data from other parts.
• Representation (embedding): A vector of numbers summarising an input so that similar content gives similar vectors.
• JEPA: Joint Embedding Predictive Architecture: predicts the embedding of a hidden part of the input from the embedding of the visible part.
• Representation collapse: A failure where encoders map every input to (nearly) the same vector, making the loss trivially small.
• EMA target encoder: An encoder whose weights are an exponential moving average of the trained encoder's weights, giving stable targets.
• Energy-based model: A model that scores compatibility of input pairs with an energy: low for compatible, high for incompatible.

← 17.11 How do Voice And Video Call Work? · 18.2 World Models: Teaching AI to Simulate Its Environment →