Multi-Modal Expansion: Applying predictive architectures to sensory data beyond pixels and text.
JEPA principles are modality-agnostic. Whether processing sound waves or geometric point clouds, the objective remains the same: predict abstract features of masked regions from visible context.
Audio-JEPA learns semantic representations of sound by masking portions of a spectrogram. 3D-JEPA (Point-JEPA) processes spatial data, predicting the geometry of occluded objects in a 3D environment.
Click to mask frequency/time blocks. The model predicts the latent audio features.
Drag to rotate. Blue points are visible context; Red points are predicted geometry.