Built by

Learning by Comparison

IG Igor Lima @igor17400 9 page

Every merged contribution is typed and scored at review. Chips next to a heading show who touched that section. Scores feed the contributor share of project support.

01 · Learning by Comparison

Introduction

What is self-supervised learning?

The Core Intuition

Rather than mapping inputs to fixed class labels, contrastive learning shapes representation space directly:

Pull Together representations of semantically similar examples (positives).
Push Apart representations of semantically different examples (negatives).

The result is an embedding space where proximity encodes semantic similarity, learned entirely from structure of the data rather than from explicit labels.

Contrastive learning: pull positives together, push negatives apart in embedding space — Pull positives together, push negatives apart in the embedding space.

Contrastive Training Objectives

Given a set of input samples with labels we seek an encoder such that:

Examples from the same class have similar embeddings.
Examples from different classes have dissimilar embeddings.

Let's unpack each piece of this formulation:

Input samples — the raw data points: images, sentences, audio clips, etc. Each lives in some input space (e.g. for images).
Labels — the class each sample belongs to. In supervised contrastive learning these define what counts as "same" vs "different". In self-supervised learning, we replace labels with data augmentations to define positives.
Encoder — a neural network (e.g. ResNet, ViT) with learnable parameters . It maps each high-dimensional input into a compact representation.
— the encoder maps from the input space to a -dimensional vector space. Typical values: or . This is the embedding space where we measure similarity.
Parameters — the weights and biases of the network. Training adjusts so that the geometry of reflects semantic relationships: nearby points share meaning, distant points do not.

Encoder maps scattered points in input space to clustered embeddings in R^d — The encoder maps from a high-dimensional input space, where classes are interleaved, to a compact embedding space where same-class points cluster together.

How can we then force our encoder to enforce this separation in our embedding space? One of the earliest answers came from Chopra et al., 2005:

"The idea is to learn a function that maps input patterns into a target space such that the norm in the target space approximates the 'semantic' distance in the input space."

distances

Distance metrics measure how far apart two points and are in . Two of the most common are the and norms.

distance (Manhattan)

Sum of the absolute differences along each dimension. It measures distance as if you were walking on a grid, where only horizontal and vertical moves are "allowed":

In 2D, the set of points at distance 1 from the origin forms a diamond (rotated square). The norm is more robust to outliers in individual dimensions because it doesn't square the differences.

L1 distance: Manhattan path between two points — distance follows axis-aligned paths.

distance (Euclidean)

The straight-line distance between two points:

In 2D, the set of points at distance 1 from the origin forms a circle. Squaring the differences means large deviations in any single dimension are penalised heavily.

L2 distance: straight-line path between two points — distance is the straight-line distance.

Why does it matter here?

The contrastive loss from Chopra et al., 2005 uses , while many modern methods (SimCLR, MoCo) use or cosine similarity. The choice affects how the embedding space is shaped:

encourages sparse representations dimensions can independently be zero.
encourages smooth representations points spread evenly on a hypersphere.
Cosine similarity ( on normalised vectors) ignores magnitude entirely and only compares direction.

The Contrastive Loss

A great resource to build intuition for contrastive learning is Weng, 2021. The blog post retrieves a quite intuitive equation to understand what we're trying to learn:

𝟙

𝟙

where is a hyperparameter defining the lower bound distance between samples of different classes. Let's break this down term by term:

$𝟙$ — an indicator function that equals 1 when the two samples belong to the same class, and 0 otherwise. This "activates" the first term only for positive pairs.
— the squared Euclidean distance between the embeddings. When this term is active (same class), minimising the loss pulls the representations together.
$𝟙$ — the opposite indicator: equals 1 when the samples belong to different classes. This activates the second term only for negative pairs.
— a hinge-like term. If the distance between negatives is already greater than , this term is zero (they're far enough). If the distance is less than , the loss pushes them apart until they reach the margin.
(margin) — controls how far apart negative pairs need to be. Too small and the embedding space is cramped; too large and the model wastes capacity enforcing unnecessary separation.

Contrastive loss: positive pairs pulled together, negative pairs pushed apart beyond margin epsilon — **Left**: positive pairs are pulled together by minimising their distance. **Right**: negative pairs are pushed apart until their distance exceeds the margin .

Now that the intuition has been built, the next sections will cover the methods that have been proposed across the literature to achieve better separation between negative and positive classes.