N-pair Loss

One negative at a time is too slow

"We propose to address the problem of slow convergence of triplet loss by allowing joint comparison among more than one negative examples — and propose the (N-1)-pair loss."

Sohn (2016) identifies a fundamental limitation of the triplet loss: each update only pushes the anchor away from one negative class at a time. When there are many classes, this is inefficient. The anchor might move away from one negative only to drift closer to another.

The (N-1)-pair loss (commonly called N-pair loss) fixes this by considering one positive and negatives simultaneously. Instead of asking "is the anchor closer to its positive than to this one negative?", it asks "is the anchor closer to its positive than to any of the negatives?" This leads to faster convergence and more uniformly structured embeddings.

Batch construction

The training batch is constructed by sampling pairs, one from each of different classes:

Each pair contains an anchor and a positive from the same class. The key insight is that the positives of other pairs serve as negatives. For anchor , the points are all from different classes and therefore act as negatives.

This is efficient: from pairs ( examples), we get training signals, each comparing against negatives. No extra sampling is needed beyond the pairs themselves.

The N-pair loss

The loss for a single anchor with positive and negatives is:

and the total loss over the batch is:

Let us unpack the terms inside:

• is the dot product between the anchor and its positive. We want this to be large (high similarity).
• is the dot product between the anchor and a negative. We want this to be small.
• measures how much closer the negative is compared to the positive. If this is positive, the negative is winning.
• The and create a softmax-like penalty. Negatives that are closer than the positive contribute exponentially more to the loss.

Connection to softmax cross-entropy

Let us walk through this step by step. First, recall the standard softmax cross-entropy loss for classification. Given logits and a correct class , the loss is:

This can also be written as , i.e. the negative log-probability assigned to the correct class.

Now, in the N-pair setting, define the "logits" as the dot-product similarities between the anchor and each positive:

The correct class is (the anchor's own positive). Substituting into the cross-entropy formula:

Now let us expand the N-pair loss. Starting from:

We can factor the subtraction out of the exp. Since :

Rewriting the as and combining:

Using :

which is exactly the softmax cross-entropy loss with logits and correct class . In other words, N-pair loss is equivalent to classifying which positive belongs to the anchor using a softmax over similarities.

This connection is important because it links metric learning directly to classification, and it foreshadows the InfoNCE loss, which generalises this idea further by adding a temperature parameter.

Triplet loss as a special case

When (one positive, one negative), the N-pair loss reduces to:

which is a soft version of the triplet loss (using softplus instead of a hinge). So the triplet loss is a special case of N-pair loss with only one negative class. The power of N-pair comes from scaling this to negatives simultaneously.

N-pair Loss in PyTorch

A minimal implementation. The loss reduces to a cross-entropy over the similarity matrix, where the diagonal entries (anchor-positive pairs) are the correct classes.

# npair_loss_example.py
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

# ── Synthetic data: sample N pairs from N classes ─────────────────

def sample_npairs(n_classes=8, noise=0.8):
    """Sample one anchor-positive pair per class."""
    # Random class centres on a circle
    angles = torch.linspace(0, 2 * 3.14159, n_classes + 1)[:n_classes]
    centres = torch.stack([angles.cos(), angles.sin()], dim=-1)

    anchors  = centres + noise * torch.randn(n_classes, 2)
    positives = centres + noise * torch.randn(n_classes, 2)
    return anchors, positives

# ── Embedding network ─────────────────────────────────────────────

class EmbeddingNet(nn.Module):
    def __init__(self, embed_dim=32):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(2, 64),
            nn.ReLU(),
            nn.Linear(64, embed_dim),
        )

    def forward(self, x):
        return F.normalize(self.net(x), p=2, dim=-1)

# ── N-pair loss ───────────────────────────────────────────────────

def npair_loss(anchors_emb, positives_emb):
    """
    N-pair loss = cross-entropy over the similarity matrix.
    anchors_emb:   (N, d)
    positives_emb: (N, d)
    """
    # Similarity matrix: (N, N) — entry (i,j) = f(x_i)^T f(x_j^+)
    logits = anchors_emb @ positives_emb.T

    # Correct class for anchor i is positive i (the diagonal)
    labels = torch.arange(logits.size(0))

    return F.cross_entropy(logits, labels)

# ── Training loop ─────────────────────────────────────────────────

model = EmbeddingNet()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

for step in range(3000):
    a, p = sample_npairs(n_classes=8)

    e_a = model(a)
    e_p = model(p)

    loss = npair_loss(e_a, e_p)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (step + 1) % 500 == 0:
        with torch.no_grad():
            logits = e_a @ e_p.T
            preds = logits.argmax(dim=-1)
            acc = (preds == torch.arange(len(preds))).float().mean()
        print(f"Step {step+1:4d} | loss = {loss.item():.4f} | match acc = {acc.item():.2%}")

A few things to notice:

• The entire N-pair loss is just F.cross_entropy applied to the similarity matrix. The diagonal entries are the correct classes.
• The similarity matrix anchors_emb @ positives_emb.T computes all dot products in a single matrix multiply.
• No explicit margin parameter is needed. The softmax naturally handles the competition between positive and negative similarities.