Noise-Contrastive Estimation

What problem does NCE solve?

"The idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise, using the model log-density function in the regression nonlinearity."

Gutmann and Hyvärinen (2010) consider the problem of estimating unnormalised statistical models. Before diving into their solution, we need to understand what this problem actually involves. We will unpack four ideas: (1) What an unnormalised model is; (2) What it means to observe data from an unknown density; (3) How we structure our search with a parameterised family; (4) And what it means for the true density to live inside that family.

(1) What is an unnormalised statistical model?

A probability density function must integrate to one. This is the basic rule that makes probabilities meaningful: if we sum up the likelihood of every possible outcome, the total must be exactly one.

In many practical settings, however, we can only write down a function that tells us the relative likelihood of different inputs. We call this function the unnormalised model and denote it . Let us break this notation apart:

• is the input vector. What represents depends on the domain. It could be a vector of pixel intensities in an image, a sequence of word embeddings, sensor readings from a robot, or any other high-dimensional observation we want to model.
• is the parameter vector that controls the shape of the model. For a Gaussian, would contain the mean and covariance. For a neural network, would be the collection of all weights and biases. Changing changes which inputs the model considers likely.
• is the unnormalised score that the model assigns to input . The superscript distinguishes it from the normalised density ; it signals that this is the "raw" form before dividing by . It is always non-negative, and higher values mean the model considers more plausible. Crucially, these scores do not need to sum (or integrate) to one across all possible .

To turn these raw scores into a valid probability density, we divide by the normalisation constant (also called the partition function):

Here is simply the total area under the unnormalised curve. After dividing by , the resulting function integrates to one and is a proper density.

The difficulty is that computing requires evaluating an integral over the entire input space. In low dimensions this may be feasible, but when lives in a high-dimensional space (images, audio, text), the integral quickly becomes intractable.

This is not a minor inconvenience; it is the central obstacle that NCE is designed to overcome. The problem arises across many model families, including Markov random fields, energy-based models, and deep generative networks.

(2) The unknown data density

Suppose we observe samples of a random vector , collected into a dataset . These samples were generated by some process whose probability density we denote . We cannot evaluate directly; that is, all we have access to are the samples themselves.

The task of density estimation is to use these samples to learn a model that approximates . In other words, we want to find a function that tells us how likely any given is under the data-generating process, using only the finite set of observations we have collected.

(3) Parameterised families

To make the estimation problem tractable, we choose a parameterised family of candidate densities , where is a vector of parameters. Each value of produces a different density. Changing changes the shape of the model: it might shift a peak, widen a mode, or redistribute probability mass across the input space.

The estimation problem then reduces to a search: which value of makes best match the observed data?

(4) The true parameters

We make one key assumption: the true data density belongs to our model family. Formally, there exists some parameter vector such that

This is the well-specified (or realisable) setting. It means the model family is rich enough to contain the true density as one of its members. The goal of estimation is to recover from the observed data.

With all four pieces in place, we can now state the problem precisely: given samples from an unknown density , and given a parameterised family of unnormalised models , find the parameters without having to compute the intractable partition function .

The normalisation constraint

Any solution to this estimation problem must yield a properly normalised density , meaning

Here denotes our estimate of the true parameters . The hat notation signals that is a quantity we compute from data, as opposed to the fixed but unknown ground truth . A good estimator produces an that is close to .

This normalisation requirement defines a constraint on the optimisation problem. In principle, the constraint can always be satisfied by dividing by as we saw earlier. In practice, however, computing is the very thing we cannot do.

Why not maximum likelihood?

A natural first idea would be to treat as an additional parameter of the model and estimate it alongside using Maximum Likelihood Estimation (MLE). This does not work: the likelihood can be made arbitrarily large by letting approach zero, so MLE has no finite optimum.

This failure has motivated methods that estimate the model directly through , bypassing the integral that defines . Two prominent examples are contrastive divergence (Hinton, 2002) and score matching (Hyvärinen, 2005).

Gutmann and Hyvärinen (2010) propose a different approach. Their key insight is to recast density estimation as a binary classification problem: learn to discriminate between samples from the true data distribution and samples from an artificially generated noise distribution . Both the parameters and the normalisation constant can be estimated by maximising the same objective function. Because the method relies on noise with which the data is contrasted, they call it noise-contrastive estimation (NCE).

Noise-Contrastive Estimation (NCE)

Definition of the estimator

We saw that treating as a free parameter breaks MLE. NCE takes a different route: instead of estimating through an integral, it absorbs the normalisation constant into the model as a learnable parameter. Concretely, we define

Why work in log-space?

• Normalisation means dividing by : .
• Taking logs turns the division into a subtraction: .
• So the scalar plays the role of , and the additive form is simply the log-space version of dividing by the partition function.

Here groups together the original model parameters and the new scalar . The parameter serves as an estimate of the negative log-partition function, i.e. . Note that will only integrate to one for some specific choice of ; for all other values it remains unnormalised. The estimation procedure itself will drive toward the correct value.

Next, we introduce a noise distribution from which we can both sample and evaluate the density. Denote by the observed dataset, consisting of observations of , and by an artificially generated dataset of noise with distribution .

The NCE estimator is defined as the that maximises the following objective function:

where

The function is the logistic sigmoid applied to the log-ratio . If we denote the logistic function by , we can write .

Intuitively, compares how likely a point is under the model versus under the noise.

• When the model assigns higher density than the noise, is positive and is close to 1.
• When the noise assigns higher density, is negative and is close to 0.

The objective rewards the model for correctly assigning high to data points and low to noise points. Take a look at the figure below which illustrates this process:

Notice that and play very different roles. The model density is what we are trying to learn; its shape changes as we update . The noise density , on the other hand, is fixed and known. We choose it ourselves (for example, a Gaussian) and use it to generate the artificial samples . The classifier succeeds when the model density is high where data actually appears and low everywhere else, which is exactly what a good density estimate should do.

Connection to supervised learning

The objective may look familiar: it is the log-likelihood of a logistic regression model that discriminates the observed data from the noise . This connection to supervised learning provides the central intuition behind NCE: by learning to tell data apart from noise, we learn the statistical structure of the data itself.

To make this connection explicit, consider the union of both datasets, , and assign each point a binary class label :

• if the point came from the data
• if the point came from the noise

Since the data density is unknown, we model the class-conditional density for with our parametric model . The class-conditional densities are:

Because we draw samples from each class, the prior probabilities are equal: . We can now apply Bayes' rule to compute the posterior probability that a point came from the data. Recall that Bayes' rule states:

The denominator is the total probability of observing regardless of its class. By the law of total probability:

Substituting the class-conditionals and , together with the equal priors :

The factors cancel in numerator and denominator, giving:

The class labels are Bernoulli-distributed, so the log-likelihood of the parameters over the combined dataset is

Substituting for the data points and for the noise points, this becomes

which is, up to the factor , exactly our objective . In other words, maximising the NCE objective is equivalent to fitting a logistic regression that classifies data versus noise.

But why does this avoid normalisation?

When I was first studying the NCE paper, this was the part that confused me the most. We start with a model that does not integrate to one, we train a classifier to tell data from noise, and somehow we end up with a properly normalised density.

I kept asking myself: where did the normalisation come from?

Here is the intuition that helped me to understand. The key observation is that NCE never needs to evaluate in isolation. The objective only ever uses the ratio between the model density and the noise density, . This ratio is what the classifier sees.

Now consider what happens if the model is incorrectly normalised:

• If is too large (the model overestimates densities everywhere):
  • The ratio will be inflated.
  • The classifier will assign to everything, including noise points.
  • This hurts the second term of the objective, , which wants to be small on noise.
• If is too small (the model underestimates densities):
  • The ratio will be deflated.
  • The classifier will assign everywhere, including data points.
  • This hurts the first term, , which wants to be large on data.

The only way to maximise both terms simultaneously is for to be high where the data actually lives and low where only noise appears. This forces the model to assign the right amount of total probability mass, not too much, not too little.

In other words, correct normalisation emerges as a consequence of optimal classification, not as an explicit constraint.

This is why the learnable parameter , which controls the overall scale of (remember that ), converges to at the optimum. The classifier "discovers" the right normalisation constant because any other value would make the classification worse.

Properties of the estimator

But why does maximising actually recover the true density?

Gutmann and Hyvärinen (2010) prove three key results. We state them informally here; the full proofs can be found in the original paper.

1. The objective has the right optimum

In the infinite-data limit, the population objective attains its maximum when , i.e. when the model matches the true data density. Moreover, there are no other extrema, provided that the noise density is nonzero wherever the data density is nonzero. This condition is easy to satisfy by choosing a broad noise distribution such as a Gaussian.

A crucial point: this result holds without imposing any normalisation constraint. Unlike MLE (where must integrate to one), the NCE objective automatically finds a properly normalised solution. The estimation procedure learns both the shape of the density () and the correct normalisation constant () simultaneously.

2. Consistency

As the number of samples grows, the NCE estimator converges in probability to the true parameters . In other words, given enough data, NCE recovers the correct model, including the correct value of the log-normalisation constant .

3. Asymptotic normality

For large , the estimation error is approximately Gaussian with mean zero. This means we can construct confidence intervals and quantify uncertainty, just as with MLE. Furthermore, if the model is powerful enough to represent exactly, the parameter converges to the true value of , so the model effectively self-normalises.

NCE in six steps

Noise-contrastive estimation (Gutmann and Hyvärinen, 2010) provides a way to learn unnormalised probability distributions without ever computing the intractable partition function. Here is the full picture (Balestriero et al., 2023):

1. Setup. We observe i.i.d. samples from an unknown distribution . We want to approximate with a parametric function without enforcing during training.

2. Introduce noise. We draw noise samples and form a mixture: flip a Bernoulli coin with , then observe if (data) or if (noise).

3. Apply Bayes' rule. Writing for the noise-to-data ratio, the posterior probability that a sample came from the data is

4. Parametrise. We model with learnable parameters .

5. Train. Minimise the negative log-likelihood of the logistic regression (the standard binary classification loss):

6. Result. At the optimum, provided that . If is powerful enough (e.g. a deep network), one can fix and the model will self-normalise.

In short, NCE turns an intractable density estimation problem into a tractable binary classification problem. The noise distribution provides a reference against which the model is compared, and the classifier learns to tell data from noise. As a byproduct, the model converges to the true data density, including the correct normalisation. This idea of "learning by comparison" is at the heart of many modern self-supervised methods, most notably InfoNCE.

NCE in PyTorch

To make everything concrete, here is a minimal PyTorch implementation of NCE. We estimate a 2D density where the true distribution is a mixture of Gaussians. The model is a small MLP that outputs the unnormalised log-density , plus a learnable scalar .

# nce_example.py
import torch
import torch.nn as nn
import torch.optim as optim

# ── True distribution: mixture of 3 Gaussians in 2D ──────────────

def sample_data(n):
    """Sample from a mixture of three 2D Gaussians."""
    means = torch.tensor([[-2.0, 0.0], [2.0, 0.0], [0.0, 3.0]])
    k = torch.randint(0, 3, (n,))
    samples = means[k] + 0.5 * torch.randn(n, 2)
    return samples

# ── Noise distribution: standard 2D Gaussian ─────────────────────

noise_dist = torch.distributions.MultivariateNormal(
    torch.zeros(2), torch.eye(2)
)

def sample_noise(n):
    return noise_dist.sample((n,))

def log_p_noise(y):
    return noise_dist.log_prob(y)

# ── Unnormalised model: MLP + learnable c ─────────────────────────

class NCEModel(nn.Module):
    def __init__(self, hidden=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(2, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
            nn.Linear(hidden, 1),       # outputs ln p_m^0(x; alpha)
        )
        self.c = nn.Parameter(torch.zeros(1))  # log-normalisation constant

    def log_pm(self, x):
        """Compute ln p_m(x; theta) = ln p_m^0(x; alpha) + c."""
        return self.net(x).squeeze(-1) + self.c

# ── NCE objective ─────────────────────────────────────────────────

def nce_loss(model, x_data, y_noise):
    """
    Compute -J_T(theta).
    We minimise the negative objective (equivalent to maximising J_T).
    """
    # Log-ratio G(u; theta) = ln p_m(u; theta) - ln p_n(u)
    G_data  = model.log_pm(x_data) - log_p_noise(x_data)
    G_noise = model.log_pm(y_noise) - log_p_noise(y_noise)

    # h(u; theta) = sigmoid(G(u; theta))
    # J_T = (1/2T) * sum[ ln h(x_t) + ln(1 - h(y_t)) ]
    # Using log-sigmoid for numerical stability:
    #   ln h(u)     = ln sigmoid(G)  = log_sigmoid(G)
    #   ln(1-h(u))  = ln sigmoid(-G) = log_sigmoid(-G)
    loss_data  = -torch.nn.functional.logsigmoid(G_data).mean()
    loss_noise = -torch.nn.functional.logsigmoid(-G_noise).mean()

    return (loss_data + loss_noise) / 2

# ── Training loop ─────────────────────────────────────────────────

model = NCEModel()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
T = 1024  # samples per batch

for step in range(5000):
    x = sample_data(T)
    y = sample_noise(T)

    loss = nce_loss(model, x, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (step + 1) % 1000 == 0:
        print(f"Step {step+1:5d} | loss = {loss.item():.4f} | c = {model.c.item():.4f}")

# ── After training ────────────────────────────────────────────────
# model.log_pm(x) now approximates ln p_d(x)
# model.c has converged to approximately -ln Z(alpha)

print(f"\nLearned c = {model.c.item():.4f}")
print("(If the model has self-normalised, c should be close to 0)")

A few things to notice in the code:

• The model outputs (the raw unnormalised log-density), and we add the learnable scalar self.c to get . This is the reparameterisation from the paper.
• The loss function computes for both data and noise, then uses logsigmoid for numerical stability (equivalent to and ).
• We minimise the negative objective, which is the standard convention in PyTorch (minimise loss = maximise ).
• The noise distribution is a simple standard Gaussian. We can evaluate its log-density exactly via noise_dist.log_prob(), which is all NCE requires from the noise.