Likelihood over Estimation: The Story Behind Stable‑QDA

Today I'm presenting our paper at ICML, and I wanted to write up the ideas the way I'd explain them to a friend over coffee — no jargon walls, just the story of what we found and why I think it matters. If you're stopping by the poster session, this is the extended director's cut.

The problem: bell curves punish the wrong points

Quadratic Discriminant Analysis (QDA) is one of the oldest, simplest classification methods we have. The idea is charming in its directness: for each class — say, "fraud" vs. "legitimate" — assume the data looks like a bell curve (Gaussian). Most points cluster near the center; points far away are unlikely. To classify a new point, ask which class's bell curve makes it most likely, and pick that class. Closed-form training, fast prediction, decision boundaries you can actually inspect.

The problem is that real-world data is often not bell-curve shaped. It has heavy tails — extreme values show up far more often than a Gaussian would predict. Financial transactions, network traffic, faint astrophysical signals: in all of these domains, the tail isn't noise. The tail is often where the interesting things live. The huge transaction, the anomalous packet, the weak pulsar signal.

And here's what goes wrong: the Gaussian likelihood punishes extreme points exponentially harshly. A point that's a perfectly plausible (if rare) occurrence under heavy-tailed reality gets treated by the Gaussian model as essentially impossible. The classifier's judgment gets distorted exactly where correct judgment matters most.

The fix: one line of math

Our answer is Stable-QDA. We replace the Gaussian assumption with α-stable distributions — a family controlled by a tail dial α ∈ (0, 2]. At α = 2 you recover the Gaussian; at α = 1 you get the wild Cauchy distribution; in between, moderately heavy tails. This family isn't an arbitrary choice: by the generalized central limit theorem, α-stable laws are the only possible limits of sums of random variables, which makes them the natural model for heavy-tailed data.

There's an obstacle, though. α-stable distributions have no closed-form density. You can't just write down the likelihood and compute it — which would normally kill the idea.

The escape hatch is realizing that classification never needed the exact likelihood value. It only needs to know which class's likelihood is bigger. Ranking is everything. So we derived an order-equivalent surrogate, and the entire innovation compresses into one comparison:

That log(1 + D) is the whole paper in one expression. A point at Mahalanobis distance 100 isn't punished 100× more than a point at distance 1 — only about 4.6× more. Tail points stay in the game. Everything else about QDA — estimate a center and spread per class, compare scores, pick the winner — stays identical. Same O(Kp²) speed, same interpretability, same calibrated probabilities.

The surprising part: fix the formula, not the statistics

Here's where the paper picks a fight with conventional wisdom, and it's the finding I'm most excited to discuss at the poster.

When statisticians see heavy-tailed data, the reflex is: "Your sample mean and covariance are corrupted by extremes — swap in robust estimators." The spatial median instead of the mean; Tyler's M-estimator instead of the sample covariance. Decades of beautiful theory back these tools.

We found that this reflex often makes things worse — and that the real culprit was the Gaussian formula all along. Two reasons:

First, classification depends on likelihood ratios. Estimation errors that corrupt both classes similarly largely cancel when you compare them. But a wrong likelihood formula — exponential decay where reality decays polynomially — distorts every comparison systematically, and nothing cancels. Estimation error is noise; likelihood misspecification is bias.

Second, Tyler's estimator throws away scale. It captures the shape of a class's spread but normalizes away its overall size. Yet in QDA, size differences between classes are discriminative signal — the log-determinant term uses them directly. In our credit card fraud experiments, the spread difference between fraud and legitimate transactions was enormous (determinant ratio over 10⁷). That difference is the fraud signal. Standard estimators preserved it and cut errors by 53%; Tyler's normalization erased it, catastrophically.

To be fair to the robust statistics tradition — because our result is emphatically not a general critique of it — there is a genuine crossover. At very heavy tails (α < 1.5) with similar class scales, robust estimators win decisively. We map this crossover precisely and package it into a data-driven diagnostic: estimate α with a cheap quantile trick, measure the scale ratio, consult a table, done. We ship the diagnostic script with the paper.

And one result that made deployment genuinely painless: you barely need to know α. A fixed α = 1.5 performs within 1% of the oracle across every regime we tested. The magic was never the precise tail index — it was switching from D to log(1 + D). Stable-QDA has effectively zero hyperparameters.

The theory: Gaussian QDA isn't just worse — it's provably stuck

We prove that Stable-QDA is Bayes consistent: with enough data, its error converges to the theoretical optimum. The more pointed result is the converse. Under genuinely heavy tails, Gaussian QDA's error stays bounded away from the optimum forever — no amount of data rescues it. The failure has two independent sources: the sample covariance never converges when variance is infinite, and — more fundamentally — even with perfect parameters, the exponential-decay formula systematically misclassifies a non-vanishing chunk of tail points. Robust estimators can patch the first wound. Only fixing the likelihood fixes the second. That asymmetry is the theoretical backbone of the title: likelihood over estimation.

We also prove the α-insensitivity result formally: risk varies at most linearly and gently in α, because α enters only as a multiplier on log(1 + D). The functional form is the load-bearing wall; α is just paint.

The evidence: 15–53% error reduction where it counts

On synthetic data where we control the ground truth, the picture is stark: at α = 1.0, Gaussian QDA sits at coin-flip accuracy (~50%) while Stable-QDA reaches ~76%. Under 20% training contamination, Gaussian QDA degrades by up to 17%; Stable-QDA barely moves — the logarithmic likelihood limits any single point's influence, providing robustness for free, even with non-robust estimators.

The diagnostic correctly selected the estimator configuration each time — standard for the first three (moderate tails, big spread differences), robust for Weekly (heavier tails, similar scales). We also report two datasets with near-Gaussian tails where improvements were modest and non-significant, exactly as the theory predicts. I believe negative results belong in papers.

Because standard accuracy is dominated by easy points near class centers, we introduced tail-conditional recall: among the most extreme true positives, how many do you catch? On network traffic data, Stable-QDA catches 99.2% of tail malware versus 89.3% for Gaussian QDA. In fraud and intrusion detection, the tail cases are the expensive ones to miss — the improvement lands precisely where it counts.

Why this matters in the deep learning era

A fair question: who uses QDA in 2026? Here's the connection I find genuinely exciting. Zheng et al. (ICML 2023) proved that when you train a classifier head on frozen pre-trained features — the standard linear-probing workflow with ViT, CLIP, DINOv2 — generative classifiers converge in O(log n) samples versus O(n) for logistic regression. But their result assumes Gaussian features. Meanwhile, Hodgkinson et al. (ICML 2025) showed that pre-trained deep network features are empirically heavy-tailed.

So the fastest-converging classifier family assumes exactly the distribution deep features violate. Stable-QDA fills that gap: a correctly-specified generative head for the features modern models actually produce, with closed-form training and calibrated probabilities. It doesn't compete with deep learning — it rides on top of it.

What's next

The honest boundaries: symmetric stable laws may be suboptimal for heavily skewed data like financial returns, and the p ≫ n regime remains open. Asymmetric extensions and sparse-dispersion interactions are where we're headed next.

Treat heavy tails as structure to model, not noise to suppress — and when your data misbehaves, check your likelihood before you reach for a robust estimator.