We built algorithms to rise above bias. Predictive policing to remove prejudice from patrol routes. Credit scoring to eliminate discrimination in lending. Hiring systems to judge résumés without seeing race or gender.

The promise was simple: code doesn't have prejudice, so code can be fair.

But here's the uncomfortable truth: perfect fairness in AI is mathematically impossible.

Not difficult. Not impractical. Impossible. Proven by theorem. Multiple fairness definitions cannot coexist when the world they measure is unequal. Every AI system must choose which fairness to honor, and that choice is no longer technical. It's moral.

What Fairness Means (And Why It Fractures)

Before we can prove impossibility, we need to define what we're trying to achieve. In algorithmic fairness, there are four major definitions, each intuitive on its own:

Demographic Parity: Equal outcomes across groups. If 10% of Group A gets approved, 10% of Group B should too.

P(Ŷ=1|A=a) = P(Ŷ=1|A=b)

Equalized Odds: Equal error rates across groups. False positives and false negatives should occur at the same rate regardless of group membership.

Ŷ ⊥ A | Y

Predictive Parity: Equal precision across groups. If the algorithm says "yes," it should be equally likely to be correct for both groups.

P(Y=1|Ŷ=1,A=a) = P(Y=1|Ŷ=1,A=b)

Calibration: Scores mean the same thing across groups. A 70% risk score should correspond to a 70% event rate for everyone.

E[Y|S=s,A=a] = s for all groups

Each makes sense in isolation. Each reflects a different moral intuition about what fairness requires. And together, when base rates differ between groups, they collapse into contradiction.

The Theorems That Broke the Dream

Theorem 1: Kleinberg, Mullainathan & Raghavan (2016)

Kleinberg, Mullainathan, and Raghavan proved that calibration and equalized odds cannot both hold when base rates differ between groups.

Here's the intuition: calibration requires that a risk score means the same thing for everyone. A 70% score must correspond to a 70% actual rate of the outcome, regardless of group. But if two groups have different base rates (say, Group A has a 20% default rate and Group B has a 10% default rate), achieving equal error rates (equalized odds) requires setting different decision thresholds for each group. Different thresholds break calibration.

The math is geometric: you're trying to find a single operating point on two different ROC curves that satisfies both conditions. When base rates differ, no such point exists.

Theorem 2: Chouldechova (2017)

Alexandra Chouldechova proved that predictive parity and equalized odds are mutually exclusive when base rates differ.

The proof comes from the definition of positive predictive value (PPV):

PPV = (p × TPR) / (p × TPR + (1-p) × FPR)

where p is the base rate (prevalence), TPR is the true positive rate, and FPR is the false positive rate.

If you hold TPR and FPR constant across groups (equalized odds), but the groups have different base rates (p), then PPV must differ. Conversely, if you force PPV to be equal across groups with different base rates, you must allow TPR and FPR to vary, breaking equalized odds.

The takeaway: Fix precision, break equality. Fix equality, break precision.

Theorem 3: Kearns, Neel, Roth & Wu (2018)

Kearns and colleagues introduced the concept of "fairness gerrymandering": an algorithm can appear fair when measured on broad demographic categories while hiding bias in subgroups.

For example, a hiring algorithm might achieve gender parity overall (50% men, 50% women hired) while systematically discriminating against older Black women. The bias is invisible at the macro level but real at the intersection.

The takeaway: Fairness metrics can hide harm in the folds. Coarse-grained fairness does not guarantee fine-grained justice.

What "Mathematically Impossible" Really Means

These aren't edge cases or statistical noise. They're structural contradictions. When base rates differ between groups (and in the real world, they do), multiple fairness definitions cannot simultaneously hold.

The equations don't betray us. They reveal our contradictions. We want equal outcomes, equal opportunity, and equal accuracy, but the math says: pick two at most, and even that's fragile.

Once the math ends, ethics begins.

Why the Real World Makes This Worse

Unequal Base Rates Are Reality

Crime rates, loan default rates, disease prevalence: all are shaped by centuries of unequal access, opportunity, and treatment. The theorem's precondition (different base rates) is the world's condition.

The impossibility isn't theoretical. It's the ground we stand on.

Data Feedback Loops

Every algorithmic decision changes tomorrow's data. Predictive policing sends more officers to certain neighborhoods, leading to more arrests, which "confirms" the model's accuracy. The algorithm doesn't just predict crime; it produces it.

Fairness today shifts base rates tomorrow, which breaks fairness again. We're not solving bias; we're chasing it as it mutates.

Metric Gaming and Optimization Drift

Optimize one fairness metric, and another degrades. Enforce demographic parity by randomizing decisions? Accuracy drops. Optimize for calibration? Systemic exclusion persists.

This is Goodhart's Law in action: when a measure becomes a target, it ceases to be a good measure.

The Moral Fork: Choosing Your Fairness

The impossibility theorems force a choice. Each fairness metric encodes a different moral philosophy:

Demographic Parity → Equality of Outcome
Distribute benefits equally, regardless of merit or risk.

Equalized Odds → Equality of Opportunity
Ensure errors are distributed fairly, preserving meritocratic ideals.

Predictive Parity / Calibration → Procedural Fairness
Trust the accuracy of predictions; let the chips fall where they may.

These aren't competing algorithms. They're competing visions of justice.

Example 1: Lending

The outcome: A bank that chooses demographic parity faces financial losses and regulatory scrutiny for "unsafe lending." A bank that chooses calibration faces lawsuits for discrimination. A bank that chooses equalized odds splits the difference but satisfies no one fully. Each choice creates winners and losers, and the algorithm simply makes the trade-off explicit.

Example 2: Healthcare Triage

The outcome: A hospital that prioritizes demographic parity faces malpractice suits when preventable cases are missed. A hospital that prioritizes calibration perpetuates historical disparities in care access. A hospital that prioritizes equalized odds still sees unequal health outcomes because base rates differ. The algorithm doesn't create the problem (it inherits it from decades of unequal treatment) but it does force the institution to choose which injustice it can live with.

Each is rational. None is neutral. Every threshold is a value judgment rendered in code.

Who Decides?

Fairness is governance, not gradient descent. The actors are:

Engineers: Measure, test, and surface trade-offs. They can quantify fairness, but they can't decree justice.

Regulators: Define acceptable trade-offs through frameworks like the EU AI Act and NIST AI Risk Management Framework.

Institutions: Embed fairness choices in domain-specific policy. A hospital's fairness priorities differ from a bank's.

Society: Expresses tolerance for risk, bias, and harm through law, activism, and market pressure.

Math can constrain us, but it cannot absolve us.

Beyond the Equations: Governance and Grace

If perfect fairness is impossible, what do we do?

Transparency

Publish which fairness metric you prioritize and why. Include subgroup reports, drift analyses, and threshold audits. Make the trade-offs visible.

Continuous Monitoring

Fairness isn't a one-time achievement. It's a discipline. Build dashboards that track fairness metrics over time and alert when base rates shift.

Governance Frameworks

Adopt structured approaches like the NIST AI RMF (Govern, Map, Measure, Manage) or the EU AI Act's documentation requirements. Create internal "Fairness Boards" with cross-disciplinary representation.

Fairness as Stewardship

The goal isn't perfection. It's accountable imperfection. Acknowledge bias, document trade-offs, monitor outcomes, and adjust when the world changes.

The Mirror and the Menu

We asked algorithms to make the world fairer. The math told us: you can't have it all. Choose.

Algorithms don't solve morality. They expose it. The impossibility theorems aren't barriers to fairness; they're blueprints for humility.

---

Further Reading

• Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores
• Chouldechova, A. (2017). Fair Prediction with Disparate Impact
• Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2018). Preventing Fairness Gerrymandering
• NIST AI Risk Management Framework
• EU Artificial Intelligence Act