Algorithmic Bias and Anti-Discrimination Law

Introduction

In our previous lesson, we examined how AI can create market-level economic harms, focusing on algorithmic collusion and data-driven dominance under competition law. We discussed how autonomous systems could distort markets in ways that challenge traditional legal definitions of "agreement" and "abuse."

Today, we shift our focus from the market to the individual. This lesson addresses the learning outcome: Apply anti-discrimination laws to cases of algorithmic bias in employment and service provision. We will investigate how the same data-driven processes that optimize markets can also encode and amplify societal biases, leading to discriminatory outcomes against individuals. You will learn to connect the technical realities of algorithmic bias to the constitutional protections against discrimination, with a specific focus on the Indian legal landscape.

Our journey will cover:

The technical and social origins of algorithmic bias.
The application of India's constitutional guarantee of equality (Article 14) to AI systems.
The complex and often conflicting technical definitions of "fairness."
A framework for analyzing and arguing a case of algorithmic discrimination in court.

1. The Anatomy of Algorithmic Bias

Algorithmic bias is not merely a technical error or a bug in the code. It is a complex, socio-technical phenomenon where computational systems consistently and systematically produce unfair outcomes that disadvantage individuals based on their membership in a particular social group.

To grasp the deep-seated nature of this problem, especially within the Indian context, the paper "Algorithmic Bias and Substantive Equality" provides an essential foundation.

Algorithmic Bias and Substantive Equality.docx

This paper is central to our lesson. It explains how algorithmic bias emerges and why it poses a constitutional challenge in India, offering concrete local examples.

Please read Part I — Algorithmic Bias: Concept and Indian Manifestations. Focus on the 'three core mechanisms' of how bias emerges and the specific Indian examples provided for education, finance, and justice.

As the paper details, bias creeps into AI systems through several key pathways:

Biased Training Data: AI models learn from historical data. If this data reflects past discrimination, the model will learn to reproduce it. For example, if a loan approval dataset reflects a historical bias against women or certain castes, an AI trained on it will learn to associate these groups with higher risk, regardless of individual merit.
Proxy Variables: Algorithms often use seemingly neutral data points as "proxies" for protected characteristics. For instance, an AI hiring tool might not use "caste" as a variable, but it might use an applicant's postal code. If certain postal codes are highly correlated with specific castes or socio-economic classes, the algorithm can inadvertently discriminate based on this proxy. Given your work with CRM systems and KPI tracking, you can see how easily such proxies could be integrated without any discriminatory intent.
Feedback Loops: Bias can become self-reinforcing. An algorithm that flags individuals from a certain neighborhood for higher police scrutiny will generate more arrest data in that area. A future version of the algorithm, trained on this new data, will see the higher arrest rate as confirmation of higher criminality, leading to even more scrutiny. This creates a vicious cycle.

These are not abstract risks. The paper highlights real-world Indian examples, from biased facial recognition for student attendance in Telangana to exclusionary credit scoring models that penalize rural users.

2. The Constitutional Shield: Article 14 and Substantive Equality

When an algorithm denies someone a job or a loan based on their group identity, it implicates fundamental rights. In India, the primary constitutional shield against such discrimination is Article 14, which guarantees "equality before the law" and "equal protection of the laws."

However, applying this 70-year-old provision to 21st-century technology requires an evolution in legal thinking. The paper you're reading brilliantly maps out this necessary evolution.

Algorithmic Bias and Substantive Equality.docx

Let's continue with the same paper to understand how legal doctrine must adapt to confront algorithmic bias.

Now, please read Part II — Rethinking Article 14 in the AI Era. Pay close attention to why the 'classical doctrines' falter and how the concept of 'substantive equality' provides a more robust framework.

Here's a breakdown of the key legal arguments:

The Limits of Classical Doctrine: Traditionally, Article 14 was tested against two standards:
- Reasonable Classification: Is there a rational basis for treating groups differently? This fails because algorithms often don't use explicit classifications; they use proxies.
- Non-Arbitrariness: Is the state action fair and not capricious? This is difficult to apply when the decision-making logic is hidden within an opaque "black box" model.
The Rise of Substantive Equality: The Indian Supreme Court has moved beyond a purely formal view of equality. In landmark cases like Navtej Singh Johar (decriminalizing homosexuality), the court embraced substantive equality. This doctrine is not just about treating everyone the same; it's about actively dismantling structures of systemic disadvantage and ensuring equality in outcomes, not just opportunities.
Towards an "Algorithmic Equality" Doctrine: To apply this to AI, the paper proposes a new interpretation of Article 14 that would:
- Recognize Proxy Discrimination: Treat discrimination via proxies (like postal code) as equivalent to direct discrimination.
- Equate Opacity with Arbitrariness: If a state-endorsed system cannot explain its decisions, that opacity itself constitutes arbitrariness and violates Article 14.
- Mandate Structural Fairness: Evaluate AI systems based on their impact on disadvantaged groups, not just their supposed neutrality.

This doctrinal shift moves the legal question from "Did the developer have discriminatory intent?" to "Does the system's outcome perpetuate structural inequality?"

3. The Engineer's Dilemma: Measuring "Fairness"

To comply with the law, a developer can't just tell an algorithm to "be fair." They must translate the abstract legal principle of equality into a precise, mathematical objective for the model to optimize. This is where the challenge intensifies, as "fairness" has many conflicting technical definitions.

Let's dive into the technical paper "Algorithmic Fairness: Choices, Assumptions, and Definitions" to understand these concepts. This will give you the technical vocabulary needed to critically analyze claims of algorithmic fairness.

Algorithmic Fairness: Choices, Assumptions, and Definitions

This paper provides the technical 'deep dive' you're interested in. It catalogues the various mathematical definitions of fairness and explains why they often conflict. We'll walk through the key ideas.

This is a dense academic paper. Focus your reading on the concepts, not every mathematical detail. Skim Section 2.3 ('Axes of Fairness and Protected Groups') to understand the concepts of protected characteristics and intersectionality. Read Section 4 ('Flavors of Fairness Definitions from Data Alone'). Focus on understanding the intuition behind the different definitions presented in Figure 2 and the main sections: 4.2.1 (Conditional on outcome/decision): This explains metrics like equal false positive rates. 4.3 (Equal Decision Measures): This explains demographic parity. Read Section 4.4 ('Impossibilities'), especially the part about the COMPAS debate. This is crucial for understanding why these metrics are in conflict.

Let's simplify the main "flavors" of fairness:

Fairness Metric	Intuitive Question	Example in Hiring
Demographic Parity	Do different groups receive the positive outcome at equal rates?	The percentage of male applicants hired is the same as the percentage of female applicants hired.
Equal Opportunity	Among all qualified candidates, do different groups have an equal chance of being hired?	Of all candidates (male and female) who would be successful on the job, the algorithm correctly identifies an equal proportion from each group.
Predictive Parity	Among the candidates the algorithm recommends, do they succeed at equal rates regardless of group?	Of the men recommended by the AI and hired, 80% succeed. Of the women recommended by the AI and hired, 80% also succeed.

Fairness Tree for AI Interventions — This "Fairness Tree" provides a helpful mental model for how a practitioner might choose a fairness metric based on the context of the AI system. For example, for a punitive action (like denying a benefit), one might prioritize minimizing false positives (FPR Parity), whereas for an assistive action (like granting a scholarship), one might focus on maximizing true positives (Recall/FNR Parity).

The Impossibility Problem

Here is the critical insight from the technical paper: it is mathematically impossible to satisfy all these fairness metrics at the same time if the underlying base rates of the outcome differ between groups.

The famous COMPAS debate illustrates this perfectly. COMPAS is an algorithm used in the US to predict a defendant's risk of re-offending.

ProPublica's analysis found it was biased because it had a much higher false positive rate for Black defendants (it was more likely to wrongly flag a non-recidivating Black defendant as "high-risk" than a white one). They argued for Equalized Odds.
The algorithm's creator, Northpointe, defended it by showing it had predictive parity: among defendants flagged as "high-risk," the actual rate of re-offending was roughly equal for Black and white defendants.

Both were correct. You cannot have both equal false positive rates and equal predictive parity when the overall recidivism rates differ between the two groups. This means the choice of a "fairness" metric is not a neutral, technical decision. It is a normative choice with profound legal and ethical implications.

4. Putting it all Together: Litigating Algorithmic Discrimination

Imagine you are a lawyer in a case where an AI system used by a large bank for approving business loans has rejected your client, a female entrepreneur. You suspect algorithmic bias. How would you apply what we've learned?

Your Argument (as the Plaintiff):

Establish Disparate Impact: First, you would use discovery to get data on the algorithm's decisions. You would show that the loan approval rate for female applicants is significantly lower than for male applicants. This demonstrates a failure to meet Demographic Parity.
Invoke Substantive Equality (Article 14): You would argue that even if the algorithm is facially neutral (it doesn't have a gender == 'female' rule), its outcome perpetuates systemic disadvantage for women in business, violating the principle of substantive equality.
Challenge Proxies and Opacity: You'd argue that the model likely uses discriminatory proxies (e.g., analyzing past borrowing from informal networks, which might be more common for certain groups). You would also argue that if the bank cannot provide a clear, justifiable reason for the rejection, its opacity is a form of arbitrariness, violating Article 14.

The Bank's Defense (as the Defendant):

Claim a Different Fairness: The bank's data scientists would argue their model is fair according to a different metric. They might show it has Predictive Parity—i.e., among the men and women who are given loans, the default rate is the same.
Argue Business Necessity: They will claim the model is more accurate at predicting default than human loan officers and is therefore essential for their business. They may even argue it is less biased than the humans it replaced.

The Court's Task:

The court cannot simply accept the bank's technical defense. Guided by the principle of substantive equality, the judge must ask: Does the bank's chosen definition of fairness (Predictive Parity) have the effect of locking out a protected group from economic opportunity, thereby reinforcing historical inequality? The legal obligation under Article 14 may require the bank to choose a different fairness metric (like Equal Opportunity) or implement other corrective measures, even if it leads to a marginal loss in predictive accuracy.

This is where law guides technology. The choice of fairness metric becomes a matter of constitutional interpretation, not just statistical optimization. The EU's AI Act, for instance, takes a proactive approach by classifying AI in employment as "high-risk" and mandating human oversight and fundamental rights impact assessments before deployment, aiming to prevent such harms from occurring in the first place.

Test your understanding!

A prestigious university uses an AI model to screen applicants for its engineering program. The model is trained on 20 years of alumni data, which shows that students who attended elite urban high schools were most likely to graduate with top honors. Consequently, the AI disproportionately rejects applicants from rural and semi-urban schools.

The university defends the model, providing data that it has excellent Predictive Parity: students admitted by the AI, whether from urban or rural schools, graduate at nearly identical rates. A public interest litigation is filed, arguing the model is unconstitutional under Article 14.

What fairness metric is the university's model violating?
What is the core argument the petitioners would make under the doctrine of substantive equality?
How does the university's defense (claiming Predictive Parity) fall short when viewed through the lens of substantive equality?

Show answer

Violated Metric: The model is violating Demographic Parity because it is not admitting students from rural and urban backgrounds at comparable rates. It is also likely violating Equal Opportunity, as it is systematically failing to identify qualified candidates from rural backgrounds simply because their profiles don't match the historical data from elite schools.
Substantive Equality Argument: The petitioners would argue that the AI model, by relying on historical data reflecting pre-existing societal inequalities (unequal access to elite high schools), is perpetuating a structure of disadvantage. It is denying educational opportunities to potentially brilliant students from non-traditional backgrounds. Under substantive equality, the state (or a state-like institution) has a duty to dismantle such barriers, not automate and entrench them.
Shortfall of the Defense: The university's claim of Predictive Parity is a narrow, technical defense that ignores the broader social context. It essentially says, "Our model is good at picking the kind of people who have always succeeded here." From a substantive equality perspective, this is precisely the problem. The model is simply reinforcing the status quo and failing in its duty to provide equal opportunity and correct for historical imbalances. The court would likely find that an admissions process that systematically excludes entire classes of people, even if "accurate" in a narrow sense, is unfair and arbitrary under Article 14.

Conclusion

Today we have connected the technical world of machine learning with the foundational principles of constitutional law. You've seen that tackling algorithmic discrimination is a multi-disciplinary challenge that demands both technical literacy and legal creativity.

Key Takeaways:

Algorithmic Bias is Systemic: It arises from biased data, discriminatory proxies, and feedback loops, reflecting and amplifying societal inequalities.
Article 14 is a Powerful Tool: The doctrine of "substantive equality" allows Indian courts to look beyond formal neutrality and assess the real-world impact of AI systems on disadvantaged groups.
"Fairness" is Not One-Size-Fits-All: There are multiple, mathematically conflicting definitions of fairness. The choice of which metric to use is a normative, policy-driven decision that the law must guide.
Litigating Bias: A legal challenge to algorithmic bias involves demonstrating disparate impact and arguing that the system, despite any claims of technical "accuracy" or "fairness," perpetuates structural discrimination in violation of constitutional principles.

Preview of the Next Module:

Having examined AI through the lenses of Indian competition and anti-discrimination law, we will now broaden our perspective. In our next module, we begin a comparative analysis of global AI governance frameworks, starting with the world's most comprehensive piece of AI legislation: the EU AI Act. We will contrast its precautionary, risk-based approach with other global philosophies and analyze its potential impact on India.

Recommended resources