Introduction
In our last lesson, we explored the phenomenon of 'hallucination' in LLMs, an inherent vulnerability where models confidently generate incorrect information due to their fundamental design. We saw how this poses serious risks to reliability, particularly in the legal field where factual accuracy is paramount.
Today, we broaden our focus from inherent flaws to other critical system vulnerabilities. This lesson addresses the learning outcome: Evaluate AI system vulnerabilities, including 'adversarial attacks' and 'catastrophic forgetting'.
We will cover two distinct but equally important types of weaknesses:
- Adversarial Attacks: These are external, malicious efforts to intentionally deceive or manipulate an AI system. We will explore how attackers can poison training data, fool a trained model with cleverly crafted inputs, or even steal the model itself.
- Catastrophic Forgetting: This is an internal, dynamic vulnerability where a model, upon learning a new task, suffers a drastic and sudden performance degradation on previously learned tasks.
Understanding these vulnerabilities is fundamental to AI governance. For anyone aspiring to work at the intersection of law and technology, knowing how AI systems can be broken—either by malicious actors or by their own learning processes—is essential for drafting robust contracts, assigning liability for AI-induced harm, and designing effective regulatory frameworks.
1. Adversarial Attacks: Weaponizing AI's Weaknesses
While hallucinations are an unintended byproduct of an AI's design, adversarial attacks are deliberate exploits of those same design principles. An adversary can use their knowledge of how machine learning works to make a system fail in a specific, desired way.
Let's start with a clear overview of what this looks like in practice and the main categories of these attacks.
Overview of Adversarial Machine Learning
This video from Carnegie Mellon University's Software Engineering Institute provides an excellent introduction to adversarial machine learning. It opens with a compelling example and then lays out a useful taxonomy of attacks.
Please watch the following segments: 00:00 - 00:36: A quick introduction using the example of a self-driving car misreading a stop sign. 01:19 - 02:04: This section introduces the core idea that an adversary can influence a system by manipulating the model or its data, forcing it to 'learn the wrong thing, do the wrong thing, or reveal the wrong thing'. 02:04 - 05:18: This is the core of the video for our purposes. It explains the three major classes of attacks: Poisoning attacks (corrupting training data). Evasion attacks (fooling a trained model, e.g., the panda-gibbon example). Information revealing attacks like model extraction and model inversion (stealing the model or its data).
The video gives us a clear technical framework. Let's formalize this taxonomy:
- Poisoning Attacks (Learn the Wrong Thing): These attacks target the training phase. An adversary introduces corrupted or malicious data into the training set. The model then learns incorrect patterns or even a "backdoor," causing it to misbehave on specific inputs later on. This is analogous to contaminating the evidence used in a legal case before the trial even begins.
- Evasion Attacks (Do the Wrong Thing): These attacks target the inference (or deployment) phase. The model is already trained and functions correctly on normal data. The adversary crafts a special input—an "adversarial example"—that is often imperceptible to humans but is designed to cause the model to make a specific error. The panda classified as a gibbon is the classic example.
- Extraction/Inversion Attacks (Reveal the Wrong Thing): These attacks target the confidentiality of the model or its data. Through clever queries to a model's API, an adversary can reconstruct the model itself (model extraction) or recover sensitive information from its training data (model inversion), such as faces or medical records.
Legal Implications of Adversarial Attacks
These technical vulnerabilities immediately raise legal questions. If an AI system is compromised by such an attack and causes harm, who is responsible? Does existing law provide any recourse?
Your work in consulting on digital transformation projects involved implementing systems like CRMs and chatbots. Imagine a chatbot for a healthcare clinic being poisoned to give out dangerous advice, or a competitor stealing your client's proprietary patient triage model via model extraction. The legal fallout would be immense. Let's explore how current legal frameworks might address this.
Law and Adversarial Machine Learning
The paper 'Law and Adversarial Machine Learning' by Salome Viljoen begins to map these technical attacks onto existing US legal doctrines. It's written to be accessible but provides a solid foundation for legal analysis.
Please read the following three sections: Section 2 (Cybersecurity law and... attacks): Focus on how poisoning and evasion (perturbation) attacks might be prosecuted under the Computer Fraud and Abuse Act (CFAA), the primary US 'anti-hacking' law. Note the discussion on whether a manipulated input constitutes 'code' that causes 'damage'. Section 3 (Copyright law and Model Inversion, Model Extraction attacks): This section explains why intellectual property laws like copyright may offer less protection against model theft than one might assume. Section 4 (Liability laws in the context of adversarial ML): This addresses the difficult question of liability in tort law when a product fails due to an adversarial attack. Pay attention to the challenge of establishing a 'standard of care' for ML development.
As the paper highlights, the law is still catching up. While a classic "man-in-the-middle" supply chain attack fits neatly into the CFAA, the legal status of a subtly modified image (an evasion attack) is less clear. Similarly, the fact that copyright protects the expression of code but not its function creates a significant gap that model extraction attacks exploit. This uncertainty is a core challenge for AI governance.
2. Catastrophic Forgetting: The Perils of Continuous Learning
We now shift from external, malicious attacks to a fundamental vulnerability in how AI models learn over time. Humans can learn new things incrementally without completely forgetting old skills. Most neural networks cannot. This phenomenon is known as catastrophic forgetting or catastrophic interference.
Continual Learning and Catastrophic Forgetting
This lecture on 'Continual Learning and Catastrophic Forgetting' provides a comprehensive introduction to the problem. It explains why simply training a model on new data causes it to forget old data and frames the core challenge as the 'plasticity-stability dilemma'.
Please watch the first part of the video (00:00 - 11:57). Focus on understanding: The real-world motivation for 'continual learning' (e.g., updating a drone's capabilities). The mathematical formulation of the problem: why training on new data (Task B) degrades performance on old data (Task A). The limitations of naive solutions like retraining from scratch or replaying all old data. The core concept of the plasticity-stability dilemma: the need to be flexible enough to learn new things (plasticity) while remaining stable enough not to forget old things (stability).
The plasticity-stability dilemma is a central concept in this field. A model that is too "stable" cannot adapt to new information, becoming outdated. A model that is too "plastic" is susceptible to catastrophic forgetting, erasing valuable prior knowledge.
Mitigating Catastrophic Forgetting
Researchers have developed various strategies to address this dilemma. These generally fall into three categories:
- Regularization-based methods: These methods add a penalty to the loss function to discourage large changes to network weights that were important for previous tasks.
- Architectural methods: These methods dynamically change the network's structure, for example by allocating new neurons for new tasks while freezing the old ones.
- Replay-based methods: These methods store a small subset of old data (or use a generative model to create synthetic old data) and "replay" it during training on the new task.
Let's take a brief technical dive into one of the most well-known regularization methods to satisfy your interest in the underlying mechanics.
Continual Learning and Catastrophic Forgetting
Continuing with the same video, this next segment explains 'Elastic Weight Consolidation' (EWC), a classic algorithm for mitigating catastrophic forgetting. It also shows a powerful graph demonstrating its effectiveness.
Watch the segment from 24:34 to 31:47. You don't need to memorize the equations, but focus on grasping the conceptual points: The core idea of EWC: identify which neural network weights were most important for the old task and penalize changes to them more heavily. The purpose of the Fisher Information matrix (Fi) as a way to quantify 'importance'. Observe the final graph comparing plain Stochastic Gradient Descent (SGD) with EWC. See how with SGD, performance on Task A plummets when the model starts training on Task B, while with EWC, it remains high.
Legal and Regulatory Implications of "Unlocked" AI
Catastrophic forgetting isn't just a technical curiosity; it has profound implications for regulation, especially in high-stakes domains like healthcare. An AI medical device that continuously learns from new patient data could improve over time. However, it could also "forget" how to correctly identify a rare disease it was originally trained on. This creates a massive challenge for regulators.
An Overview of Continuous Learning Artificial Intelligence
This report from the Canadian Journal of Health Technologies provides an overview of the regulatory challenges posed by 'continuous learning' or 'unlocked' AI.
Please read the introduction and the 'Regulatory Status' section. In the introduction, note how 'catastrophic forgetting' is explicitly identified as a key risk of continuous learning AI, alongside model drift. In the 'Regulatory Status' section, pay attention to the concepts being developed to manage this risk, particularly the US FDA's proposed 'Predetermined Change Control Plan'. This is a concrete example of a legal/regulatory mechanism designed to govern AI that changes post-deployment.
The idea of a "Predetermined Change Control Plan" is a crucial governance concept. It represents a shift from approving a static product to approving a process for change. The manufacturer must specify in advance what kinds of modifications the AI is expected to make, how it will be monitored, and what guardrails are in place to manage risks like catastrophic forgetting. This is a practical legal response to a deep technical vulnerability.
Test your understanding!
A company develops an AI-powered security camera system for a large retail chain. The system is trained to identify various types of theft. After a few months, the company issues a software update to improve the detection of a new, sophisticated shoplifting technique. Following the update, the system successfully detects the new technique but now fails to identify a more common, older form of theft, leading to significant losses for the retailer.
- Which of the vulnerabilities discussed in this lesson is most likely the root cause of the failure?
- The retailer sues the AI company. From a legal perspective, why is this situation more complex than a simple software bug? Consider the concepts of 'standard of care' and regulatory approval.
Show answer
-
The most likely root cause is catastrophic forgetting. The model was updated to learn a new task (detecting the new shoplifting technique) and, in the process, its performance on a previously learned task (detecting the common theft) degraded catastrophically. It's a classic example of the plasticity-stability dilemma, where plasticity (learning the new technique) came at the cost of stability (retaining old knowledge).
-
This situation is legally complex for several reasons:
- Standard of Care: Unlike a simple bug, which might clearly violate established software engineering practices, there is no universally agreed-upon "standard of care" for preventing catastrophic forgetting. The AI company could argue they used state-of-the-art continual learning techniques, but the vulnerability still manifested. A court would struggle to determine if the company was negligent without industry benchmarks.
- The Nature of the "Defect": The AI company didn't necessarily introduce a flaw; they tried to improve the product. The "defect" is an emergent property of the learning process itself. This complicates product liability claims, which often rely on identifying a specific design or manufacturing defect.
- Regulatory Framework: If this were a regulated product (like a medical device), the key question would be whether the company followed an approved Predetermined Change Control Plan. Did they anticipate this risk? Did their post-update monitoring and validation processes fail? The liability might hinge on compliance with the approved process of change, rather than the specific outcome.
Conclusion
In this lesson, we have moved beyond inherent model flaws to evaluate specific, high-impact vulnerabilities. We have seen how AI systems can be actively manipulated by adversaries and how their own learning processes can lead to unexpected and severe failures.
Key Takeaways:
- Adversarial Attacks are intentional exploits that can be categorized into poisoning (corrupting training), evasion (fooling inference), and extraction (stealing the model/data). Existing legal frameworks like the CFAA and copyright law provide an incomplete and uncertain patchwork of protection.
- Catastrophic Forgetting is a fundamental vulnerability of AI systems that learn sequentially, where acquiring new knowledge overwrites or destroys old knowledge. It stems from the plasticity-stability dilemma.
- Both vulnerabilities have direct legal consequences, complicating the assignment of liability and requiring new regulatory paradigms. Concepts like a "standard of care" for AI security and "Predetermined Change Control Plans" for adaptive AI are emerging as crucial governance tools.
Preview of the Next Lesson:
We have now examined several key failure modes of AI systems: biased data, opaque models, hallucinations, and specific vulnerabilities like adversarial attacks. Our next and final lesson in this module, "Analyze how choices in data collection, model selection, and deployment create or mitigate legal risks," will tie all these concepts together. We will adopt a holistic, lifecycle perspective to understand how every technical decision made during an AI project's development is also a legal risk management decision.