Auditing AI Applications for Security and Reliability Risks

Hi! This course starts with an application you already know how to build, then makes its behavior more dependable and its deployment easier to own. In this first module, we’ll harden a vibe-coded AI application one boundary at a time.

Today’s deliverable is an audit note, not a rewrite. You’ll trace one model-powered feature and record evidence of four risks: hidden model coupling, unsafe secret handling, missing validation, and unbounded model calls. An ecommerce offer assistant will serve as the example; you can apply the same method to an analytics assistant or one of your existing applications. Allow roughly 40 minutes, including the short resource selections and a review of your own code.

Start with the feature, not the framework

Suppose an assistant suggests a customer discount and saves it. A quickly assembled version might resemble this:

# Illustrative code only: vendor_sdk is not a real package.
from vendor_sdk import Client

def propose_offer(customer_id: str, notes: str, db):
    client = Client(api_key="PASTED_KEY_HERE")
    prompt = f"Suggest a discount percent for {customer_id}: {notes}"

    while True:
        reply = client.generate(model="premium-model-v1", prompt=prompt)
        if reply.text:
            break

    print(prompt, reply.text)
    discount = int(reply.text)
    db.execute(
        f"UPDATE offers SET discount_pct = {discount} "
        f"WHERE customer_id = '{customer_id}'"
    )
    return discount

The key string is a placeholder, not a credential. The feature’s intended behavior is straightforward, but its implementation joins several decisions that deserve separate scrutiny. The function chooses a provider and model, handles a credential, accepts customer notes, decides when to call the model again, interprets its response, and writes to a database.

Audit by following the feature’s trust boundaries. Start with the user-facing entry point. Find where inputs are assembled into a prompt, where the model request occurs, where the response is interpreted, and what happens afterward. At each boundary, record what the code actually checks—not what a prompt asks the model to do.

For this example, a useful first-pass note would look like this:

This is an evidence-based audit, not a claim that every possible incident has occurred. For example, the placeholder does not prove a real key was committed. It tells you where to check the deployed configuration and repository history.

Find where the model is entangled with the application

Hidden model coupling means provider-specific details are embedded in code that should express the business task. Here, propose_offer knows the vendor client, a model identifier, the method generate, and the response field reply.text. If a different provider returns a list of message objects instead of a .text field, editing the client construction alone will not be enough.

The audit question is practical: How many places must change to replace the model or substitute a controlled response in a test? Search beyond the obvious API call. Provider assumptions can hide in prompt formatting, response parsing, exception handling, configuration defaults, and tests.

A small design boundary can prevent that spread: the offer function asks for a discount recommendation through a narrow interface, while separate code handles the provider’s API. You do not need an enterprise gateway or multiple microservices to benefit from this separation. The short video below demonstrates the underlying technique with storage rather than an LLM: notice that the caller no longer needs to change when the implementation is replaced.

AWS Prescriptive Guidance describes a larger-scale version of the same boundary. Read it for the reason to separate provider details, not as a recommendation to deploy a gateway for this first audit.

For the audit, keep coupling and secrets as separate findings. A configurable model name does not make the application provider-neutral if parsing still assumes one vendor’s response shape. Likewise, moving a key from a string literal to an environment variable avoids committing it in source but does not, by itself, control who can read it in a deployed environment.

Trace the key’s entire route: where it is stored, who or what can access it, how the application receives it, and whether it can appear in prompts, logs, exceptions, screenshots, or test fixtures. A credential in a system prompt is still exposed to the model; a credential removed from the current file may still exist in Git history. If you discover a real exposed key, don’t paste it into an audit note: treat it as compromised and arrange rotation or revocation.

Treat model output as input to the next system

The model’s response crosses another trust boundary when application code uses it. In the example, int(reply.text) checks only whether Python can parse an integer. It does not establish that the discount is within policy, that the recommendation belongs to the right customer, or that this caller is authorized to change an offer. If parsing fails, the code also has no deliberate failure path.

The database statement adds a separate problem: it interpolates customer_id into SQL. That input should be handled with a parameterized query appropriate to the database driver. Even a well-formed model response would not make unsafe SQL construction acceptable. And if changing an offer has a meaningful business impact, the workflow may need approval rather than an automatic write.

OWASP calls insufficient scrutiny of model-generated material before downstream use improper output handling. Its guidance is useful here because the relevant control is in application code, not merely in a better prompt.

For the offer feature, an audit finding about validation should name both sides of the model call. Before submission, are customer notes bounded in size and screened for information that should not be sent to this provider? After the response, does code require the expected type and an allowed discount range before any write? Are authorization and approval enforced by the application or downstream service rather than delegated to the model? You are locating these checks today; later lessons will implement typed outputs and explicit validation.

Put a ceiling on every route to inference

The while True loop is the clearest call-budget defect. A provider could repeatedly return empty text, so a single user request might cause an indefinite series of model calls. Even without that loop, large inputs, unrestricted output length, automatic retries, many concurrent requests, or a publicly accessible endpoint can make usage expensive or degrade service. A local model avoids per-token API charges but still consumes compute and capacity.

During the audit, look for limits at distinct levels:

• One attempt: input-size limit, requested output limit, and provider-call timeout.
• One user request: maximum attempts or total elapsed time, including retries and tool-triggered calls.
• Across users: authentication, rate limits or quotas, and visibility into usage.

A provider’s own quota is not a substitute for an application’s per-request stopping rule. Conversely, an attempt limit does not prevent a single user from starting thousands of requests. Record which boundary is missing rather than writing only “add rate limiting.” This module will later address provider timeouts; production-traffic lessons will add more detailed request and token controls.

Make the audit note on your application

Set aside about 10 minutes for a static review of one feature. Follow its entry point through the model call to its downstream action. Your editor’s search is sufficient; if you use rg, its -l option lists filenames without printing potentially sensitive matching lines:

# Adjust the directories to match your repository.
rg -l -i 'api_key|secret|generate\(|invoke\(|while |retry|execute\(' app tests

Inspect matches locally. Do not copy real keys, customer data, or unredacted prompts into notes or an AI chat. For each of the four risks, record a file/function, a short description of the evidence, the plausible impact, and one next change. Mark a finding “not verified” if you cannot inspect the deployed configuration or provider settings. The result should be a compact artifact a teammate could use to prioritize fixes—not a speculative list of everything that might go wrong.

What you should leave with

A useful AI-app audit follows data and decisions across boundaries: business logic to provider, configuration to credential use, model response to downstream action, and user request to resource consumption. The strongest findings cite observable evidence, distinguish confirmed problems from unknowns, and propose a specific control at the boundary where it belongs.

Keep your audit note. In the next lesson, you’ll address one of its foundations by creating a reproducible Python environment with locked dependencies and externalized configuration.