Building & Shipping AI Products

Set up API keys and authenticate with the OpenAI and Anthropic APIs.

Send a basic chat completion request and parse the response programmatically in JavaScript.

Structure a multi-turn conversation using the messages array (system, user, assistant roles).

Use streaming to display LLM responses incrementally in a simple web interface.

Implement basic error handling for API calls using try-catch to manage network or server failures.

Implement a retry mechanism with exponential backoff to handle transient errors and rate limits.

Apply a structured framework ('Who is the user?', 'What is their goal?', 'What slows them down?') to identify user friction points.

Identify a specific friction point in your family's event business and brainstorm how an LLM could help solve it.

Score a user pain point by rating its severity, frequency, and potential impact to prioritize AI solutions.

Classify a set of example problems as well-suited or poorly suited for LLMs by testing each one hands-on.

Identify an existing workflow in a business that could be augmented or automated with AI.

Define the core user problem and success metrics for a simple AI tool idea.

Identify potential risks (e.g., hallucinations, bad advice) and define simple input/output guardrails for the tool.

Evaluate a proposed AI feature by building a quick prototype prompt and testing its feasibility.

Write clear system prompts that define the AI's role, tone, and constraints.

Apply the few-shot prompting technique by providing example input-output pairs to steer output format.

Use delimiters and structured formatting to reduce ambiguity in prompts.

Apply chain-of-thought prompting to improve reasoning accuracy on multi-step tasks.

Iterate on a prompt systematically by identifying failure modes and adjusting instructions.

Build a reusable prompt template for a specific use case (e.g., summarization, extraction, classification).

Explain how RAG works at a high level and why injecting external context improves response accuracy.

Implement a strategy to split a document into smaller, semantically relevant chunks.

Generate embeddings for a list of text chunks using an embedding model API.

Set up a simple vector store and add document embeddings to it.

Perform a similarity search against the vector store to retrieve context relevant to a user query.

Inject retrieved context into a prompt to generate a grounded response.

Create a small test set and compare RAG responses vs. plain prompt responses side by side.

Debug a common RAG failure mode, such as irrelevant context retrieval or contradictory answers.

Sketch where an AI feature fits in a product's UI by analyzing its prioritization, placement, and prominence.

Implement a basic AI interaction UI with clear input and output components.

Add a 'Special Instructions' field that lets users guide the AI's behavior.

Incorporate a feedback loop mechanism (e.g., thumbs up/down buttons) to capture user ratings of AI output.

Build a human-in-the-loop workflow where users review and edit AI outputs before they are finalized.

Classify AI features by autonomy level (suggestion vs. full automation) based on risk scenarios.

Implement a simple reporting mechanism that lets users flag bad AI outputs.

Define the scope of a minimal AI prototype, including the target user, core features, and user journey.

Create the HTML structure and basic CSS for a chat-style interface with an input field, send button, and message area.

Write JavaScript to capture user messages, display them in the chat, and show a mock AI response.

Explain why calling an LLM API directly from a public frontend is insecure and why a backend proxy is needed.

Initialize a basic Node.js project and install necessary packages like Express.js.

Set up a simple Node.js server with an Express route to act as a backend endpoint.

Modify the backend endpoint to receive a frontend request, call the LLM API, and return the response.

Update the frontend JavaScript to send requests to the backend endpoint and display real AI responses.

Explain why manual 'vibe checks' fail at scale and the need for systematic evaluation.

Define specific quality metrics (e.g., correctness, relevance, tone) for your AI system's outputs.

Create a golden dataset of at least 10 input/expected-output pairs scoped to a specific task.

Write a script to run your AI system on the golden dataset and save the generated outputs.

Design a grader prompt that instructs an LLM to compare a generated output against an expected output and return a structured score.

Automate the full evaluation loop across the golden dataset and calculate an average quality score.

Test for prompt injection vulnerabilities by attempting to override the system's instructions.

Run adversarial tests to trigger hallucinations and check for PII leaks.

Build a minimal agent and compare the result to a single LLM call to understand why multi-step tool use matters.

Map the components of your agent to the Sense-Plan-Act framework to understand each part's role.

Define a tool schema for an LLM API call and parse the model's decision to invoke it.

Write the application logic to execute a tool call requested by the LLM and return the result for a final response.

Build a multi-step agent that chains two actions in sequence (e.g., search the web, then summarize the results).

Store the history of an agent's tool calls and append it to the prompt to give the agent memory.

Implement termination rules to detect and prevent task looping in an agent.

Add error handling and fallback behavior when an agent's tool call fails.

Investigate the key trade-offs (cost, latency, quality, hosting) between proprietary models (OpenAI, Anthropic) and open-source models (Llama, Mistral).

Estimate the cost of an AI feature based on token usage, request volume, and model pricing.

Measure time-to-first-token (TTFT) and total latency for a basic API call.

Optimize perceived latency by implementing streaming and measuring the improvement in TTFT.

Apply prompt compression techniques to reduce input token count for a given use case.

Implement a basic caching mechanism to avoid redundant API calls for identical prompts.

Implement a classifier function that labels incoming queries as 'simple' or 'complex' based on heuristics.

Implement a router function that sends simple queries to a cheaper model and complex queries to a more powerful model.

Select a real problem and target user for your capstone project using the AI opportunity methods learned previously.

Define the core user journey and draft a minimal set of features for the first version.

Write a concise project requirements document specifying the problem, solution, risks, and key success metrics.

Set up the project structure and build the basic frontend UI for your capstone.

Implement the core backend logic to call the chosen LLM API for your capstone.

Integrate the final prompt chain and any RAG components for the capstone's main feature.

Implement input/output guardrails and error handling tailored to the capstone's risks.

Deploy the capstone product to a live URL using a simple hosting service.

Create a golden dataset and run a full evaluation suite for your capstone product.

Implement server-side logging to save user inputs and AI-generated outputs for every interaction.

Add a frontend feedback mechanism (e.g., thumbs up/down) and log user feedback alongside responses.

Analyze evaluation results and logs to identify the single most impactful failure mode.

Implement a specific change to address the identified failure and re-run the evaluation to confirm improvement.

Outline a product demo narrative using a Problem-Solution-Key Decisions framework.

Write a concise script for a 3-minute video demo that walks through your product's core user flow.