Building AI Teaching Agents

Explain Chain-of-Thought (CoT) and Self-Consistency, and quantify when they help versus hurt performance.

Implement and trace the ReAct pattern (Thought/Action/Observation), explaining why interleaving reasoning with tool use outperforms pure CoT.

Describe Tree-of-Thoughts (ToT) and Graph-of-Thoughts, including the search algorithms (BFS/DFS/beam) used over reasoning nodes.

Explain the Reflexion pattern: verbal self-reflection, episodic memory of past failures, and its relationship to RL-style credit assignment.

Contrast Plan-and-Execute (e.g., Plan-Solve, LLMCompiler) with ReAct in terms of latency, cost, and error recovery.

Describe Self-Refine and LLM-as-a-Judge loops and the conditions under which iterative refinement converges or diverges.

Explain ReWOO (Reasoning WithOut Observation) and how decoupling planning from tool execution can reduce token cost.

Select an appropriate reasoning pattern for a given task, considering constraints on latency, cost, accuracy, and complexity.

Describe how modern LLMs are post-trained for function calling (OpenAI tools, Anthropic tool_use) and the role of the JSON-schema contract.

Design robust tool schemas, focusing on naming, descriptions, parameter typing, and error messages that facilitate self-correction.

Implement parallel tool calling and reason about when it is safe based on tool idempotency and independence.

Explain the Toolformer and Gorilla approaches to teaching tool use via self-supervised data generation.

Explain Model Context Protocol (MCP) and how it standardizes tool exposure across agents relative to native function-calling APIs.

Compare 'code as actions' (e.g., CodeAct, Voyager) with discrete tool calls, analyzing trade-offs in expressivity and debuggability.

Implement reliable structured outputs using native JSON mode coupled with Pydantic schema validation.

Implement guaranteed-correct structured outputs for tool calls using constrained decoding libraries like Outlines or jsonformer.

Describe the memory hierarchy for agents (working, episodic, semantic, procedural) and OS-inspired architectures like MemGPT.

Implement short-term memory strategies, including rolling windows, summarization buffers, and structured scratchpads.

Design a long-term memory store using a vector database, selecting appropriate embedding models, chunking strategies, and metadata filters.

Implement a two-stage retrieval pipeline incorporating hybrid search (lexical + semantic), fusion (RRF), and cross-encoder re-ranking.

Implement context compression techniques (e.g., LLMLingua, selective context) to manage the agent's limited context window.

Design a learner/user model as a form of persistent memory, defining its schema, update policy, and privacy considerations.

Evaluate memory systems using metrics such as recall@k, faithfulness, and their impact on downstream task performance.

Explain when multi-agent decomposition is beneficial versus when it adds overhead and error compounding.

Contrast hierarchical (supervisor/worker) with decentralized (peer-to-peer) multi-agent architectures on control, robustness, and complexity.

Implement a basic supervisor/worker pattern where a manager agent delegates sub-tasks to one or more worker agents.

Describe role-based collaboration as seen in CAMEL, AutoGen GroupChat, and MetaGPT.

Implement a multi-agent debate pattern where agents refine a solution by arguing from different viewpoints.

Implement an adversarial-critic pattern (e.g., Constitutional AI) to align agent outputs with a set of principles.

Design hand-off protocols and communication patterns (e.g., blackboard vs. message-passing, context compression, termination criteria).

Diagnose multi-agent failure modes like echo chambers, context bloat, coordination deadlocks, and groupthink.

Construct an agentic graph in LangGraph with nodes, edges, and conditional routing for dynamic state transitions.

Integrate checkpointing, persistence, and human-in-the-loop interrupts using LangGraph's checkpointer abstraction.

Build a role-based crew in CrewAI and explain its task delegation and process models (sequential vs. hierarchical).

Build a conversational multi-agent system in AutoGen using its GroupChat manager and code-execution agent.

Build an agent with the OpenAI Agents SDK, focusing on handoffs, guardrails, and session management.

Describe agent abstractions in Semantic Kernel and LlamaIndex for planning, tool-calling, and state management.

Contrast Pydantic-AI's type-safe LLM interactions with DSPy's programmatic optimization approach to building agents.

Choose an appropriate framework based on requirements for graph control, multi-agent support, type-safety, and ecosystem.

Distinguish between trajectory-level, step-level, and final-answer metrics for agent evaluation and identify when each is appropriate.

Describe the LLM-as-a-Judge pattern and contrast it with human evaluation and traditional metrics.

Design high-quality judge rubrics and implement pairwise comparison with mitigations for position and verbosity bias.

Select among agent-specific benchmarks (AgentBench, WebArena, SWE-bench, GAIA) for a given evaluation goal.

Instrument an agent with OpenTelemetry-based tracing (e.g., LangSmith, Langfuse) and analyze a trace to debug a failure.

Design and run cost and latency profiling on agent traces to identify performance bottlenecks.

Design a red-team test suite for an agent, including tests for jailbreaks, indirect prompt injection, and data exfiltration.

Diagram a production deployment topology for an agent service, including a stateless API, state store, vector DB, and tool gateway.

Compare state-management strategies (in-memory vs. external DB) and tool-gateway patterns for security and scalability.

Manage long-running agents using durable execution frameworks (e.g., Temporal, Inngest, LangGraph Cloud) and idempotent steps.

Apply prompt/context caching and model-routing strategies, measuring their impact on cost and latency.

Implement token-level and intermediate-step streaming for stateful agents and design UI patterns that consume them.

Design a sandboxed code-execution tool and enumerate its security threat model.

Implement input/output guardrails (e.g., Guardrails AI, NeMo Guardrails) to defend against prompt injection and enforce content policies.

Design RBAC, multi-tenant isolation, and PII redaction for a B2B agent deployment.

Reverse-engineer Sierra's customer-experience agent architecture: skill graphs, tool integration, supervision, and outcome-based evaluations.

Analyze Harvey AI's legal workflow agents, focusing on domain-tuned models, document understanding, and lawyer-in-the-loop design.

Analyze Legora's research/drafting agents and contrast its multi-document reasoning with Harvey's workflow approach.

Design the ingestion and connector architecture for a B2B knowledge agent (e.g., Glean) across heterogeneous sources.

Design a permission-aware retrieval pipeline that enforces Access Control Lists (ACLs) during search and ranking.

Design a reliable answer attribution and citation mechanism for a retrieval-augmented agent.

Compare retrieval-grounded answering and human-in-the-loop patterns across Sierra, Harvey, and Legora.

Describe the design of an audit trail for compliance and debuggability in regulated B2B agents.

Explain Bloom's 2-sigma problem and how it motivates the development of personalized AI tutors.

Map pedagogical strategies like the Socratic method, ZPD scaffolding, and worked examples to concrete agent dialogue policies.

Apply Cognitive Load Theory (intrinsic, extraneous, germane) to design the response patterns and UI of a tutoring agent.

Incorporate formative assessment, spaced repetition (e.g., FSRS), and retrieval practice as core mechanics in a tutoring agent.

Explain Bayesian Knowledge Tracing (BKT) and Deep Knowledge Tracing (DKT) for estimating skill mastery.

Analyze the design of LLM-based open learner models that are both predictive and explainable to the learner.

Design prompts, few-shot examples, and judge rubrics that operationalize an 'understanding-building' (not answer-giving) tutor.

Design ethical and safety controls for learner-facing agents, addressing concerns for minors, dependency, and factual accuracy.

Analyze LearnLM's positioning as a Gemini variant post-trained for pedagogical instruction-following.

Explain LearnLM's seven pedagogical behaviors (e.g., inspire active learning, manage cognitive load) and how they are operationalized.

Describe the data and post-training recipe used for LearnLM, including pedagogical SFT and preference tuning.

Explain LearnLM's evaluation methodology, including pedagogy-focused rubrics and expert-rater studies.

Analyze LearnLM integrations across Google products (Search, NotebookLM, YouTube) through the lens of agent architecture.

Analyze NotebookLM's source-grounded architecture: per-notebook retrieval, citation enforcement, and audio-overview generation.

Critique LearnLM: identify where its pedagogical instruction-following may fail and discuss its extensibility to other languages/cultures.

Reverse-engineer Khanmigo's architecture, including its base model, system prompts, content grounding, and 'never give the answer' policy.

Describe the architectures and pedagogical approaches of Duolingo Max (Roleplay, Explain My Answer) and Synthesis Tutor.

Compare Khanmigo, LearnLM products, and Duolingo Max along axes of base model, grounding data, dialogue policy, and learner modeling.

Design the schema, update logic, and real-time adaptation policy for a capstone tutor's learner-modeling subsystem.

Design an end-to-end evaluation plan for the capstone tutor, including learning-gain proxies and Socratic-adherence rubrics.

Specify the capstone tutor's retrieval and memory architecture for a chosen subject domain.

Specify the capstone tutor's reasoning loop, dialog policy, tool set, and safety guardrails.

Write a build-vs-buy decision document for the capstone, justifying the choice between a specialized API (LearnLM) and an orchestrated general model (GPT/Claude).