Master AI Engineering & Architecture

Explain the HTTP request-response cycle, including methods (GET, POST), status codes, and headers.

Explain the concept of asynchronous I/O and the event loop, and why it's beneficial for I/O-bound web applications.

Create a FastAPI application with path parameters, query parameters, and request body endpoints using async handlers.

Define strict request and response schemas using Pydantic models for automatic data validation and API documentation.

Implement a centralized error handling strategy using custom exception handlers to return structured and secure error responses.

Write integration tests for API endpoints using pytest and TestClient, verifying both success and error cases.

Instrument endpoints with middleware to add structured logging for requests and response timing.

Reverse-engineer FastAPI's routing by reading `fastapi/routing.py` to trace how an incoming request is matched to a path operation.

Explain the difference between a container and a virtual machine, and the benefits of containerization for development and deployment.

Write a multi-stage Dockerfile for the FastAPI application, optimizing for layer caching and a small final image size.

Create a `docker-compose.yml` file to define and run the multi-container application stack (API service, PostgreSQL database).

Manage the application's lifecycle (build, start, stop, rebuild) using docker-compose commands.

Configure a persistent volume for the PostgreSQL container to ensure data is not lost when the container is removed.

Use Docker networking to enable communication between the application container and the database container.

Configure environment variables and secrets for the application within the docker-compose setup for secure configuration management.

Debug a running container by accessing its shell and inspecting logs.

Demonstrate the failure modes of basic string splitting for tokenization using examples with punctuation and contractions.

Implement a simple rule-based tokenizer from scratch that correctly handles punctuation and common English contractions.

Explain how subword tokenization (e.g., BPE) balances vocabulary size and out-of-vocabulary issues, and trace the algorithm for three iterations on a sample text.

Implement the core logic of BPE training from scratch: counting character pair frequencies and merging the most frequent pair.

Use a production-grade tokenizer (e.g., from Hugging Face's `tokenizers`) and compare its output and features to your from-scratch implementation.

Analyze the tradeoffs of lemmatization and stopword removal, explaining why they are less critical for modern contextual embedding models.

Tokenize a document using both a word-level and a subword-level tokenizer and compare the resulting token counts and sequences.

Estimate the memory and compute cost of tokenizing a large corpus (10,000 documents) and identify the bottleneck operations.

Explain how an inverted index maps terms to document posting lists and implement a simple one from scratch.

Implement TF-IDF scoring from scratch, explaining the intuition behind term frequency, inverse document frequency, and document length normalization.

Explain how the BM25 ranking formula improves upon TF-IDF through term frequency saturation (k1) and length normalization (b).

Implement BM25 scoring from scratch and use it to rank documents for a sample query.

Define the core retrieval metrics of Precision@K, Recall@K, and Mean Reciprocal Rank (MRR).

Construct a relevance judgment set (qrels) for a small set of narrative-based questions to serve as a ground truth for evaluation.

Evaluate the BM25 implementation using the defined metrics and qrels, and analyze specific query failures.

Articulate the principle of Evaluation-Driven Development, explaining how having an evaluation set first guides effective system development.

Explain how chunk size creates a fundamental tension between context preservation and retrieval precision.

Implement fixed-size chunking with a configurable overlap and analyze the impact of changing these parameters.

Implement a structure-aware chunking strategy that respects paragraph or section boundaries.

Attach structured metadata (e.g., source document, chapter, page number, position index) to each chunk for traceability.

Analyze and categorize chunking failure modes: lost context (too small), diluted relevance (too large), and fragmented semantics (boundary errors).

Design and implement a narrative-aware chunking function that attempts to identify scene or dialogue breaks as chunk boundaries.

Run a controlled experiment comparing fixed-size vs. narrative-aware chunking strategies, using the BM25 system and evaluation set to measure retrieval performance.

Write a report summarizing the experiment's findings and defending the choice of a default chunking strategy for the project's MVP.

Explain the conceptual limitations of sparse, bag-of-words representations (e.g., TF-IDF) that dense vectors aim to solve.

Describe the key insight of word2vec: learning vector representations by predicting a word's context, forcing semantically similar words to have similar vectors.

Distinguish between static embeddings (e.g., word2vec) and contextual embeddings (e.g., BERT) using an example of a polysemous word.

Implement cosine similarity, Euclidean distance, and dot product from scratch, and explain when each metric is appropriate for comparing vectors.

Generate sentence embeddings for several narrative passages using a pre-trained Sentence Transformer model.

Compare the embeddings produced by two different pre-trained models on the same text and articulate which is better suited for narrative retrieval.

Use a dimensionality reduction technique (e.g., PCA, t-SNE) to visualize a small set of chunk embeddings and verify if semantic clusters match your intuition.

Identify and provide examples of common embedding failure modes, such as domain mismatch and antonyms appearing as semantically similar.

Implement a brute-force (exact) k-nearest neighbor search and measure its O(N) query time to establish a baseline.

Install the `pgvector` extension in PostgreSQL, create a table with a vector column, and perform exact KNN search using its distance operators.

Explain the core tradeoff in Approximate Nearest Neighbor (ANN) search between recall (accuracy) and query latency.

Describe how HNSW constructs a multi-layer navigable small-world graph to enable efficient greedy search.

Create an HNSW index in `pgvector`, tune the `m` and `ef_construction` parameters, and measure the impact on index build time.

Run a benchmark to measure the recall-vs-latency tradeoff of the HNSW index at different `ef_search` settings.

Explain the core idea behind Inverted File (IVF) indexes and how partitioning the vector space reduces the search area.

Write an ADR comparing `pgvector` with a standalone library like FAISS, considering tradeoffs in operational complexity, data consistency, and scalability.

Implement an end-to-end retrieval function that takes a query, embeds it, and retrieves the top-k chunks from `pgvector`.

Design a robust prompt template that clearly separates system instructions, retrieved context, and the user query.

Integrate an LLM API call into the pipeline, wrapping it with logic for request timeout, retries, and error handling.

Implement a strategy to handle context window limitations by truncating or prioritizing retrieved chunks based on relevance scores.

Add structured logging with unique request IDs to trace a single query's journey through each pipeline stage (retrieval, prompt assembly, generation).

Implement a fallback response strategy for when the retrieval step returns no relevant documents.

Trace a query through LlamaIndex's `QueryEngine` source code to understand how a production framework orchestrates retrieval and synthesis.

Refactor your from-scratch pipeline to mirror the modular design of a production framework, separating retriever, synthesizer, and engine components.

Run an A/B test comparing two prompt templates on a fixed set of 10 queries, measuring their impact on response quality.

Implement few-shot prompting by dynamically including relevant examples in the prompt, and analyze the tradeoffs versus zero-shot prompting.

Apply chain-of-thought prompting to a multi-step narrative reasoning question and compare the output's quality and coherence.

Force an LLM to return structured JSON that conforms to a Pydantic schema, and implement a validation and retry loop for malformed outputs.

Identify and document common prompt failure modes: instruction drift, context ignoring, and hallucination due to insufficient context.

Create a 'prompt bestiary' by documenting 5 failed prompts, your hypothesis for the failure, and the change that fixed it.

Distinguish which prompt engineering decisions are 'engineering' (template structure, output format) versus 'empirical' (temperature, few-shot example selection).

Defend the final prompt design for the MVP, explaining the alternatives considered and the reasoning behind your choices.

Evaluate the end-to-end RAG system on the evaluation set, computing both retrieval metrics (MRR) and qualitative generation scores.

Categorize observed system failures into 'retrieval failures' (wrong context) versus 'generation failures' (right context, wrong answer).

Implement hybrid search by combining BM25 and dense vector scores using Reciprocal Rank Fusion (RRF).

Run a comparative evaluation of vector-only, BM25-only, and hybrid retrieval, and synthesize the results to identify the strengths of each.

Conduct an ablation study by changing one variable (e.g., chunk size, embedding model) and measuring its impact on end-to-end RAG quality.

Profile the end-to-end RAG pipeline to measure latency at each stage (embedding, retrieval, generation) and identify the primary bottleneck.

Trace Haystack's `Pipeline` class to understand how it models the system as a directed acyclic graph (DAG) of components.

Write a formal evaluation report documenting metrics, categorized failures, ablation study results, and a prioritized backlog of improvement hypotheses.

Write unit tests using `pytest` for critical, non-trivial pure functions in your pipeline (e.g., chunking logic, RRF fusion).

Reflect on the process of adding tests after implementation versus a test-first approach, noting what would have been designed differently.

Write a comprehensive README.md for the project, including architecture overview, setup instructions, and how to run the evaluation.

Prepare and deliver a presentation defending the complete V1 system architecture to a hypothetical panel of senior engineers.

In the defense, walk through every component, justify its existence, explain the engineering tradeoffs made, and address potential failure modes.

For each major dependency (PostgreSQL, FastAPI, Sentence-Transformers), articulate a plan for what it would take to replace it.

Identify the top three sources of technical debt in the current codebase and create backlog tickets to address them.

Perform a 'Failure Analysis' by intentionally breaking a component (e.g., return empty context, corrupt the prompt) and observing the system's resilience.