AI Infra Engineering

Explain how GPU architecture (streaming multiprocessors, memory hierarchy, warp-based execution) impacts AI workload performance.

Differentiate between GPU memory types (global, shared, registers) and their performance implications for CUDA kernels.

Analyze GPU memory bandwidth versus compute throughput as potential bottlenecks in AI workloads.

Describe multi-GPU and multi-node communication topologies, including NVLink, PCIe, and InfiniBand/Ethernet (RDMA).

Explain collective communication primitives used in distributed AI: AllReduce, AllGather, and ReduceScatter.

Build Docker images with GPU support using the NVIDIA Container Toolkit.

Describe Kubernetes core resources (Pods, Deployments, Services) and their roles in managing AI workloads.

Configure Kubernetes for GPU scheduling using device plugins and resource requests/limits.

Configure persistent storage for model artifacts and datasets in Kubernetes using PersistentVolumes.

Create and deploy a Helm chart for a single-container AI serving application.

Customize a Helm chart to deploy a multi-container AI application with dependencies and configuration.

Trace the forward pass of a Transformer, identifying the most compute- and memory-intensive operations (attention and FFN).

Calculate the GPU memory required to load an LLM given its parameter count and data precision (e.g., FP16, INT8).

Explain how the KV cache works during autoregressive generation and how its memory usage scales with sequence length and batch size.

Differentiate between the prefill and decode phases of LLM inference and their distinct computational profiles.

Estimate the theoretical inference latency and throughput for an LLM on a given hardware configuration.

Explain the role of vector embeddings in AI applications and why specialized databases are required for retrieval.

Compare vector search algorithms (brute-force, IVF, HNSW) on their accuracy, speed, and memory trade-offs.

Describe the graph construction and search process of the HNSW algorithm for approximate nearest neighbor search.

Explain how vector databases handle storage, indexing, and retrieval at scale, including sharding, replication, and metadata filtering.

Compare managed versus self-hosted vector database solutions (e.g., Pinecone, Milvus) on scalability, features, and operational complexity.

Design a scalable vector search pipeline, including embedding generation, indexing strategy, and query execution.

Explain data parallelism and how gradient synchronization is achieved using AllReduce.

Explain tensor parallelism and how it splits individual model layers across multiple devices.

Explain pipeline parallelism and how it partitions a model into stages executed on a sequence of devices.

Compare the memory savings and communication overheads of data, tensor, and pipeline parallelism.

Describe how the ZeRO (Zero Redundancy Optimizer) stages progressively reduce memory consumption in data parallelism.

Explain the concept of mixed-precision training and its impact on memory usage and throughput.

Explain how weight quantization (e.g., INT8, INT4) reduces model size and increases inference speed, and analyze the associated quality trade-offs.

Describe how FlashAttention optimizes the attention mechanism for speed and memory efficiency.

Explain how PagedAttention optimizes KV cache memory management to reduce internal fragmentation.

Describe how continuous batching improves server throughput over static batching by dynamically managing requests.

Explain speculative decoding and identify scenarios where it provides significant latency benefits.

Use a tool like NVIDIA TensorRT or `torch.compile` to build an optimized inference engine for an LLM and benchmark its performance.

Deploy an LLM using a high-throughput serving framework like vLLM, and configure its key performance parameters.

Explain how vLLM integrates PagedAttention and continuous batching to achieve high throughput.

Deploy an LLM using Hugging Face Text Generation Inference (TGI) and configure it for production use.

Describe the architecture of NVIDIA Triton Inference Server, including its model repository, scheduling, and multi-backend support.

Deploy an LLM using Triton Inference Server and configure multi-GPU tensor-parallel serving.

Compare serving frameworks (vLLM, TGI, Triton) on latency, throughput, and operational complexity for a specific use case.

Explain the role of an LLM gateway in production for routing, abstraction, and control.

Set up an API gateway (e.g., LiteLLM) to route requests across multiple LLM providers or self-hosted models.

Implement a request routing strategy to direct traffic based on priority, cost, or model capabilities.

Configure rate limiting and usage quotas for different users or API keys.

Implement fallback and retry logic to improve service reliability across multiple model endpoints.

Set up cost tracking and token usage monitoring across different LLM providers.

Define and instrument key LLM serving metrics: time-to-first-token, inter-token latency, and end-to-end latency.

Set up GPU monitoring (utilization, memory, temperature) using Prometheus and DCGM exporters.

Implement distributed tracing to track an LLM request across service boundaries (e.g., gateway, inference server).

Define Service Level Objectives (SLOs) for an LLM service and configure alerting rules to detect violations.

Implement health checks and readiness probes for LLM serving containers in Kubernetes.

Design a graceful degradation strategy for an LLM service, such as falling back to a smaller model under high load.

Design an autoscaling policy for an LLM inference service based on request load and GPU utilization metrics.

Design a request queuing system to manage burst traffic and prevent server overload.

Explain the role of LoRA for efficient fine-tuning and how to serve LoRA adapters alongside a base model.

Implement a canary release for a new LLM version using Kubernetes traffic splitting features.

Design a resource allocation strategy for co-locating multiple models or LoRA adapters on a shared GPU fleet.

Analyze the cost-performance trade-offs between different GPU types (e.g., A100, H100, L4) for a given inference workload.

Set up a model registry (e.g., MLflow, ClearML) for versioning and managing LLM artifacts.

Design a CI/CD pipeline for automatically building, testing, and deploying LLM serving infrastructure.

Explain the key infrastructure requirements for multi-agent systems, including state management, inter-agent communication, and tool execution.

Design a high-level architecture for a multi-agent orchestration platform, identifying key components and their interactions.

Integrate structured logging and monitoring into a CI/CD pipeline to ensure observability from deployment to production.

Develop a comprehensive system design for a production-grade LLM application, integrating all concepts from the course.