Parallel Computing, GPUs, and LLMs

Determine whether a GPU global-memory access pattern is coalesced.

Explain how tensor cores and reduced-precision formats accelerate matrix multiplication.

Classify a GPU kernel bottleneck using profiler counters and roofline evidence.

Estimate collective communication time from message volume, latency, and bandwidth.

Map collective traffic onto PCIe, NVLink, and network topology to identify likely bottlenecks.

Distinguish pretraining, continued pretraining, supervised fine-tuning, and alignment by their data and objectives.

Calculate causal-language-model cross-entropy loss with the correct token masking and shifting.

Explain how AdamW updates parameters and accounts for optimizer-state memory.

Select a learning-rate schedule and warmup strategy for a stated training configuration.

Construct a compute and memory budget from transformer dimensions, token count, precision, and optimizer choice.

Design an evaluation matrix covering training health, task quality, robustness, safety, and regression detection.

Differentiate the data requirements of supervised fine-tuning, preference optimization, and retrieval-augmented generation.

Design a data-sourcing plan that addresses licensing, privacy, provenance, representation, and intended use.

Apply deduplication, contamination, leakage, and quality checks to a proposed training dataset.

Calculate inter-annotator agreement and interpret what it implies for preference-label quality.

Choose chunking, metadata, filtering, and refresh policies for a RAG corpus.

Calculate the communication volume and synchronization points in a synchronous data-parallel training iteration.

Compare tensor, pipeline, context, and expert parallelism by the state partitioned and communication required.

Select a combined parallelism strategy for stated model, memory, and cluster-topology constraints.

Compare ZeRO and FSDP sharding stages by what they shard and when communication occurs.

Estimate expected work lost to failures and choose a distributed checkpoint interval and restart strategy.

Map an ML experiment into XManager-style experiments, work units, jobs, parameters, artifacts, and retry policies.

Calculate model FLOP utilization from achieved token throughput and hardware peak performance.

Diagnose input-pipeline, communication, or straggler bottlenecks from GPU timelines and per-rank metrics.

Diagnose loss spikes, NaNs, stalled learning, or overfitting from loss curves, gradient statistics, and optimizer settings.

Use controlled comparisons to isolate a failure to model logic, data, training code, or infrastructure.

Trace a request through gateway, admission control, routing, scheduling, model execution, and token streaming.

Explain how a vLLM-style serving engine combines continuous batching with paged KV-cache management.

Select data, tensor, pipeline, or expert parallelism for an inference workload with stated latency, memory, and topology constraints.

Estimate GPU capacity and choose autoscaling and overload controls from traffic distributions, service rates, headroom, and latency objectives.

Choose routing and colocated or disaggregated prefill-decode strategies from workload characteristics, cache locality, and KV-transfer costs.

Trace a RAG or tool-using request through retrieval, prompt construction, model inference, tool execution, and response validation.

Define the roles of planning, state, memory, tools, and termination conditions in an agent architecture.

Threat-model prompt injection, data exfiltration, excessive agency, and malicious tool outputs.

Apply authentication, authorization, sandboxing, secret isolation, and egress controls to tool execution.

Design evaluations for task success, retrieval quality, tool correctness, security, and cost in an agentic system.

Translate evidence from users and operations into a clearly defined user pain point.

Convert a user problem into measurable product, model-quality, latency, reliability, and business outcomes.

Prioritize an AI architecture or model investment by comparing user impact, engineering cost, risk, and reversibility.

Identify the APIs, abstractions, evaluation tools, observability, and versioning capabilities ML developers need as agent systems evolve.

Design a consent-aware user-interaction data flywheel and an experiment that measures its incremental benefit.

Compare open-source LLM serving projects such as vLLM, TensorRT-LLM, TGI, and SGLang by architecture, performance features, ecosystem, and operability.

Assess trends in models, accelerators, training, serving, and agents while distinguishing durable shifts from short-lived hype.

Formulate strategic AI opportunities for Google by connecting market needs with its research, model, accelerator, cloud, data, and developer-platform assets.