Inference Engineering

Design a reproducible inference benchmark with controlled prompts, generation settings, warm-ups, and repeated trials.

Measure latency percentiles, time to first token, inter-token latency, token throughput, and error rate.

Create asynchronous Python load tests at multiple concurrency levels.

Measure how input length and generated-output length affect latency and throughput.

Monitor CPU, memory, and process utilization during an inference benchmark on macOS.

Identify whether a measured workload is limited primarily by compute, memory, or queueing.

Explain how weight precision affects model memory, bandwidth demand, speed, and output quality.

Compare common GGUF and MLX quantization choices for a constrained device.

Quantize or convert a small open-weight model into a format supported by a local inference engine.

Evaluate two quantized variants against a compact task-specific quality test set.

Tune context length, batch parameters, and thread settings within a fixed memory limit.

Select a model-format-engine combination from benchmark and quality evidence.

Launch an OpenAI-compatible HTTP endpoint backed by a local inference engine.

Implement request validation and generation controls in a FastAPI inference gateway.

Stream generated tokens to a client using server-sent events.

Implement client cancellation and inference timeouts without leaking server work.

Enforce bounded concurrency and return explicit overload responses.

Add liveness and readiness checks that distinguish process health from model readiness.

Explain why static batching wastes capacity for variable-length generation workloads.

Describe how continuous batching improves accelerator utilization and throughput.

Choose a request scheduling policy for interactive and batch traffic with different priorities.

Explain how paged KV-cache management reduces fragmentation and enables larger active batches.

Determine when prefix caching can reduce prefill work for repeated prompt prefixes.

Assess when speculative decoding can reduce latency and what limits its benefit.

Configure prompt caching or speculative decoding in a supported local engine and measure its effect.

Explain how GPU compute, memory capacity, and memory bandwidth constrain LLM inference.

Use arithmetic intensity to reason about why prefill and decode exhibit different bottlenecks.

Compare llama.cpp, MLX, vLLM, Hugging Face TGI, and TensorRT-LLM against explicit workload constraints.

Construct a valid vLLM serving configuration for a specified model, GPU budget, and context requirement.

Distinguish tensor, pipeline, and data parallelism by communication cost and suitable use case.

Choose a single-node or distributed serving layout for a specified model and traffic profile.

Explain the roles of container images, registries, instances, persistent storage, and network endpoints in a deployment.

Build a reproducible container image for an inference gateway.

Run the gateway and model server as separate services with Docker Compose.

Externalize model paths, runtime settings, credentials, and environment-specific configuration.

Draw a cloud-neutral inference architecture with routing, replicas, model storage, and observability.

Choose scaling boundaries for the gateway and model-serving tiers independently.

Define service-level indicators and objectives for latency, availability, throughput, and quality.

Expose inference request, token, queue, latency, and error metrics in Prometheus format.

Add structured logs that correlate requests with model settings and timing data without recording prompt content.

Create a dashboard that reveals saturation, queue growth, latency degradation, and failures.

Diagnose out-of-memory errors, stalled generation, request timeouts, and overload from operational evidence.

Apply backpressure, load shedding, retry limits, and graceful shutdown to protect the service.

Implement basic API authentication, input limits, and secret handling for an inference endpoint.

Translate a workload distribution and latency objective into a capacity estimate.

Calculate cost per request and cost per generated token from serving-resource prices and utilization.

Compare scale-up, scale-out, autoscaling, and scale-to-zero strategies for a variable workload.

Choose autoscaling signals that account for queue depth and active generation rather than CPU alone.

Evaluate model routing and fallback policies for balancing quality, latency, and cost.

Run a before-and-after load test that demonstrates one evidence-based serving optimization.

Produce an architecture diagram and operations runbook for the optimized service.

Present benchmark results, trade-offs, limitations, and next steps in a portfolio-ready technical report.