Data Engineering Concepts

Trace a Spark SQL operation from driver-side planning through executor task completion.

Infer Spark stage boundaries from narrow and wide dependencies in a DAG.

Predict when lazy evaluation will cause recomputation or defer a runtime failure.

Calculate effective task parallelism from input files, partitions, and available executor slots.

Differentiate driver, executor, and scheduler failures using production symptoms.

Diagnose cluster underutilization from a Spark DAG and its task metrics.

Interpret the analyzed, optimized, and physical sections of a Spark SQL execution plan.

Determine how missing or inaccurate statistics affect Spark's physical-plan choices.

Select an appropriate Spark join strategy from table sizes, data distribution, and cluster constraints.

Explain how columnar execution and whole-stage code generation affect query performance.

Analyze the changes made by Adaptive Query Execution in before-and-after physical plans.

Diagnose a Spark query-plan regression using plans, statistics, and runtime evidence.

Trace the shuffle write, transfer, and fetch lifecycle for a wide Spark transformation.

Choose an initial partition count from data volume, cluster parallelism, and target task duration.

Detect data skew using partition-level task and shuffle metrics.

Select between repartitioning, coalescing, and range partitioning for a specified workload.

Apply an appropriate skew-mitigation technique to a production join or aggregation.

Benchmark two partitioning strategies while controlling for cache and cluster effects.

Explain how Spark's unified memory model divides execution and storage memory.

Classify a Spark out-of-memory failure as driver heap, executor heap, overhead, or result-size related.

Derive an executor-sizing configuration from worker resources and workload characteristics.

Diagnose garbage-collection pressure and disk spill using executor metrics.

Select a persistence level based on recomputation cost, data size, and serialization overhead.

Evaluate how dynamic allocation changes resource utilization and shuffle behavior.

Explain how Parquet row groups, pages, encodings, and column statistics support selective reads.

Predict when column pruning, predicate pushdown, and partition pruning will reduce physical I/O.

Diagnose the performance impact of small files on distributed reads and metadata operations.

Select a file-sizing and compaction strategy for a specified ingestion pattern.

Trace a Delta Lake transaction from snapshot read through optimistic commit validation.

Compare Delta Lake, Apache Iceberg, and Apache Hudi transaction models for a stated workload.

Determine how Kafka partitioning constrains Spark Structured Streaming parallelism.

Compare Spark micro-batch execution with Flink's record-at-a-time execution model.

Choose an event-time watermark from lateness observations and correctness requirements.

Estimate state-store growth from key cardinality, event rate, and retention policy.

Trace checkpoint, offset, and sink behavior during recovery from a streaming failure.

Diagnose a streaming bottleneck caused by backpressure, state growth, or checkpoint latency.

Interpret Spark UI and event-log metrics for jobs, stages, tasks, storage, and executors.

Differentiate CPU, disk I/O, network, and scheduler bottlenecks from runtime evidence.

Diagnose straggler tasks and determine whether speculative execution is appropriate.

Predict Spark recovery behavior after task, executor, or driver failure.

Design a bounded fault-injection test for a distributed data pipeline.

Construct a controlled tuning experiment with a baseline, one changed variable, and measurable acceptance criteria.

Compare Spark and Flink for a workload with explicit latency, state, and recovery requirements.

Compare Spark and Trino for transformation and interactive-query workloads.

Explain the distinct roles of Kafka as a distributed log and Spark or Flink as processing engines.

Compare lakehouse compute-storage separation with a shared-nothing MPP architecture.

Select a distributed processing engine using workload requirements and operational constraints.

Critique a polyglot data architecture for bottlenecks, consistency risks, and failure propagation.