PySpark Execution and DataFrame Foundations

Explain how the Spark driver, executors, lazy evaluation, jobs, stages, and tasks interact when a DataFrame action runs.

Predict which PySpark DataFrame transformations create narrow dependencies, wide dependencies, and shuffles.

Implement DataFrame filtering, projection, aggregation, and conditional logic for a given data-transformation requirement.

Apply PySpark window functions to deduplicate records and calculate ranked or cumulative results.

Enforce explicit schemas and safely process null, malformed, and nested records in PySpark.