Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Databricks data engineer certification
·
Module 6
Incremental Data Ingestion
1
Auto Loader's Purpose in Cloud File Ingestion
Describe the purpose of Auto Loader for incrementally ingesting files from cloud storage.
2
Auto Loader for New Files with `cloudFiles`
Configure Auto Loader using `cloudFiles` format to read new files from a directory.
3
Schema Inference with Auto Loader
Configure Auto Loader to infer the schema of streaming data from source files.
Schema Evolution in Auto Loader Streams
Manage schema evolution in an Auto Loader stream to accommodate changes in source data.
5
Processing Corrupt Records with Auto Loader's Rescued Data Column
Handle corrupt or malformed records using Auto Loader's rescued data column.
6
Idempotent Data Ingestion with COPY INTO
Ingest data idempotently from files into a Delta table using the COPY INTO command.
Choosing the Right Ingestion Tool: Auto Loader vs. COPY INTO
Select the appropriate tool (Auto Loader vs. COPY INTO) for a given file ingestion scenario.
Previous module
Transforming Data with PySpark DataFrames
Next module
Processing Streaming Data with Structured Streaming