GCP Data Engineering

Design a GCP folder-and-project layout that isolates development, test, and production data workloads.

Authenticate a local Python workload to GCP using Application Default Credentials and service account impersonation.

Construct least-privilege IAM role bindings for a data pipeline's human and service identities.

Select compatible regions or multi-regions for connected GCP data services.

Choose a private connectivity pattern for a managed data workload that should not use public IP addresses.

Create a budget alert for a GCP learning or development project.

Diagram the resource, data, and trust boundaries of a production GCP data platform.

Select among BigQuery, Cloud SQL, Spanner, Bigtable, and GCS for a workload from its access and scaling requirements.

Choose between a star schema and nested or repeated BigQuery fields for a stated analytics use case.

Design BigQuery partitioning and clustering keys from query patterns and data distribution.

Interpret a BigQuery execution plan and INFORMATION_SCHEMA job metrics to locate performance bottlenecks.

Rewrite a BigQuery query to reduce bytes scanned and expensive data shuffling.

Implement an idempotent incremental load with a BigQuery MERGE statement.

Plan a backward-compatible BigQuery schema evolution.

Select an appropriate BigQuery access-control mechanism from authorized views, row-level policies, and policy tags.

Explain how Pub/Sub delivery guarantees affect downstream processing design.

Choose between pull, push, and BigQuery subscriptions for a stated ingestion workload.

Publish schema-valid events to Pub/Sub from a Python producer.

Implement a pull subscriber that acknowledges messages only after successful processing.

Configure Pub/Sub retry behavior and a dead-letter topic for poison messages.

Design message ordering and deduplication using ordering keys and stable event identifiers.

Evaluate a Pub/Sub schema change for backward and forward compatibility.

Design a Datastream-based change-data-capture path from PostgreSQL or MySQL to BigQuery.

Map familiar Spark transformations to the Apache Beam pipeline and PTransform model.

Implement a bounded Apache Beam pipeline that reads from GCS and writes transformed records to BigQuery.

Choose between a side input and CoGroupByKey for an Apache Beam join.

Write deterministic unit tests for an Apache Beam transform with TestPipeline.

Parameterize an Apache Beam pipeline for reuse across environments.

Package and launch a Dataflow Flex Template.

Submit an existing PySpark batch job to Dataproc Serverless.

Choose among BigQuery SQL, Dataflow, and Dataproc for a batch-processing workload.

Distinguish event time from processing time in an out-of-order event stream.

Select fixed, sliding, or session windows for a streaming requirement.

Interpret a Dataflow watermark and predict when a window can produce results.

Configure triggers and allowed lateness for a stated latency-and-completeness requirement.

Implement a streaming Apache Beam pipeline from Pub/Sub to BigQuery.

Apply an event-identifier strategy to make streaming outputs resilient to duplicate delivery.

Diagnose hot keys and backpressure from Dataflow job metrics.

Estimate streaming throughput and select initial Dataflow worker and autoscaling settings.

Define a Cloud Composer DAG with explicit task dependencies and parameterized execution dates.

Choose an Airflow operator, sensor, or deferrable operator for a GCP workflow task.

Make an Airflow task idempotent under retries.

Configure an Airflow backfill without duplicating downstream data.

Orchestrate BigQuery, Dataflow, and Dataproc Serverless jobs from Cloud Composer.

Store and retrieve workflow credentials with Secret Manager and Airflow connections.

Implement and test an incremental dbt model on BigQuery.

Choose among Cloud Composer, Workflows, Cloud Scheduler, and event-driven orchestration for a stated workflow.

Define a measurable service-level indicator and objective for a data pipeline.

Build a Cloud Monitoring dashboard for pipeline freshness, throughput, failures, and resource pressure.

Configure an actionable alert for a breached pipeline service-level objective.

Use Cloud Logging and service metrics to isolate the cause of a failed or delayed pipeline.

Implement a data-quality check that quarantines invalid records without blocking valid data.

Register and trace data assets and lineage with Dataplex Universal Catalog.

Provision a small data-pipeline stack with Terraform using separate environment variables.

Create a CI/CD workflow that tests and deploys a data-pipeline change safely.

Structure a data-system-design response by clarifying functional and non-functional requirements.

Estimate storage volume, event throughput, and peak processing capacity from business assumptions.

Design a cost-aware GCP architecture for a daily batch analytics workload.

Design a low-latency GCP architecture for an event-streaming workload.

Plan replay, disaster recovery, and regional-failure handling for a GCP data pipeline.

Critique a proposed architecture across scalability, reliability, security, operability, and cost.

Diagnose a production data-pipeline incident from logs, metrics, and symptoms in an interview scenario.

Present the course project as a concise architecture walkthrough and evidence-based interview story.