Data Engineering

Use Python scalar types, operators, and type conversions to transform individual data values.

Use conditional statements and loops to apply data-processing rules.

Create, index, slice, and modify sequential data using Python lists and tuples.

Map, look up, and deduplicate data records using Python dictionaries and sets.

Write focused Python functions with parameters, return values, and clear docstrings.

Organize reusable Python code into modules and import it from an entry-point script.

Read and write text, CSV, and JSON data with explicit character encoding.

Parse, compare, and format timezone-aware dates and timestamps in Python.

Publish a local feature branch to GitHub, open a pull request, and synchronize the merged changes.

Use exceptions and context managers to release resources safely when a pipeline step fails.

Load configuration and secrets from environment variables without committing credentials.

Retrieve JSON data from an HTTP API using query parameters and response-status checks.

Handle HTTP timeouts, rate limits, and transient failures with bounded retries.

Flatten and normalize nested API records into a tabular structure.

Emit structured logs with useful severity, timestamp, and execution-context fields.

Write pytest unit tests for deterministic transformation functions.

Use psql to connect to PostgreSQL and implement a provided schema with appropriate data types, primary keys, foreign keys, and nullability constraints.

Retrieve selected rows and calculated columns with SELECT, WHERE, and expressions.

Handle nulls and control query output with ordering, distinct values, and row limits.

Combine related tables with inner, left, and full joins.

Summarize data with aggregate functions, GROUP BY, and HAVING.

Structure multi-step queries with subqueries and common table expressions.

Modify rows safely with INSERT, UPDATE, DELETE, and transaction control.

Bulk-import CSV data into PostgreSQL with COPY while handling headers, delimiters, and null values.

Calculate rankings, running totals, and period comparisons with SQL window functions.

Transform dates, text, and categories with SQL date, string, and conditional expressions.

Deduplicate records by retaining the latest row for each business key.

Write SQL checks for duplicate keys, missing values, invalid ranges, and broken relationships.

Choose a useful PostgreSQL index for a stated query and workload.

Interpret an EXPLAIN ANALYZE plan to identify scans, joins, and performance bottlenecks.

Solve and validate a timed analytical SQL scenario using CTEs and window functions.

Derive entities, relationships, and cardinalities from a set of business rules.

Normalize a relational model through third normal form to remove update anomalies.

Choose natural or surrogate keys and enforce integrity with appropriate constraints.

Declare the grain of a proposed analytical fact table before selecting its columns.

Classify measures, dimensions, and fact-table types for an analytics requirement.

Design a star schema that supports a defined set of e-commerce metrics.

Implement a slowly changing dimension Type 2 history with effective dates and a current-row indicator.

Select and justify CSV, JSON, or Parquet for a pipeline stage based on schema, interoperability, compression, and scan efficiency.

Load, inspect, and assign explicit data types to tabular data with pandas.

Clean and transform pandas DataFrames by handling missing values, filtering rows, and modifying data types.

Merge and aggregate pandas DataFrames while verifying expected row counts to prevent accidental data loss or duplication.

Build a paginated API extractor that writes unchanged source records to a raw-data area.

Extract newly created or changed database rows using a persisted high-water mark.

Load records into PostgreSQL with a transactional upsert.

Make a batch load idempotent so rerunning the same input does not duplicate results.

Reconcile source and target counts and record audit metadata for each pipeline run.

Implement automated schema, null, uniqueness, range, and relationship checks for a dataset.

Define a simple data contract and detect an incompatible source-schema change.

Package a Python pipeline and its pinned dependencies in a Docker image.

Run a pipeline and PostgreSQL together with Docker Compose and persistent storage.

Express pipeline tasks and dependencies as an Apache Airflow DAG.

Configure Airflow schedules, retries, timeouts, and failure callbacks for a batch workflow.

Perform a date-range backfill without relying incorrectly on the wall-clock time.

Use Airflow task logs and metadata to diagnose and recover a failed pipeline run.

Secure an AWS learning account with multi-factor authentication, a budget alert, and cost-cleanup habits.

Interpret and create a least-privilege IAM policy for access to specified data resources.

Configure AWS command-line access without embedding long-lived credentials in source code.

Organize raw, cleaned, and curated data in Amazon S3 using consistent prefixes and partitions.

Catalog S3 data with AWS Glue and query it through Amazon Athena.

Load staged S3 data into Amazon Redshift Serverless and validate the resulting table.

Select AWS services for a batch analytics pipeline and estimate the main sources of cost.

Trace a simple Spark DataFrame job through lazy evaluation, partitions, stages, tasks, and shuffles.

Create a PySpark DataFrame with an explicit schema and inspect malformed input records.

Transform PySpark columns and rows using built-in expressions rather than Python row loops.

Aggregate PySpark DataFrames using groupBy and built-in summary functions.

Calculate rankings, running totals, and lead or lag values using PySpark Window specifications.

Choose a join strategy by considering data size, shuffles, broadcast joins, and key skew.

Read and write partitioned Parquet datasets without creating an excessive number of small files.

Interpret a Spark physical plan and improve a job using filtering, partitioning, and selective caching.

Implement an incremental PySpark transformation that produces repeatable output.

Convert stakeholder questions into pipeline requirements, freshness targets, and acceptance checks.

Draw a source-to-serving AWS architecture for an incremental e-commerce analytics platform.

Design the capstone's PostgreSQL source schema to capture operational e-commerce entities and transactions.

Design a Redshift dimensional star schema and declare explicit grains for its analytical tables.

Implement an incremental extractor that lands replayable source records in date-partitioned Amazon S3 raw storage.

Use PySpark to clean and deduplicate raw records into validated, conformed staging datasets.

Load dimensions incrementally into Redshift while preserving slowly changing dimension Type 2 history.

Build an idempotent incremental fact load that resolves dimensional keys and enforces the declared fact-table grain.

Orchestrate normal runs and date-range backfills with an Airflow DAG.

Integrate data-quality checks and execution audit records into the capstone pipeline.

Publish a capstone README that explains setup, architecture, data flow, design decisions, cost controls, and verification steps.

Configure a GitHub Actions workflow to run unit tests and data checks on proposed code changes.

Present a concise capstone walkthrough that connects technical choices to business requirements.

Answer a data-pipeline system-design prompt using requirements, scale, reliability, security, and cost trade-offs.

Solve a timed Python data-transformation exercise and test the result against edge cases.

Translate capstone evidence and transferable implementation experience into résumé bullets targeted to data-engineering roles in India.