AWS Data Engineering

S3 • Glue • Athena • Kinesis • Redshift

Schedule a counselling call

Duration10–12 weeks

Projects2+

Duration

10–12 weeks

Level

Intermediate → Advanced

Mode

Online (Mentor-led)

About this program

Overview

Program snapshot

Design and build AWS data platforms: ingestion, data lake, batch ETL, orchestration, streaming pipelines, analytics warehouses, and production-ready deployments.

Prerequisites

SQL fundamentals
Basic Python programming
Basic cloud awareness is helpful

What you will be able to do

Build data lakes on S3 with partitioning and best practices
Develop ETL pipelines in Glue/EMR (PySpark)
Query using Athena and build marts/warehouse in Redshift
Create streaming pipelines using Kinesis/Kafka patterns
Automate pipelines with orchestration + CI/CD

Module-by-module breakdown

Syllabus

M1Foundations: SQL + Python for Data Engineering

5 topics - 2 labs

Open module

Topics

Advanced SQL patterns used in analytics
Data modeling basics (dimensional concepts)
Python data processing patterns
File formats: CSV/JSON/Parquet
Data quality checks concepts

Hands-on / Labs

Clean + validate raw files with Python
SQL reporting queries for marts

M2S3 Data Lake + Partitioning

5 topics - 2 labs

Open module

Topics

S3 layout design (raw/clean/curated)
Partition strategy
Compression and Parquet
Catalog concepts
Cost/performance basics

Hands-on / Labs

Design lake folder strategy
Convert CSV to partitioned Parquet

M3AWS Glue (Batch ETL)

5 topics - 2 labs

Open module

Topics

Glue jobs overview
Spark basics for ETL
Data transforms, joins, aggregations
SCD patterns overview
Error handling + retries

Hands-on / Labs

Glue ETL: raw → curated
Implement SCD2-style history

M4Athena + Query Optimization

5 topics - 2 labs

Open module

Topics

Athena querying on Parquet
Partition pruning
CTEs and analytics patterns
Cost optimization practices
Views and marts concepts

Hands-on / Labs

Build Athena views for reporting
Optimize partitioned queries

M5Redshift + Warehouse Patterns

5 topics - 2 labs

Open module

Topics

Warehouse vs lakehouse concept
Loading patterns
Dimensional marts (star schema)
Query tuning basics
Access control concepts

Hands-on / Labs

Build a star schema in Redshift
Load curated data and run BI queries

M6Streaming: Kinesis + Event Patterns

5 topics - 2 labs

Open module

Topics

Streaming basics: producers/consumers
Kinesis core concepts
Near real-time aggregation patterns
Idempotency and dedupe concepts
Monitoring concepts

Hands-on / Labs

Stream events and compute aggregates
Store outputs back into S3

M7Orchestration + Automation

5 topics - 2 labs

Open module

Topics

Airflow DAG concepts
Step Functions overview
Lambda in pipelines
Retries, SLAs, backfills
Alerting basics

Hands-on / Labs

Build a DAG for batch pipeline
Trigger ETL via lambda/step functions

M8IaC + CI/CD + Production Readiness

5 topics - 2 labs

Open module

Topics

Terraform basics for data infra
Git workflows
CI pipeline patterns
Environment promotion concepts
Documentation + runbooks

Hands-on / Labs

Terraform skeleton for S3/Glue
CI pipeline for lint/test/deploy

Portfolio-ready work

Projects

Project 1: Batch Data Lake Pipeline

Raw → curated data lake pipeline with Glue and Athena marts.

Deliverables

Partitioned curated dataset
Athena views + reporting queries
Pipeline documentation

Project 2: Streaming Aggregation Pipeline

Event ingestion and aggregation using Kinesis patterns, outputting to S3/marts.

Deliverables

Streaming aggregation logic
Backfill strategy
Monitoring checklist

Capstone

Capstone: End-to-End AWS Data Platform

Design an end-to-end AWS data platform including ingestion, lake, ETL, orchestration, and analytics-ready marts.

Preparing your next view

Fetching content, finishing layout, and keeping navigation smooth.

Overview

Program snapshot

Design and build AWS data platforms: ingestion, data lake, batch ETL, orchestration, streaming pipelines, analytics warehouses, and production-ready deployments.

Prerequisites

SQL fundamentals
Basic Python programming
Basic cloud awareness is helpful

What you will be able to do

Build data lakes on S3 with partitioning and best practices
Develop ETL pipelines in Glue/EMR (PySpark)
Query using Athena and build marts/warehouse in Redshift
Create streaming pipelines using Kinesis/Kafka patterns
Automate pipelines with orchestration + CI/CD

Syllabus

M1Foundations: SQL + Python for Data Engineering

5 topics - 2 labs

Open module

Topics

Advanced SQL patterns used in analytics
Data modeling basics (dimensional concepts)
Python data processing patterns
File formats: CSV/JSON/Parquet
Data quality checks concepts

Hands-on / Labs

Clean + validate raw files with Python
SQL reporting queries for marts

M2S3 Data Lake + Partitioning

5 topics - 2 labs

Open module

Topics

S3 layout design (raw/clean/curated)
Partition strategy
Compression and Parquet
Catalog concepts
Cost/performance basics

Hands-on / Labs

Design lake folder strategy
Convert CSV to partitioned Parquet

M3AWS Glue (Batch ETL)

5 topics - 2 labs

Open module

Topics

Glue jobs overview
Spark basics for ETL
Data transforms, joins, aggregations
SCD patterns overview
Error handling + retries

Hands-on / Labs

Glue ETL: raw → curated
Implement SCD2-style history

M4Athena + Query Optimization

5 topics - 2 labs

Open module

Topics

Athena querying on Parquet
Partition pruning
CTEs and analytics patterns
Cost optimization practices
Views and marts concepts

Hands-on / Labs

Build Athena views for reporting
Optimize partitioned queries

M5Redshift + Warehouse Patterns

5 topics - 2 labs

Open module

Topics

Warehouse vs lakehouse concept
Loading patterns
Dimensional marts (star schema)
Query tuning basics
Access control concepts

Hands-on / Labs

Build a star schema in Redshift
Load curated data and run BI queries

M6Streaming: Kinesis + Event Patterns

5 topics - 2 labs

Open module

Topics

Streaming basics: producers/consumers
Kinesis core concepts
Near real-time aggregation patterns
Idempotency and dedupe concepts
Monitoring concepts

Hands-on / Labs

Stream events and compute aggregates
Store outputs back into S3

M7Orchestration + Automation

5 topics - 2 labs

Open module

Topics

Airflow DAG concepts
Step Functions overview
Lambda in pipelines
Retries, SLAs, backfills
Alerting basics

Hands-on / Labs

Build a DAG for batch pipeline
Trigger ETL via lambda/step functions

M8IaC + CI/CD + Production Readiness

5 topics - 2 labs

Open module

Topics

Terraform basics for data infra
Git workflows
CI pipeline patterns
Environment promotion concepts
Documentation + runbooks

Hands-on / Labs

Terraform skeleton for S3/Glue
CI pipeline for lint/test/deploy

Projects

Project 1: Batch Data Lake Pipeline

Raw → curated data lake pipeline with Glue and Athena marts.

Deliverables

Partitioned curated dataset
Athena views + reporting queries
Pipeline documentation

Project 2: Streaming Aggregation Pipeline

Event ingestion and aggregation using Kinesis patterns, outputting to S3/marts.

Deliverables

Streaming aggregation logic
Backfill strategy
Monitoring checklist

Capstone

Capstone: End-to-End AWS Data Platform

Design an end-to-end AWS data platform including ingestion, lake, ETL, orchestration, and analytics-ready marts.