Data Engineers, Python Developers, Freshers

Section 1: Spark Architecture & Internals

- Distributed Computing Fundamentals

  - RDD lineage, DAG scheduler, lazy evaluation

  - Cluster managers overview

- Spark 4.x Updates

  - Adaptive Query Execution (AQE) enhancements

  - Catalyst optimizer improvements

- Performance Tuning

  - Joins

  - Partitioning, broadcast variables

  - Memory management

Section 2: PySpark DataFrames & SQL

- Data Manipulation

  - Complex types (JSON, arrays, maps)

  - Window functions, pivot tables, UDFs/Pandas UDFs

- Spark SQL Deep Dive

  - Temp views, catalog API, Hive metastore integration

  - SQL syntax for Delta Lake operations

- Execution Plans

  - Reading `explain()` output

  - Predicate pushdown, partition pruning

Section 3: Incremental Data Processing & Apache Kafka

- Structured Streaming

  - Event-time processing, watermarking, state management

  - Kafka integration (source/sink)

- Delta Lake Essentials

  - ACID transactions

  - Schema evolution

Section 4: Spark Optimizations

- Catalyst Internals

  - Logical vs. physical plans

  - Custom optimization extensions

- Performance Best Practices

  - File formats (Parquet/Delta)

  - Resource allocation (executors/cores)

Section 5: Databricks Lakehouse Platform

- Lakehouse fundamentals

- Workspace Navigation

  - DBFS, clusters, notebooks

- Delta Lake UI

  - Viewing table history/schema

- Data Governance

  - Unity Catalog basics (no Admin tasks)

Section 6: Apache Kafka Fundamentals  

- Architecture

  - Brokers, topics, partitions, consumer groups

- Spark-Kafka Integration

  - Structured Streaming with Kafka

  - Job execution

Section 7: Spark ML Introduction

- MLlib Workflow

  - Transformers vs. estimators, pipelines

  - Feature engineering (VectorAssembler, StringIndexer)

- Model Training

  - Regression demo (no hyperparameter tuning)

Section 8: Capstone Project

- Pipeline implementation

- Domain Examples: IoT monitoring, retail analytics

1. Python: Language Fundamentals, Functions, Collections, Pandas, ...

2. SQL: CRUD Operations, Group By, Joins, Analytical queries, …

3.Good to have: Linux basics, Hadoop/Hive knowledge beneficial

- Local Installation: Spark 4.x, Java 11, Python 3.10  

- Cloud: Databricks Community/Free Edition

• Master PySpark DataFrames/SQL for batch & stream processing
• Build optimized pipelines using Catalyst insights
• Understand Spark job execution internals
• Understand Apache Kafka and Integrate with Spark
• Hands-on implementation of capstone project
• Certification-ready skills

1. Developer-Centric Focus:

- Covers PySpark application development (coding, debugging, optimization).

- Excludes: Cluster administration, infrastructure setup (YARN/K8s), or Spark cluster tuning.

2. Machine Learning Scope:

- Only introductory-level Spark ML (pipeline structure, basic concept).

- Excludes: Advanced ML concepts (hyperparameter tuning, etc), DL frameworks, or MLOps.

3. Language & Environment:

- PySpark (Python API) only – Scala/Java/R APIs not covered.

- Databricks usage focuses on developer work, not account/admin management.

4. Kafka Integration:

- Covers Spark-as-Consumer/Producer – not professional Kafka cluster setup, security, or Streams API.

5. Infrastructure Assumptions:

- All labs use local/standalone mode or Databricks Community Edition.

17th Nov 25