Module 1: Explore compute and storage options for data engineering
workloads
This module provides an overview of the Azure compute and storage technology
options that are available to data engineers building analytical workloads. This
module teaches ways to structure the data lake, and to optimize the files for
exploration, streaming, and batch workloads. The student will learn how to
organize the data lake into levels of data refinement as they transform files
through batch and stream processing. Then they will learn how to create indexes
on their datasets, such as CSV, JSON, and Parquet files, and use them for
potential query and workload acceleration.
Module 2: Run interactive queries using Azure Synapse Analytics
serverless SQL pools
In this module, students will learn how to work with files stored in the data
lake and external file sources, through T-SQL statements executed by a
serverless SQL pool in Azure Synapse Analytics. Students will query Parquet
files stored in a data lake, as well as CSV files stored in an external data
store. Next, they will create Azure Active Directory security groups and enforce
access to files in the data lake through Role-Based Access Control (RBAC) and
Access Control Lists (ACLs).
Module 3: Data exploration and transformation in Azure
Databricks
This module teaches how to use various Apache Spark DataFrame methods to
explore and transform data in Azure Databricks. The student will learn how to
perform standard DataFrame methods to explore and transform data. They will also
learn how to perform more advanced tasks, such as removing duplicate data,
manipulate date/time values, rename columns, and aggregate data.
Module 4: Explore, transform, and load data into the Data Warehouse
using Apache Spark
This module teaches how to explore data stored in a data lake, transform the
data, and load data into a relational data store. The student will explore
Parquet and JSON files and use techniques to query and transform JSON files with
hierarchical structures. Then the student will use Apache Spark to load data
into the data warehouse and join Parquet data in the data lake with data in the
dedicated SQL pool.
Module 5: Ingest and load data into the data warehouse
This module teaches students how to ingest data into the data warehouse
through T-SQL scripts and Synapse Analytics integration pipelines. The student
will learn how to load data into Synapse dedicated SQL pools with PolyBase and
COPY using T-SQL. The student will also learn how to use workload management
along with a Copy activity in a Azure Synapse pipeline for petabyte-scale data
ingestion.
Module 6: Transform data with Azure Data Factory or Azure Synapse
Pipelines
This module teaches students how to build data integration pipelines to
ingest from multiple data sources, transform data using mapping data flowss, and
perform data movement into one or more data sinks.
Module 7: Orchestrate data movement and transformation in Azure
Synapse Pipelines
In this module, you will learn how to create linked services, and orchestrate
data movement and transformation using notebooks in Azure Synapse Pipelines.
Module 8: End-to-end security with Azure Synapse
Analytics
In this module, students will learn how to secure a Synapse Analytics
workspace and its supporting infrastructure. The student will observe the SQL
Active Directory Admin, manage IP firewall rules, manage secrets with Azure Key
Vault and access those secrets through a Key Vault linked service and pipeline
activities. The student will understand how to implement column-level security,
row-level security, and dynamic data masking when using dedicated SQL pools.
Module 9: Support Hybrid Transactional Analytical Processing (HTAP)
with Azure Synapse Link
In this module, students will learn how Azure Synapse Link enables seamless
connectivity of an Azure Cosmos DB account to a Synapse workspace. The student
will understand how to enable and configure Synapse link, then how to query the
Azure Cosmos DB analytical store using Apache Spark and SQL serverless.
Module 10: Real-time Stream Processing with Stream
Analytics
In this module, students will learn how to process streaming data with Azure
Stream Analytics. The student will ingest vehicle telemetry data into Event
Hubs, then process that data in real time, using various windowing functions in
Azure Stream Analytics. They will output the data to Azure Synapse Analytics.
Finally, the student will learn how to scale the Stream Analytics job to
increase throughput.
Module 11: Create a Stream Processing Solution with Event Hubs and
Azure Databricks
In this module, students will learn how to ingest and process streaming data
at scale with Event Hubs and Spark Structured Streaming in Azure Databricks. The
student will learn the key features and uses of Structured Streaming. The
student will implement sliding windows to aggregate over chunks of data and
apply watermarking to remove stale data. Finally, the student will connect to
Event Hubs to read and write streams.
