Projects
Assignments
Overview
With this Big Data Hadoop course, you will learn the big data framework using Hadoop and Spark, including HDFS, YARN, and MapReduce. The course will also cover Pig, Hive, and Impala to process and analyse large datasets stored in the HDFS and use Sqoop and Flume for data ingestion.
Objectives
At the end of BDHS training, participants will be able to understand:
Prerequisites
- There are no prerequisites for this course. However, it’s beneficial to have some knowledge of Core Java and SQL.
Course Outline
Introduction to Apache Hadoop and the Hadoop Ecosystemedforce2021-11-12T16:10:20+05:30
- Apache Hadoop Overview
- Data Processing
- Introduction to the Hands-On Exercises
Apache Hadoop File Storageedforce2021-11-12T16:10:43+05:30
- Apache Hadoop Cluster Components
- HDFS Architecture
- Using HDFS
Distributed Processing on an Apache Hadoop Clusteredforce2021-11-12T16:11:05+05:30
- YARN Architecture
- Working With YARN
Apache Spark Basicsedforce2021-11-12T16:11:22+05:30
- What is Apache Spark?
- Starting the Spark Shell
- Using the Spark Shell
- Getting Started with Datasets and DataFrames
- DataFrame Operations
Working with DataFrames and Schemasedforce2021-11-12T16:11:39+05:30
- Creating DataFrames from Data Sources
- Saving DataFrames to Data Sources
- DataFrame Schemas
- Eager and Lazy Execution
Analyzing Data with DataFrame Queriesedforce2021-11-12T16:23:27+05:30
- Querying DataFrames Using Column Expressions
- Grouping and Aggregation Queries
- Joining DataFrames
RDD Overviewedforce2021-11-12T16:23:44+05:30
- RDD Overview
- RDD Data Sources
- Creating and Saving RDDs
- RDD Operations
Transforming & Aggregating Data with RDDsedforce2021-11-12T16:24:48+05:30
- Writing and Passing Transformation Functions
- Transformation Execution
- Converting Between RDDs and DataFrames
- Key-Value Pair RDDs
- Map-Reduce
- Other Pair RDD Operations
Working with Datasets in Scalaedforce2021-11-12T16:25:08+05:30
- Datasets and DataFrames
- Creating Datasets
- Loading and Saving Datasets
- Dataset Operations
Writing, Configuring, and Running Spark Applicationsedforce2021-11-12T16:26:24+05:30
- Writing a Spark Application
- Building and Running an Application
- Application Deployment Mode
- The Spark Application Web UI
- Configuring Application Properties
Spark Distributed Processingedforce2021-11-12T16:26:42+05:30
- Review: Apache Spark on a Cluster
- RDD Partitions
- Example: Partitioning in Queries
- Stages and Tasks
- Job Execution Planning
- Example: Catalyst Execution Plan
- Example: RDD Execution Plan
Structured Streamingedforce2021-11-12T16:28:11+05:30
- Apache Spark Streaming Overview
- Creating Streaming DataFrames
- Transforming DataFrames
- Executing Streaming Queries
- Receiving Kafka Messages
- Sending Kafka Messages
