Projects
Assignments
Overview
This training course will help participants to gain the skills they need to store, manage, process, and analyze massive amounts of structured and unstructured data to extract meaningful insights.
Objectives
At the end of Intro to Big Data & Hadoop training course, participants will
Prerequisites
Before undertaking a Big Data and Hadoop course, participant is recommended to have a basic knowledge of programming languages like Python, Scala, Java and a better understanding of SQL and RDBMS.
Course Outline
Introductionedforce2021-06-29T11:41:32+05:30
- Understanding Big Data
- Types of Big Data
- Difference between Traditional Data and Big Data
- Introduction to Hadoop
- Distributed Data Storage In Hadoop, HDFS and Hbase
- Hadoop Data processing Analyzing Services MapReduce and spark, Hive Pig and Storm
- Data Integration Tools in Hadoop
- Resource Management and cluster management Services
Big Data Ecosystemedforce2021-06-29T11:41:38+05:30
- Need of Hadoop in Big Data
- Understanding Hadoop And Its Architecture
- The MapReduce Framework
- What is YARN?
- Understanding Big Data Components
- Monitoring, Management and Orchestration Components of Hadoop Ecosystem
- Different Distributions of Hadoop
- Installing Hadoop 3
Hadoop Cluster Configurationedforce2021-06-29T11:41:43+05:30
- Hortonworks sandbox installation & configuration
- Hadoop Configuration files
- Working with Hadoop services using Ambari
- Hadoop Daemons
- Browsing Hadoop UI consoles
- Basic Hadoop Shell commands
- Eclipse & winscp installation & configurations on VM
Big Data Processing with MapReduceedforce2021-06-29T11:41:49+05:30
- Running a MapReduce application in MR2
- MapReduce Framework on YARN
- Fault tolerance in YARN
- Map, Reduce & Shuffle phases
- Understanding Mapper, Reducer & Driver classes
- Writing MapReduce WordCount program
- Executing & monitoring a Map Reduce job
Batch Analytics with Apache Sparkedforce2021-06-29T11:25:03+05:30
- SparkSQL and DataFrames
- DataFrames and the SQL API
- DataFrame schema
- Datasets and encoders
- Loading and saving data
- Aggregations
- Joins
Real Time Analytics with Apache Sparkedforce2021-06-29T11:25:30+05:30
- A short introduction to streaming
- Spark Streaming
- Discretized Streams
- Stateful and stateless transformations
- Checkpointing
- Operating with other streaming platforms (such as Apache Kafka)
- Structured Streaming
Analysis using Pigedforce2021-06-29T11:25:53+05:30
- Background of Pig
- Pig architecture
- Pig Latin basics
- Pig execution modes
- Pig processing – loading and transforming data
- Pig built-in functions
- Filtering, grouping, sorting data
- Relational join operators
- Pig Scripting
- Pig UDF’s
Analysis using Hive Data Warehousing Infrastructureedforce2021-06-29T11:26:48+05:30
- Background of Hive
- Hive architecture
- Hive Query Language
- Derby to MySQL database
- Managed & external tables
- Data processing – loading data into tables
- Hive Query Language
- Using Hive built-in functions
- Partitioning data using Hive
- Bucketing data
- Hive Scripting
- Using Hive UDF’s
Working with HBaseedforce2021-06-29T11:27:27+05:30
- HBase overview
- Data model
- HBase architecture
- HBase shell
- Zookeeper & its role in HBase environment
- HBase Shell environment
- Creating table
- Creating column families
- CLI commands – get, put, delete & scan
- Scan Filter operations
Importing and Exporting Data using Sqoopedforce2021-06-29T11:42:37+05:30
- Importing data from RDBMS to HDFS
- Exporting data from HDFS to RDBMS
- Importing & exporting data between RDBMS & Hive tables
Oozie Workflow Management and Using Flume for Analyzing Streaming Dataedforce2021-06-29T11:28:10+05:30
- Overview of Oozie
- Oozie Workflow Architecture
- Creating workflows with Oozie
- Introduction to Flume
- Flume Architecture
- Flume Demo
Visualizing Big Dataedforce2021-06-29T11:28:45+05:30
- Introduction
- Tableau
- Chart types
- Data visualization tools
