Unveiling Hive: A Comprehensive Exploration of Hive in Hadoop Ecosystem
•Download as PPTX, PDF•
0 likes•6 views
Dive into the world of Apache Hive with our insightful presentation covering a range of topics, including Hive introduction, dispelling misconceptions about Hive, exploring its features and origins, understanding the why behind Hive's existence, delving into its architecture and working principles, dissecting the data model employed in Hive, exploring different modes of operation, and weighing the advantages and disadvantages it brings. The presentation concludes with practical examples, demonstrating how to create tables in Hive, upload data, and execute queries within the Hadoop environment. Join us on a journey through the intricacies of Hive, unraveling its capabilities and applications in big data analytics
Report
Share
More Related Content
Similar to Unveiling Hive: A Comprehensive Exploration of Hive in Hadoop Ecosystem
Similar to Unveiling Hive: A Comprehensive Exploration of Hive in Hadoop Ecosystem (20)
Recently uploaded
Recently uploaded (20)
Unveiling Hive: A Comprehensive Exploration of Hive in Hadoop Ecosystem
- 1. Hive Presented by : Mohammad Mashhoood Syed
- 2. What is Hive? • Apache Hive is a data warehouse software built on top of Hadoop that facilitates reading, writing and managing large datasets residing in distributed storage using SQL. • Hive provides the necessary SQL abstraction so that SQL- like queries can be integrated with the underlying Java code without having to implement the queries in the low-level Java API • It allows structure to be projected onto data that is already in storage.
- 3. • It can create schemas/table definitions that point to data in Hadoop, turning unstructured data into structured data. • Helps to treat your data in Hadoop as Tables; which can be partitioned and bucketed.
- 4. Hive is not • A relational database • A design for OnLine Transaction Processing (OLTP) • A language for real-time queries and row-level updates
- 5. Features of hive • Hive is fast and scalable. • It provides SQL-like queries (i.e., HQL) that are implicitly transformed to MapReduce or Spark jobs. • It is capable of analyzing large datasets stored in HDFS. • It can operate on compressed data stored in the Hadoop ecosystem. • It supports user-defined functions (UDFs) where user can provide its functionality.
- 6. Hive Origination • Hive originated as an internal project in Facebook • Later it was adopted in Apache as an open source project • Facebook deals with massive amount of data (petabytes scale) and it needs to perform more than 75k ad-hoc queries on this massive amount of data
- 7. Why Hive? • Since the data is collected from multiple servers and is of diverse nature, any RDBMS system could not fit as probable solution • Map Reduce could be a natural choice, but it had its own limitations
- 8. Architecture
- 9. Working
- 10. 1. Execute Query: The Hive interface such as Command Line or Web UI sends query to Driver (any database driver such as JDBC, ODBC, etc.) to execute 2. Get Plan: The driver takes the help of query compiler that parses the query to check the syntax and query plan or the requirement of query 3. Get Metadata: The compiler sends metadata request to Metastore (any database). 4. Send Metadata : Metastore sends metadata as a response to the compiler. 5. Send Plan : The compiler checks the requirement and resends the plan to the driver. Up to here, the parsing and compiling of a query is complete. 6. Execute Plan: The driver sends the execute plan to the execution engine. 7.Execute Job: Internally, the process of execution job is a MapReduce job. The execution engine sends the job to JobTracker, which is in Name node and it assigns this job to TaskTracker, which is in Data node. Here, the query executes MapReduce job.
- 11. Data modeling Tables Partitions buckets Here tables are organized into partitions for grouping same type of data based on partition key Partitions divided further into buckets based on some other column Tables in hive are created the same way it is done in RDBMS
- 12. Different modes of Hive • Hive can operate in two modes depending on the size of data nodes in Hadoop. • These modes are : • Local mode • Map reduce mode
- 13. Local Mode • If the Hadoop installed under pseudo mode with having one data node we use Hive in this mode • If the data size is smaller in term of limited to single local machine, we can use this mode • Processing will be very fast on smaller data sets present in the local machine
- 14. Map Reduce mode • If Hadoop is having multiple data nodes and data is distributed across different node we use Hive in this mode • It will perform on large amount of data sets and query going to execute in parallel way • Processing of large data sets with better performance can be achieved through this mode
- 15. Advantages of hive • Keeps queries running fast • Takes very little time to write Hive query in comparison to MapReduce code • HiveQL is a declarative language like SQL • Multiple users can simultaneously query the data using Hive-QL. • Very easy to write query including joins in Hive • Simple to learn and use
- 16. Disadvantages of Hive • It's not designed for Online transaction processing (OLTP), it is only used for the Online Analytical Processing (OLAP). • Hive supports overwriting or apprehending data, but not updates and deletes. • Sub-queries are not supported, in Hive
- 17. Copying file from local system into Hadoop environment • Hdfs dfs –copyFromLocal (file path) destination path
- 18. Creating table
- 19. Give full file path
- 24. IMDB dataset
- 25. Number of movies per year select startyear,count(*) as count from movies where startyear > 2000 and startyear < 2022 group by startyear order by count;
- 26. Comedy movies • select primarytitle,startyear,runtimeminutes,genres from movies where array_contains(genres,"Comedy");
- 27. • select distinct titletype from movies;
- 28. Upcoming horror movies select * from movies where titletype = 'movie' and startyear > 2021 and array_contains(genres,"Horror");
- 29. Movies in 2021 with rating more than 9 select m.startyear,m.titletype,m.primarytitle,r.averagerating,m.genres from movies as m join rating as r on m.tconst = r.tconst where m.titletype = 'movie' and m.startyear = 2021 and r.averagerating > 9 ;
- 30. Action series with rating more than 9 select m.startyear,m.titletype,m.primarytitle,r.averagerating,m.genres from movies as m join rating as r on m.tconst = r.tconst where m.titletype = 'tvSeries' and r.averagerating > 9 and array_contains(genres,"Action");
- 31. •THANK YOU !