Etu Solution Day 2014 Track-D: 掌握Impala和Spark
•
3 likes•691 views
ESD (Etu Solution Day) 2014, Track D Slide. A presentation about Impala and Spark introduction and hands-on.
Report
Share
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
More Related Content
What's hot
What's hot (20)
Similar to Etu Solution Day 2014 Track-D: 掌握Impala和Spark
Similar to Etu Solution Day 2014 Track-D: 掌握Impala和Spark (20)
More from James Chen
Recently uploaded
Recently uploaded (20)
Etu Solution Day 2014 Track-D: 掌握Impala和Spark
- 1. 掌握Impala和Spark Real-time Big Data即時應用架構研習 Etu首席顧問 陳昭宇 Oct 8, 2014
- 2. Workshop Goal Let’s talk about the 3Vs in Big Data. Hadoop is good for Volume and Variety But… How about Velocity ??? This is why we are sitting here ….
- 3. Target Audience • CTO • Architect • Software/Application Developer • IT
- 4. Background Knowledge • Linux operation system • Basic Hadoop ecosystem knowledge • Basic knowledge of SQL • Java or Python programming experience
- 5. Terminology • Hadoop: Open source big data platform • HDFS: Hadoop Distributed Filesystem • MapReduce: Parallel computing framework on top of HDFS • HBase: NoSQL database on top of Hadoop • Impala: MPP SQL query engine on top of Hadoop • Spark: In-memory cluster computing engine • Hive: SQL to MapReduce translator • Hive Metastore: Database that stores table schema • Hive QL: A SQL subset
- 6. Agenda • What is Hadoop and what’s wrong with Hadoop in real-time? • What is Impala? • Hands-on Impala • What is Spark? • Hands-on Spark • Spark and Impala work together • Q & A
- 7. What is Hadoop ? Apache Hadoop is an open source platform for data storage and processing that is : Distributed Fault tolerant Scalable CORE HADOOP SYSTEM COMPONENTS HDFS HDFS A fault-tolerant, scalable clustered A fault-tolerant, scalable clustered storage storage MapReduce MapReduce A distributed computing framework A distributed computing framework • Ask questions across structured and unstructured data • Schema-less • Scale-out architecture divides workloads across nodes. • Flexible file system eliminates ETL bottlenecks. Flexible for storing and mining any type of data Processing Complex Big Data Scales Economically • Deploy on commodity hardware • Open sourced platform
- 8. Limitations of MapReduce • Batch oriented • High latency • Doesn’t fit all cases • Only for developers
- 9. Pig and Hive • MR is hard and only for developers • High level abstraction for converting declarative syntax to MR – SQL – Hive – Dataflow language - Pig • Build on top of MapReduce
- 10. Goals • General-purpose SQL engine: – Works for both analytics and transactional/single-row workloads. – Supports queries that take from milliseconds to hours. • Runs directly within Hadoop and: – Reads widely used Hadoop file formats. – Runs on same nodes that run Hadoop processes. • High performance: – C++ instead of Java – Runtime code generation – Completely new execution engine – No MapReduce
- 11. What is Impala • General-purpose SQL engine • Real-time queries in Apache Hadoop • Beta version released since Oct. 2012 • GA since Apr. 2013 • Apache licensed • Latest release v1.4.2
- 12. Impala Overview • Distributed service in cluster: One impala daemon on each data node • No SPOF • User submits query via ODBC/JDBC, CLI, or HUE to any of the daemons. • Query is distributed to all nodes with data locality. • Uses Hive’s metadata interfaces and connects to Hive metastore. • Supported file formats: – Uncompressed/lzo-compressed text files – Sequence files and RCFile with snappy/gzip, Avro – Parquet columnar format
- 13. Impala’s SQL • High compatibility with HiveQL • SQL support: – Essential SQL-92, minus correlated subqueries – INSERT INTO … SELECT … – Only equi-joins; no non-equi-joins, no cross products – Order By requires Limit (not required after 1.4.2) – Limited DDL support – SQL-style authorization via Apache Sentry – UDFs and UDAFs are supported
- 14. Impala’s SQL limitations • No file formats, SerDes • No beyond SQL (buckets, samples, transforms, arrays, structs, maps, xpath, json) • Broadcast joins and partitioned hash joins supported (Smaller tables have to fit in the aggregate memory of all executing nodes)
- 15. Work with HBase • Functionality highlights: – Support for SELECT, INSERT INTO…SELECT…, and INSERT INTO … VALUES (…) – Predicates on rowkey columns are mapped into start/stop rows – Predicates on other columns are mapped into SingleColumnValueFilters • BUT mapping of HBase tables and metastore table patterned after Hive: – All data is stored as scalars and in ASCII. – The rowkey needs to be mapped into a single string column.
- 16. HBase in Roadmap • Full support for UPDATE and DELETE. • Storage of structured data to minimize storage and access overhead. • Composite rowkey encoding mapped into an arbitrary number of table columns.
- 17. Impala’s Architecture Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase Hive Metastore HDFS NN Statestore SQL Client ODBC 1. Request arrives via ODBC/JDBC/Beeswax/ Shell.
- 18. Impala’s Architecture Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase Hive Metastore HDFS NN Statestore SQL Client ODBC 2. Planner turns request into collections of plan fragments. 3. Coordinator initiates execution on impalad(s) local to data.
- 19. Impala’s Architecture Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase Hive Metastore HDFS NN Statestore SQL Client ODBC 5. Query results are streamed back to client. 4. Intermediate results are streamed between impalad(s).
- 20. Metadata Handling • Impala metadata – Hive’s metastore: Logical metadata (table definitions, columns, CREATE TABLE parameters) – HDFS NameNode: Directory contents and block replica locations – HDFS DataNode: Block replias’ volume ids • Caches metadata: No synchronous metastore API calls during query execution • Impala instances read metadata from metastore at startup. • Catalog Service relays metadata when you run DDL or update metadata on one of the impalads.
- 21. Metadata Handling – Cont. • REFRESH [<tbl>]: Reloads metadata on all impalads (if you added new files via Hive) • INVALIDATE METADATA: Reloads metadata for all tables
- 22. Comparing Impala to Dremel • What is Dremel? – Columnar storage for data with nested structures – Distributed scalable aggregation on top of that • Columnar storage in Hadoop: Parquet – Store data in appropriate native/binary types – Can also store nested structures similar to Dremel’s ColumnIO • Distributed aggregation: Impala • Impala plus Parquet: A superset of the published version of Dremel (does not support joins)
- 23. Comparing Impala to Hive • Hive: MapReduce as an execution engine – High latency, low throughput queries – Fault-tolerance model based on MapReduce’s on-disk check pointing; materializes all intermediate results – Java runtime allows for easy late-binding of functionality: file formats and UDFs – Extensive layering imposes high runtime overhead • Impala: – Direct, process-to-process data exchange – No fault tolerance – An execution engine designed for low runtime overhead
- 24. Impala and Hive Shares Everything Client-Facing •Metadata (table definitions) •ODBC/JDBC drivers •SQL syntax (Hive SQL) •Flexible file formats •Machine pool •GUI Resource Management Data Store But Built for Different Purposes •Hive: Runs on MapReduce and ideal for batch processing •Impala: Native MPP query engine ideal for interactive SQL Data Ingestion HDFS HBase TEXT, RCFILE, PARQUET,AVRO, ETC. RECORDS Hive SQL Syntax MapReduce Compute framework Impala SQL syntax + compute framework
- 25. Typical Use Cases • Data Warehouse Offload • Ad-hoc Analytics • Provide SQL interoperability to HBase
- 26. Hands-on Impala • Query a file on HDFS with Impala • Query a table on HBase with Impala
- 27. What is Spark? • MapReduce Review… • Apache Spark… • How Spark Works… • Fault Tolerance and Performance… • Examples… • Spark & More…
- 28. MapReduce: Good The Good: •Built in fault tolerance •Optimized IO path •Scalable •Developer focuses on Map/Reduce, not infrastructure •Simple? API
- 29. MapReduce: Bad The Bad: •Optimized for disk IO – Does not leverage memory – Iterative algorithms go through disk IO path again and again •Primitive API – Developers have to build on a very simple abstraction – Key/Value in/out – Even basic things like join require extensive code •A common result is many files require to be combined appropriately
- 30. Apache Spark • Originally developed in 2009 in UC Berkeley’s AMP Lab. • Fully open sourced in 2010 – now at Apache Software Foundation.
- 31. Spark: Easy and Fast Big Data • Easy to Develop – Rich APIs in Java, Scala, Python – Interactive Shell • 2-5x less code • Fast to Run – General execution graph – In-memory store
- 32. How Spark Works – SparkContext Cluster Master Spark Worker Spark Worker Executer Cache Executer Task Task Cache Data Node Data Node HDFS Task Task SparkContext sc=new SparkContext Rdd=sc.textfile(“hdfs://..”) Rdd.filter(…) Rdd.cache(…) Rdd.count(…) Rdd.map sc=new SparkContext Rdd=sc.textfile(“hdfs://..”) Rdd.filter(…) Rdd.cache(…) Rdd.count(…) Rdd.map
- 33. How Spark Works – RDD RDD (Resilient Distributed Dataset) sc=new SparkContext Rdd=sc.textfile(“hdfs://..” ) Rdd.filter(…) Rdd.cache(…) Rdd.count(…) Rdd.map sc=new SparkContext Rdd=sc.textfile(“hdfs://..” ) Rdd.filter(…) Rdd.cache(…) Rdd.count(…) Rdd.map Storage Types: MEMORY_ONLY, MEMORY_AND_DISK DISK_ONLY, • Fault Tolerant • Controlled • Fault Tolerant • Controlled partitioning to optimize data placement partitioning to optimize data placement • Manipulated by • Manipulated by using a rich set of operators using a rich set of operators • Partitions of Data • Dependency between partitions
- 34. RDD • Stands for Resilient Distributed Datasets • Spark revolves around RDDs • Fault-tolerant read only collection of elements that can be operated on in parallels • Cached in memory Reference: http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spar k.pdf
- 35. RDD • Read-only, partitioned collection of records DD11 DD22 DD33 3 partitions • Supports only coarse-grained operations – e.g. map and group-by transformation, reduce actions DD11 DD22 DD33 DD11 DD22 DD33 DD11 DD22 DD33 DD11 DD22 DD33 DD11 DD22 DD33 Value
- 36. RDD Operations
- 37. RDD Operations - Expressive • Transformations – Creation of a new RDD dataset from an existing: • map, filter, distinct, union, sample, groupByKey, join, reduce, etc… • Actions – Returns a value after running a computation: • Collect, count, first, takeSample, foreach, etc… • Reference – http://spark.apache.org/docs/latest/scala-programming-guide.html#rdd-operations
- 38. Word Count on Spark sparkContext.textFile(“hdfs://…”) RDD[String] textFile
- 39. Word Count on Spark sparkContext.textFile(“hdfs://…”) .map(line => line.split(“s”)) RDD[String] RDD[List[String]] textFile map
- 40. Word Count on Spark sparkContext.textFile(“hdfs://…”) .map(line => line.split(“s”)) .map(word => (word, 1)) RDD[String] RDD[List[String]] RDD[(String, Int)] textFile map map
- 41. Word Count on Spark sparkContext.textFile(“hdfs://…”) .map(line => line.split(“s”)) .map(word => (word, 1)) .reduceByKey((a, b) => a+b) RDD[String] RDD[List[String]] RDD[(String, Int)] RDD[(String, Int)] textFile map map reduceByKey map
- 42. Word Count on Spark sparkContext.textFile(“hdfs://…”) .map(line => line.split(“s”)) .map(word => (word, 1)) .reduceByKey((a, b) => a+b) .collect() RDD[String] RDD[List[String]] RDD[(String, Int)] RDD[(String, Int)] Array[(String, Int)] textFile map map reduceByKey map collect
- 43. Actions • Parallel Operations map reduce sample filter count take grougBy fold first sort reduceByKey partitionBy union groupByKey mapWith join cogroup pipe leftOuterJoin cross save rightOuterJoin zip ….
- 44. Stages textFile map map reduceByKey collect Stage 1 Stage 2 DAG (Directed Acyclic Graph) Each stage is executed as Stage 1 Stage 2 a series of Task (one Task for each partition).
- 45. Tasks Task is the fundamental unit of execution in Spark Fetch Input Execute Task Write Output HDFS /RDD HDFS/RDD/Intermediate shuffle output Core 1 Fetch Input Execute Task Write Output Fetch Input Execute Task Write Output Fetch Input Execute Task Write Output Core 2
- 46. Spark Summary • SparkContext • Resilient Distributed Dataset • Parallel Operations • Shared Variables – Broadcast Variables – read-only – Accumulators
- 47. Comparison MapReduce Impala Spark Storage HDFS HDFS/HBase HDFS Scheduler MapReduce Job Query Plan Computation Graph I/O Disk In-memory with cache In-memory, cache and shared data Fault Tolerance Duplication and Disk I/O No Fault Tolerance Hash partition and auto reconstruction Iterative Bad Bad Good Shared data No No Yes Streaming No No Yes
- 48. Hands-on Spark • Spark Shell • Word Count
- 49. Spark Streaming • Takes the concept of RDDs and extends it to Dstreams – Fault-tolerant like RDDs – Transformable like RDDs • Adds new “rolling window” operations – Rolling averages, etc.. • But keeps everything else! – Regular Spark code works in Spark Streaming – Can still access HDFS data, etc. • Example use cases: – “On-the-fly” ETL as data is ingested into Hadoop/HDFS – Detecting anomalous behavior and trigger alerts – Continuous reporting of summary metrics for incoming data
- 51. Micro-batching for on the fly ETL
- 53. Spark SQL • Spark SQL is one of Spark’s components. – Executes SQL on Spark – Builds SchemaRDD • Optimizes execution plan • Uses existing Hive metastores, SerDes, and UDFs.
- 54. Unified Data Access • Ability to load and query data from a variety of sources. • SchemaRDDs provides a single interface that efficiently works with structured data, including Hive tables, parquet files, and JSON. sqlCtx.jsonFile("s3n://...") .registerAsTable("json") schema_rdd = sqlCtx.sql(""" SELECT * FROM hiveTable JOIN json ...""") Query and join different data sources
- 55. Hands-on Spark • Parse/transform log on the fly with Spark-Streaming • Aggregate with Spark SQL (Top N) • Output from Spark to HDFS
- 56. Spark & Impala work together Data Strea m Data Strea m Spark- Streaming Spark Impala DN RS Data Strea m Spark- Streaming Spark Impala DN RS Data Strea m Spark- Streaming Spark DN RS Impala … Data Strea m Data Strea m Data Stream -Click Steam -Machine Data -Log -Network Traffic -Etc.. On-the-fly Processing -ETL, transformation, filter -Pattern Matching & Alert Real-time Analytics -Machine Learning (Rec. Cluster..) -Iterative Algorithms Near Real-time Query - Ad-hoc query - Reporting Long term data store -Batch process -Offline analytics -Historical Mining
- 57. Etu 讓 Hadoop 更容易 全自動、高效能、易管理的軟體式一體機 空機自動部署 效能最佳化 全叢集管理 唯一在地 Hadoop 專業服務 主流 X86 商用伺服器
- 58. ESA Software Stack Cloudera Manager Etu Manager 安全管理 效能最佳化 組態同步網路管理監控告警套件管理 CentOS作業系統 (64 bits) HA 管理 Rack Awareness Etu 加值模組
- 59. Etu在Hadoop生態系的定位與價值 Etu Services 人才 招聘 團隊 建立 程式開發 數據 架構 挖掘 設計 部署、調校 易 Etu 易 運維、管理 應用 平台 搶 佔 市場 Etu Professional Services 核心 價 值 資源 調配 屏蔽 Hadoop 平台 部署與運維的複雜度 易 • 快速推出服務，搶佔市場 • 應用、數據才是企業核心價值 • 依核心價值調配資源，建立競爭優 勢 Software Appliance EEttuu SSuuppppoorrtt Etu Professional Services EEttuu CCoonnssuullttiinngg EEttuu TTrraaiinniinngg 易
- 60. Question and Discussion Thank you 318, Rueiguang Rd., Taipei 114, Taiwan T: +886 2 7720 1888 F: +886 2 8798 6069 www.etusolution.com