Cloudera Impala Internals
•Download as PPT, PDF•
15 likes•4,499 views
This presentation is about internals of Cloudera Impala open source database engine. it is done based on my study of their source code.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Cloudera Impala Internals
Similar to Cloudera Impala Internals (20)
Recently uploaded
Recently uploaded (20)
Cloudera Impala Internals
- 1. Impala internals By David Gruzman BigDataCraft.com
- 3. Impala by David Gruzman ►Impala is – Relational Query Engine – Open source – Massive parallel processing
- 4. Why do we care, about internals? ► SQL is declarative, no need for internals... ► In the same time, even small problems in engine operation require good understanding of its work principles to fix... ► It is hardly possible to optimize without understanding algorithms under the hood. ► It is hard to make decisions about engine suitability to future needs without knowing technical limitations.
- 5. Engine parts
- 6. How to understand engine? What it is doing? Main principle of operation Main building block Operation sequence Operation environment Efficiency Design decisions Materials Main problems and fixes
- 7. What it is doing Impala is Relation engine. It executes SQL queries. Data is append-able only. There is no “Update” or “Delete” statements.
- 8. Principle of operation Main differentiators are: Distribution of Query among nodes (MPP) LLVM and Code generation. Impala is compiler. Relay on HDFS Use external metadata – hive metastore. Parallel query capability (per node, per cluster).
- 9. Sequence of operation Query parsing – translate SQL to AST(Abstract syntax tree) Match objects to metadata Query planning – create physical execution plan. In case of MPP – divide plan into plan fragments for nodes. Distribute plan fragments to nodes Execute plan fragments.
- 10. Main building blocks Front End. This is Java code which implements a lot of logic with non-critical performance - database objects fe/src/main/java/com/cloudera/impala/analysis/ - execution plan parts : fe/src/main/java/com/cloudera/impala/planner/
- 11. BackEnd (Be) Backend is written on C++, and used mostly for performance critical parts. Specifically: - Execution of the plan fragments on nodes - Services implementation ImpalaD service StateStore Catalog Service
- 12. Services - ImpalaD This is “main” service of impala which runs on each node. It logically consists of the following sub-services of our interest. ImpalaService – service, used to execute query. Console, JDBC/ODBC connects here. ImpalaInternalService – service is used to coordinate work within the impala cluster. Example of usage – to coordinate the job of running query fragments on planned impala nodes. What is interesting for us? Each node can serve
- 13. Dual role of ImpalaD service Query coordinator Fragment executor
- 14. Services view Front End Impala Service Impala Internal Service
- 15. ImpalaService – main methods inherited from beeswax : ExecuteAndWait Fetch Explain Impala specific : ResetCatalog GetRuntimeProfile
- 17. Services - StateStore In many clusters we have to solve “cluster synchronization” problem on some or other way. In impala it is solved by StateStore – published/subscriber service, similar to Zookeeper. Why Zookeeper is not used? It speaks with its clients in terms of topics. Clients can subscribe to different topics. So to find “endpoints” - look in the sources for the usage of “StatestoreSubscriber”
- 18. StateStore – main topics IMPALA_MEMBERSHIP_TOPIC – updates about attached and detached nodes. IMPALA_CATALOG_TOPIC – updates about metadata changes. IMPALA_REQUEST_QUEUE_TOPIC – updates in the queue of waiting queries.
- 19. Admission control There is module called AdmissionController. Via topic impala-request-queue it is know about queries currently running and their basic statistics like memory and CPU consumption. Based on this info it can decide to: -run query -queue query -reject query
- 20. Catalog Service It caches in Java code metadata from hive metastore: /fe/src/main/java/com/cloudera/impala/catalog/ It is important since Hive's native partition pruning is slow especially with large number of partitions. It use C++ code be/src/catalog/ To relay changes (delta's) to other nodes via StateStore.
- 21. Differance with hive Catalog Service store in memory and operate on metadata, leaving MetaStore for persistance only. Technically it mean that disconnection from MetaStore is not that complicated.
- 22. ImpalaInternalService - details This is place where the real heavy lifting takes place. Before diving in, what we want to understand here: Threading model File System interface Predicate pushdown Resource management
- 23. Threading model DiskIoMgr schedules access of all readers to all disks. It should include predicates. It can give optimal concurrency. Sounds coherent to the Intel TBB / Java Executor service approach: give me small tasks and I will schedule them. The rest of operations – like Joins, Group By looks like single threaded in current version. IMHO – sort joins and group by are better for concurrency.
- 24. File System interface Impala is working via LibHDFS – so HDFS (not DFS) is hard coded. Impala required and checked that short circuit is enabled. During planning phase names of the block files to be scanned are determined.
- 25. Main “database” algorithm It is interesting to see, how main operations are implemented, what options do we have: Group By, Order By (Sort), Join
- 26. Join Join is probably most powerful and performance critical part of any analytical RDBMS. Impala implements BroadCastJoin and GraceHashJoin.(be/src/exec/partitioned-hash-join- node.h). Both are kinds of Hash Join. Basic idea of GraceHashJoin is to partition data, and load in memory corresponding partitions of the tables for the join.
- 27. DiskMemory Part 2 Part 3 Part 4Part 1 Part 5 Part 2 Part 3 Part 4Part 1 Part 5 Part 2 Part 3 Part 4Part 1 Part 5Part 3 Part 4 Part 5 In-memory hash join DiskMemory Part 3 Part 4 Part 3 Part 4 Part 5 Part 5
- 28. BroadCast join Just send small table to all nodes and join with big one. It is very similar to Map Side join in Hive. Selection of join algorithm can be hinted.
- 29. Group by There are two main approaches – using dictionary or sorting. Aggregation can be subject to memory problems with too many groups. Impala is using Partitioned Hash join which can spill to disk using BufferedBlockManager. It is somewhat analogous to join implementation.
- 30. User defined functions Impala supports two kinds of UDF / UDAF - Native, written in C/C++ - Hive's UDF written in java.
- 31. Caching Impala does not cache data by itself. It delegates it to the new HDFS caching capability. In a nutshell – HDFS is capable to keep given directory in memory. Zero copy access via MMAP is implemented. Why it is better then buffer cache? Less task switching No CRC Check
- 32. Spill to Disk In order to be reliable, especially in face of Data Skews, some sort of spilling data to disk is needed. Impala approach this problem with introduction of BufferedBlockMgr It implements mechanism somewhat similar to virtual memory – pin, unpin blocks, persist them. It can use many disks to distribute load. It is used in all places where memory can be not sufficient
- 33. Why not Virtual Memory? Some databases offload all buffer management to the OS Virtual Memory. Most popular example: MongoDB. Impala create BufferedBlockManager per PlanFragment. It gives control how much memory consumed by single query on given node. We can summarize answer as : better resource management.
- 34. BufferedBlockMgr usage Partitioned join Sorting Buffered Tuple Stream Partitioned aggregation
- 35. Memory Management Impala BE has its own MemPool class for memory allocation. It is used across the board by runtime primitives and plan nodes.
- 36. Why own Runtime? Impala has implemented own runtime – memory management, virtual memory? IMHO Existing runtime (both Posix, and C++ runtime) are not multi-tenant. It is hard to track and limit resource usage by different requests in the same process. To solve this problem Impala has its own runtime with tracking and limiting capabilities.
- 37. YARN integration When Impala run as part of the Hadoop stack resource sharing is important question... Two main options are - Just divide resources between Impala and Yarn using cgroups. - Use YARN for the resource management.
- 38. Yarn Impala Impedance YARN is built to schedule batch processing. Impala is aimed to sub-second queries. Running application master per query does not sounds “low latency”. Requesting resources “as execution go” does not suit pipeline execution of query fragments.
- 39. L..LAMA ?
- 40. LLAMA Low Latency Application Master Or Long Living Application Master It enable low latency requests by living longer – for a whole application lifetime.
- 41. How LLAMA works 1. There is single LLAMA daemon to broker resources between Impala and YARN 2. Impala ask for all resources at once - “gang scheduling” 3. LLAMA cache resources before return them to YARN.
- 42. Important point Impala is capable of: - Run real time queries In YARN environment - Ask for more resources (especially memory) when needed. Main drawbacks: Impala implements own resource management among concurrent queries, thus partially duplicating YARN functionality. Possible deadlocks between two YARN applications.
- 44. What is source of similarity With all the difference, they solve similar problem: How to survive in Africa... O, sorry, How to run and coordinate number of tasks in the cluster.
- 45. Hadoop parallels QueryPlanner – Developer or Hive. Somebody who create job. Coordinator, ImpalaServer – Job Tracker PlanFragment – Task. (map or reduce) ImpalaInternalService – TaskTracker RequestPoolService+Scheduler+AdmissionContr oller = Hadoop job Scheduler. StateStore – Zookeeper.
- 46. ImpalaToGo While being a perfect product Impala is chained to the hadoop stack - HDFS - Management
- 47. Why it is a the problem? HDFS is perfect to store vast amounts of data. HDFS is built from large inexpensive SATA drives. For the interactive analytics we want fast storage. We can not afford FLASH drives for whole big data.
- 48. What is solution We can create another hadoop cluster on flash storage. Minus – another namenode to manage, replication will waste space. If replication factor is one – any problems should be manually repaired.
- 49. Cache Layer in place of DFS HDFS/Hadoop cluster ImpalaToGo cluster Data caching (LRU) Auto load
- 50. Elasticity Having cache layer in place of distributed file system it is much easier to resize cluster. ImpalaToGo is used consistent hashing for its data placement – to minimize impact on resize.
- 51. Who we are? Group of like minded developers, working on making Impala even greater.
- 52. Thank you!!! Please ask questions!