Apache Iceberg - A Table Format for Hige Analytic Datasets
•
7 likes•6,941 views
Data Orchestration Summit www.alluxio.io/data-orchestration-summit-2019 November 7, 2019 Apache Iceberg - A Table Format for Hige Analytic Datasets Speaker: Ryan Blue, Netflix For more Alluxio events: https://www.alluxio.io/events/
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Apache Iceberg - A Table Format for Hige Analytic Datasets
Similar to Apache Iceberg - A Table Format for Hige Analytic Datasets (20)
More from Alluxio, Inc.
More from Alluxio, Inc. (20)
Recently uploaded
Recently uploaded (20)
Apache Iceberg - A Table Format for Hige Analytic Datasets
- 1. Apache Iceberg Ryan Blue 2019 Big Data Orchestration Summit
- 3. ● Smarter processing engines ○ CBO, better join implementations ○ Result set caching, materialized views ● Reduce manual data maintenance ○ Data librarian services ○ Declarative instead of imperative 5-year Challenges
- 4. ● Unsafe operations are everywhere ○ Writing to multiple partitions ○ Renaming a column ● Interaction with object stores causes major headaches ○ Eventual consistency to performance problems ○ Output committers can’t fix it ● Endless scale challenges Problem Whack-a-mole
- 6. Iceberg is a scalable format for tables with a lot of best practices built in.
- 7. A format? We already have Parquet, Avro and ORC . . .
- 9. ● File formats help you modify or skip data in a single file ● Table formats do the same thing for a collection of files ● To demonstrate this, consider Hive tables . . . A table format
- 10. ● Key idea: organize data in a directory tree date=20180513/ |- hour=18/ | |- ... |- hour=19/ | |- part-000.parquet | |- ... | |- part-031.parquet |- hour=20/ | |- ... Hive Tables
- 11. ● Filter: WHERE date = '20180513' AND hour = 19 date=20180513/ |- hour=18/ | |- ... |- hour=19/ | |- part-000.parquet | |- ... | |- part-031.parquet |- hour=20/ | |- ... Hive Tables
- 12. ● Problem: too much directory listing for large tables ● Solution: use HMS to track partitions date=20180513/hour=19 -> hdfs:/.../date=20180513/hour=19 date=20180513/hour=20 -> hdfs:/.../date=20180513/hour=20 ● The file system still tracks the files in each partition . . . Hive Metastore
- 13. ● State is kept in both the metastore and in a file system ● Changes are not atomic without locking ● Requires directory listing ○ O(n) listing calls, n = # matching partitions ○ Eventual consistency breaks correctness Hive Tables: Problems
- 14. ● Everything supports Hive tables* ○ Engines: Hive, Spark, Presto, Flink, Pig ○ Tools: Hudi, NiFi, Flume, Sqoop ● Simplicity and ubiquity have made Hive tables indispensable ● The whole ecosystem uses the same at-rest data! Hive Tables: Benefits
- 15. Iceberg
- 16. ● An open spec and community for at-rest data interchange ○ Maintain a clear spec for the format ○ Design for multiple implementations across languages ○ Support needs across projects to avoid fragmentation Iceberg’s Goals
- 17. ● Improve scale and reliability ○ Work on a single node, scale to a cluster ○ All changes are atomic, with serializable isolation ○ Native support for cloud object stores ○ Support many concurrent writers Iceberg’s Goals
- 18. ● Fix persistent usability problems ○ In-place evolution for schema and layout (no side-effects) ○ Hide partitioning: insulate queries from physical layout ○ Support time-travel, rollback, and metadata inspection ○ Configure tables, not jobs ● Tables should have no unpleasant surprises Iceberg’s Goals
- 19. ● Key idea: track all files in a table over time ○ A snapshot is a complete list of files in a table ○ Each write produces and commits a new snapshot Iceberg’s Design S1 S2 S3 ...
- 20. S1 S2 S3 ... R W ● Readers use the current snapshot ● Writers optimistically create new snapshots, then commit Iceberg’s Design
- 21. In reality, it’s a bit more complicated.
- 22. ● All changes are atomic ● No expensive (or inconsistent) file system operations ● Snapshots are indexed for scan planning on a single node ● CBO metrics are reliable ● Versions for incremental updates and materialized views Iceberg Design Benefits
- 24. ● Production tables: tens of petabytes, millions of partitions ○ Scan planning fits on a single node ○ Advanced filtering enables more use cases ○ Overall performance is better ● Low latency queries are faster for large tables Scale
- 25. ● Production Flink pipeline writing in 3 AWS regions ● Lift service moving data into a single region ● Merge service compacting small files Concurrency
- 26. ● Rollback is popular ● Metadata tables ○ Track down the version a job read ○ Find the process that wrote a bad version Usability
- 27. ● Spark vectorization for faster bulk reads ○ Presto vectorization already done ● Row-level delete encodings ○ MERGE INTO ○ ID equality predicates Future Work