hbaseconasia2019 Recent work on HBase at Pinterest
•
0 likes•348 views
Lianghong Xu Track 3: Applications https://open.mi.com/conference/hbasecon-asia-2019 THE COMMUNITY EVENT FOR APACHE HBASE™ July 20th, 2019 - Sheraton Hotel, Beijing, China https://hbase.apache.org/hbaseconasia-2019/
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to hbaseconasia2019 Recent work on HBase at Pinterest
Similar to hbaseconasia2019 Recent work on HBase at Pinterest (20)
More from Michael Stack
More from Michael Stack (20)
Recently uploaded
Recently uploaded (13)
hbaseconasia2019 Recent work on HBase at Pinterest
- 2. Recent work on HBase at Pinterest Lianghong Xu Pinterest Software Engineer, Tech Lead
- 3. Introduction
- 4. HBase at Pinterest • Backend for many critical services • Graph database (Zen) • Generic KV store (UMS) • Around 50 HBase clusters • HBase 0.94 since 2013, HBase 1.2 since 2016 • Internal repo with ZSTD, CCSMAP, Bucket cache, etc.
- 5. Agenda • Omid: transaction layer for NoSQL database • Sparrow: Omid made scalable • Argus: database observer framework • Ixia: near-realtime HBase indexing
- 6. Agenda • Omid: transaction layer for NoSQL database • Sparrow: Omid made scalable • Argus: database observer framework • Ixia: near-realtime HBase indexing
- 7. NoSQL Embracing Transactions SQL NoSQL Relational Transactional Expressive Simple Fast Scalable
- 8. NoSQL Embracing Transactions SQL NoSQL Relational Transactional Expressive Simple Fast Scalable
- 9. NoSQL Embracing Transactions SQL NoSQL Relational Transactional Expressive Simple Fast Scalable
- 10. Apache Omid at Pinterest • Omid (Optimistically transaction Management In Datastores) • Transaction framework on top of KV stores with HBase support • Open-sourced by Yahoo! in 2016 • Powers next generation of Ads indexing at Pinterest
- 11. Apache Omid at Pinterest • Omid (Optimistically transaction Management In Datastores) • Transaction framework on top of KV stores with HBase support • Open-sourced by Yahoo! in 2016 • Powers next generation of Ads indexing at Pinterest • Pros: simple, reasonable performance, HA, pluggable backend with native HBase support • Cons: No SQL interface, limited isolation levels, requires MVCC support
- 12. Omid Architecture Client Transaction Manager (TM) begin/commit timestamp/commit status Data tables Commit table read/write check commit persist commit
- 13. Omid internals • Leverages Multi-version Concurrency Control (MVCC) support in HBase • Transaction ID (begin timestamp) in version, commit timestamp in shadow cell • OCC: lock-free implementation with central conflict detection mechanism Omid data and commit table
- 14. Agenda • Omid: transaction layer for NoSQL database • Sparrow: Omid made scalable • Argus: database observer framework • Ixia: near-realtime HBase indexing
- 15. Omid Architecture Client Transaction Manager (TM) begin/commit timestamp/commit status Data tables Commit table read/write check commit persist commit
- 16. Omid Scalability Problem Client Transaction Manager (TM) begin/commit timestamp/commit status Data tables Commit table read/write check commit persist commit Centralized batch commit to HBase
- 17. Omid Scalability Problem Client Transaction Manager (TM) begin/commit timestamp/commit status Data tables Commit table read/write check commit persist commit Centralized batch commit to HBase Single-threaded request/reply processor for serializability
- 18. Sparrow Architecture Client Transaction Manager (TM) begin/commit timestamp/commit status Data tables Commit table read/write check commit Single-threaded request/reply processor for serializability
- 19. Sparrow Architecture Client Transaction Manager (TM) begin/commit timestamp/commit status Data tables Commit table read/write check commit Single-threaded request/reply processor for serializability persist commit Distributed client-side commit
- 20. Sparrow Architecture Client Transaction Manager (TM) begin/commit timestamp/commit status Data tables Commit table read/write check commit persist commit Distributed client-side commit Parallel request processing
- 21. Sparrow: Omid made scalable Client Transaction Manager (TM) begin/commit timestamp/commit status Data tables Commit table read/write check commit persist commit Distributed client-side commit Parallel conflict detection persist commit Performance bottleneck
- 22. Sparrow techniques • Client-side commit • Client writes to commit table when there is no conflicts • Explicitly mark aborted txn in commit table (-1) • Reader may back off and abort concurrent writer in case of client failure or network partition • Avoid performance bottleneck on TM • Parallel request processing • Multi-threaded request processor with in-memory conflict map • beginTx no longer needs to wait until whole commit batch is written to HBase • Timestamp allocation still needs to be synchronized (with negligible overhead)
- 23. Sparrow vs. Omid beginTx P99: ~100X reduction commitTx P99: ~3X reduction
- 24. Agenda • Omid: transaction layer for NoSQL database • Sparrow: Omid made scalable • Argus: database observer framework • Ixia: near-realtime HBase indexing
- 25. Argus: Motivation and Problem Statement • Clients request a real-time notification feature similar to a database trigger • Incremental processing based on database changes • Notification cannot be missed - ”at least once” • Notification events could have different priorities and object types
- 27. Kafka-based Notification Pipeline Percolator (Google) • Special notification column • Observer threads periodically scan for changes • Heavy-weight distributed scan and locking
- 28. Kafka-based Notification Pipeline Percolator (Google) Argus • Special notification column • Observer threads periodically scan for changes • Heavy-weight distributed scan and locking • Async notification by tailing HBase WAL • Kafka for replayable DB change stream • Support different priorities and types • Lightweight, minimal impact on DB
- 29. Argus Architecture Client Argus Observer HBase Annotated requests Replication proxy WAL Notification events (Kafka) read/write • HBase annotation: extra metadata in HBase requests to be passed down into WAL • Replication Proxy: ”fake” regionservers with only replication RPC implemented
- 30. Argus Observers • Process notification events in parallel with user-defined handlers • Event dispatching, filtering, collapse, etc. • Notification Handlers can be chained
- 31. Argus Observers • Process notification events in parallel with user-defined handlers • Event dispatching, filtering, collapse, etc. • Notification Handlers can be chained Use case on Ads indexing: Batch processing (15 mins) -> incremental indexing (several seconds)
- 32. Agenda • Omid: transaction layer for NoSQL database • Sparrow: Omid made scalable • Argus: database observer framework • Ixia: near-realtime HBase indexing
- 33. Ixia: Motivation • Clients ask for secondary indexing support in HBase • Analytics queries on HBase columns (filtering, range, aggregation) • Why not SQL? • Index build could take a long time • Lack of horizontal scalability and tuning expertise
- 34. Ixia: Near-realtime Indexing with HBase + Muse (In-house Search Engine) • Inspired by Lily indexer (HBase + Solr) • Secondary indexes in Muse (written in C++, fast in-memory inverted/forward index) • Source-of-truth data in HBase • Index built asynchronously with HBase WAL through Kafka • Ixia query engine: Thrift-based query service with a SQL-like interface
- 35. Ixia: Architecture Client HBase Replication proxy WAL Indexer Index events (Kafka) write Muse index docs MuseQuery engine query search query DB retrieval Index manager Index schema
- 36. Ixia: Pros and Cons Pros Cons Minimal impact on write path Index and data stores scaled separately Efficient indexing & retrieval No strong index consistency
- 37. Ixia: status and future work • Batch indexing in prod, reducing indexing time by ~15X • Query engine serving full dark traffic, reducing query latency by up to 100X • Future work: • Realtime indexing into production • SQL support • Dynamic index backfilling
- 38. Thanks！