Cosco: An Efficient Facebook-Scale Shuffle Service
•
5 likes•3,023 views
Cosco is an efficient shuffle-as-a-service that powers Spark (and Hive) jobs at Facebook warehouse scale. It is implemented as a scalable, reliable and maintainable distributed system. Cosco is based on the idea of partial in-memory aggregation across a shared pool of distributed memory. This provides vastly improved efficiency in disk usage compared to Spark's built-in shuffle. Long term, we believe the Cosco architecture will be key to efficiently supporting jobs at ever larger scale. In this talk we'll take a deep dive into the Cosco architecture and describe how it's deployed at Facebook. We will then describe how it's integrated to run shuffle for Spark, and contrast it with Spark's built-in sort-based shuffle mechanism and SOS (presented at Spark+AI Summit 2018).
Report
Share
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
More Related Content
What's hot
What's hot (20)
Similar to Cosco: An Efficient Facebook-Scale Shuffle Service
Similar to Cosco: An Efficient Facebook-Scale Shuffle Service (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Cosco: An Efficient Facebook-Scale Shuffle Service
- 2. Cosco: an efficient facebook-scale shuffle service Brian Cho & Dmitry Borovsky Spark + AI Summit 2019
- 3. Disaggregated compute and storage • Advantages • Server types optimized for compute or storage • Separate capacity management and configuration • Different hardware cycles • Compute clusters • CPU, RAM, no disks for data • Spark executors • Storage clusters • Spindle disks • DFS (Warm Storage) • Permanent data: size dominant, uses less IO • Temporary data: IO dominant, uses less space
- 4. Spindle disk storage • A single spindle is used to read/write data on the drive • Small IO sizes cause low throughput as seek times dominate . 64 KiB, 140s 128 KiB, 73s 256 KiB, 39 s 1 MiB, 14s 4 MiB, 8s 0 5 10 15 20 0 50 100 150 Readrequest size(MiB) Time (s)
- 5. Spindle disk storage • A single spindle is used to read/write data on the drive • Small IO sizes cause low throughput as seek times dominate • Drive sizes increase over time • Must increase IO size to maintain the same throughput per TB 7 TB drive 1 MiB IO size 15 TB drive 7 MiB IO size 0 5 10 15 0 5 10 15 20 AvgIOsize(MiB) HDD capacity (TiB) 10 MiBs/TiB 64 KiB, 140s 128 KiB, 73s 256 KiB, 39 s 1 MiB, 14s 4 MiB, 8s 0 5 10 15 20 0 50 100 150 Readrequest size(MiB) Time (s)
- 6. Spindle disk storage • A single spindle is used to read/write data on the drive • Small IO sizes cause low throughput as seek times dominate • Drive sizes increase over time • Must increase IO size to maintain the same throughput per TB, or • Read/write less data to reduce throughput demand 0 5 10 15 0 5 10 15 20 AvgIOsize(MiB) HDD capacity (TiB) 10 MiBs/TiB 8 MiBs/TiB 6 MiBs/TiB 64 KiB, 140s 128 KiB, 73s 256 KiB, 39 s 1 MiB, 14s 4 MiB, 8s 0 5 10 15 20 0 50 100 150 Readrequest size(MiB) Time (s)
- 7. Spindle disk storage: key metrics ØDisk service time ØAverage IO size ØWrite amplification • A single spindle is used to read/write data on the drive • Small IO sizes cause low throughput as seek times dominate • Drive sizes increase over time • Must increase IO size to maintain the same throughput, or • Read/write less data to reduce throughput demand
- 8. Spark shuffle recap Map 0 Map 1 Map m Mappers Map Output Files Reducers Partition Reduce 0 Reduce 1 Reduce r
- 9. Spark shuffle recap Map 0 Map 1 Map m Mappers Map Output Files Reducers Reduce 0 Reduce 1 Reduce r Partition
- 10. Spark shuffle recap Map 0 Map 1 Map m Mappers Map Output Files Reducers Reduce 0 Reduce 1 Reduce r Sort by key Iterator Iterator Iterator Partition
- 11. Spark shuffle recap: Write amplification Map 0 Map 1 Map m Mappers Map Output Files Reducers Sort by key Iterator Iterator Iterator Partition Reduce 0 Reduce 1 Reduce r Write amplification is 3x
- 12. Spark shuffle recap: Small IOs problem Map 0 Map 1 Map m Mappers Map Output Files Reducers Reduce 0 Reduce 1 Reduce r M x R Avg IO size is 200 KiB Sort by key Iterator Iterator Iterator Partition
- 13. Spark shuffle recap: SOS Map 0 Map 1 Map m Mappers Map Output Files Reducers Reduce 0 Reduce 1 Reduce r Sort by key Iterator Iterator Iterator SOS: merge map outputs 10-way merge increases Avg IO size to 2 MiB Partition
- 14. Spark shuffle using Cosco • Mappers share a write-ahead buffer per reduce partition • Reducers can read the written data sequentially • Solves the small IOs problem • Sequential reads: Avg IO size 200 KiB à 2.5 MiB • Solves the write amplification problem • Avoiding spills: Write amplification 3x à 1.2x
- 15. Results / Current status • Hive • Rolled out to 90%+ of Hive workloads, in production for 1+ year • 3.2x more efficient disk service time • Spark • Analysis shows potential 3.5x more efficient disk service time • Analysis shows CPU neutral • Integration is complete, rollout planned during next few months
- 16. Cosco deep dive Dmitry Borovsky Spark + AI Summit 2019
- 17. Problem • Shuffle exchange on spinning disks (disaggregated compute and storage) • Single shuffle exchange scale: PiBs size, 100Ks of mappers, 10Ks reducers • Write amplification is ~3x (1PiB shuffle does 3PiB writes to disk) • Small Average IO size: ~200KiB (at least MxR reads) • IO is spiky (all readers may start at the same time and do MxR reads) • Cosco is shared between users
- 18. Sailfish: a framework for large scale data processing SoCC '12 Proceedings of the Third ACM Symposium on Cloud Computing, Article No. 4, San Jose, California — October 14 - 17, 2012 Source code: https://code.google.com/archive/p/sailfish/
- 19. Write-ahead buffers Cosco Shuffle Services (thousands) DFS Cosco Shuffle Services (thousands) Mapper 0 Mapper 1 Mapper 2 Reducer 0 Reducer 1 Partition 0 (file 1 buffer) File 0 File 1 File 2 File 0 File 1 network Partition 0 (file 2 buffer) Spark process (cosco client) Dependency Cosco process (shared between apps) Sorts (if needed)
- 20. Exactly once delivery DFS Cosco Shuffle Services (thousands) Mapper 0 Reducer 0 File 0 File 1 File 2 Partition 0 (file 2 buffer) Spark process (cosco client) Cosco process (shared between apps) Dependency Data Ack Partition 0’ (file 3 buffer) Failover File 3
- 21. Exactly once delivery DFS Cosco Shuffle Services (thousands) Mapper 0 Reducer 0 File 0 File 1 File 2 Partition 0 (file 2 buffer) Spark process (cosco client) Cosco process (shared between apps) Dependency Data Ack Partition 0’ (file 3 buffer) Failover 2 1 File 3
- 22. At least once delivery and deduplication DFS Cosco Shuffle Services (thousands) Mapper 0 Reducer 0 File 0 File 1 File 2 Partition 0 (file 2 buffer) Spark process (cosco client) Cosco process (shared between apps) Dependency Data Ack Partition 0’ (file 3 buffer) Failover 2 1 Adds row_id and mapper_id to each row De-duplicates resends non-acked data File 3
- 23. Replication DFS Mapper 0 Reducer 0 File 0 File 1 File 2 Partition 0 (file 2 buffer) Spark process (cosco client) Cosco process (shared between apps) Dependency Data Partition 0’ (file 2’ buffer) Data Ack Ack
- 24. Replication DFS Reducer 0 File 0 File 1 Spark process (cosco client) Cosco process (shared between apps) Dependency File 2’ File 2 Mapper 0 Partition 0 (file 2 buffer) Data Partition 0’ (file 2’ buffer) Data Ack Ack
- 25. // Mapper writer = new CoscoWriter( shuffleId: String, mapper: long); writer.collect( partition: int, row: byte[]); // ... writer.collect( partition: int, row: byte[]); writer.close(); // Reducer reader = new CoscoReader( shuffleId: String, mappers: long[], partition: int); while (reader.next()) { // using row row = reader.row(); } reader.close(); // Driver shuffle = new CoscoExchange( shuffleId: String, partitions: int, recomputeWritersCallback: (mappers: long[], reason: String) -> void); // end of exchange shuffle.close();
- 26. // Mapper writer = new CoscoWriter( shuffleId: String, mapper: long); writer.collect( partition: int, row: byte[]); // ... writer.collect( partition: int, row: byte[]); writer.close(); // Reducer reader = new CoscoReader( shuffleId: String, mappers: long[], partition: int); while (reader.next()) { // using row row = reader.row(); } reader.close(); // Driver shuffle = new CoscoExchange( shuffleId: String, partitions: int, recomputeWritersCallback: (mappers: long[], reason: String) -> void); // end of exchange shuffle.close();
- 27. Cosco Shuffle Services (thousands) DFS Cosco Shuffle Services (10Ks) Mapper 0 Mapper 1 Mapper 2 Reducer 0 Reducer 1 Partition 0 (file 1 buffer) File 0 File 1 File 2 File 0 File 1 network Partition 0 (file 2 buffer) Cosco Metadata Service Spark process (cosco client) Cosco process (shared between apps) Dependency Mappers submit what files them wrote to Commits files Reducers ask for files Metadata
- 28. Cosco Shuffle Services (thousands) DFS Cosco Shuffle Services (10Ks) Mapper 0 Mapper 1 Mapper 2 Reducer 0 Reducer 1 Partition 0 (file 1 buffer) File 0 File 1 File 2 File 0 File 1 network Partition 0 (file 2 buffer) Cosco Metadata Service Spark process (cosco client) Cosco process (shared between apps) Dependency Mappers submit what files them wrote to Commits files Reducers ask for files Metadata Mapper 0 Mapper 1 Mapper 2 Mapper 3 Mapper 4 File 0 File 1 File 2 File 0 File 1
- 29. Cosco Shuffle Services (thousands) DFS Cosco Shuffle Services (10Ks) Mapper 0 Mapper 1 Mapper 2 Reducer 0 Reducer 1 Partition 0 (file 1 buffer) File 0 File 1 File 2 File 0 File 1 network Partition 0 (file 2 buffer) Cosco Metadata Service Spark process (cosco client) Cosco process (shared between apps) Dependency Mappers submit what files them wrote to Commits files Reducers ask for files Metadata Mapper 0 Mapper 1 Mapper 2 Mapper 3 Mapper 4 File 0 File 1 File 2 File 0 File 1 Mapper 3’ (recompute) Mapper 4’ (recompute) File 3 File 2
- 30. // Mapper writer = new CoscoWriter( shuffleId: String, mapper: long); writer.collect( partition: int, row: byte[]); // ... writer.collect( partition: int, row: byte[]); writer.close(); // Reducer reader = new CoscoReader( shuffleId: String, mappers: long[], partition: int); while (reader.next()) { // using row row = reader.row(); } reader.close(); // Driver shuffle = new CoscoExchange( shuffleId: String, partitions: int, recomputeWritersCallback: (mappers: long[], reason: String) -> void); // end of exchange shuffle.close();
- 31. Cosco Shuffle Services (thousands) DFS Cosco Shuffle Services (thousands) Mapper 0 Mapper 1 Mapper 2 Reducer 0 Reducer 1 Partition 0 (file 1 buffer) File 0 File 1 File 2 File 0 File 1 network Partition 0 (file 2 buffer) Driver Cosco Metadata Service Spark process (cosco client) Cosco process (shared between apps) Dependency Mappers submit what files them wrote to Commits files Reducers ask for files Recompute request Metadata
- 32. Cosco Shuffle Services (thousands) DFS Cosco Shuffle Services (thousands) Mapper 0 Mapper 1 Mapper 2 Reducer 0 Reducer 1 Partition 0 (file 1 buffer) File 0 File 1 File 2 File 0 File 1 network Partition 0 (file 2 buffer) Driver Cosco Metadata Service Spark process (cosco client) Cosco process (shared between apps) Dependency Cosco Scheduler Mappers submit what files them wrote to Commits files Reducers ask for files Recompute request Scheduler
- 33. Limits • Cosco doesn’t support large rows (<4MiB) • Capacity: shuffle services memory, number of write-ahead buffers
- 34. Future work • “Unlimited” shuffle exchange: • millions of splits/partitions • 10s of PiBs • Streaming
- 35. Questions? Brian Cho (bcho@fb.com) Dmitry Borovsky (borovsky@fb.com)