Scylla Summit 2017: How Baidu Runs Scylla on a Petabyte-Level Big Data Platform
•Download as PPTX, PDF•
0 likes•486 views
In this presentation, I'll speak of the benefits of running Scylla on our Big Data environment which stores over 500TB of data as well as using Scylla as the indexing engine to replace MongoDB and Cassandra for our log data analysis platform.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to Scylla Summit 2017: How Baidu Runs Scylla on a Petabyte-Level Big Data Platform
Similar to Scylla Summit 2017: How Baidu Runs Scylla on a Petabyte-Level Big Data Platform (20)
More from ScyllaDB
More from ScyllaDB (20)
Recently uploaded
Recently uploaded (20)
Scylla Summit 2017: How Baidu Runs Scylla on a Petabyte-Level Big Data Platform
- 1. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company How Baidu runs Scylla on PB-level big data platform Baidu Security R&D Department Zhangmei Li & Jeff Liu
- 2. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Agenda About me Introduction of Baidu Security How to use Scylla in a log analysis system What is the purpose of Scylla The development about Scylla in the future 2
- 3. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company About Baidu security 3 Protect Baidu Protect Partner Protect People Cloud WAFAnti-DDos Device fingerprint DNS Hijacking Detection Web Page Hijacking Detection Simulator Detection Threat Intelligence Service System Vulnerabilities Scan Service Others… Big Data Platform Data-Driven Security More fast, More intelligence
- 4. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Baidu security big data platform 4 Management Process Storage Collection Data Searching Data Warehouse Spark MapReduce Kafka HiveHDFS FTPMinos Client DataProxy Mola ES Storm Graph Database Service HBase Cluster Monitor Statistical Analysis System Flume Flink OpenTSDB Resource Management && Scheduler Scylla Metadata Management
- 5. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company How to analyze logs instantly
- 6. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Log analysis system • Source • Access Log • Syslog .etc • Volume • ~500TB per day • text/pb/gzip • Goal • Build Index per day • Quick Search • Value • APT • Security Analysis
- 7. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Index building architecture
- 8. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Three levels index 2nd-level-index is about 8%-10% of source log size, ~50TB 1st-level-index and 3rd-level-index is about 0.4% of source log size, ~2TB
- 9. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Search architecture
- 10. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Log index storage ✓ KV Storage ✓ High-Speed Read ✓ High-Speed Write ✓ Available Every Time ✓ Easy Maintenance Storage System Expectation
- 11. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Why scylla?
- 12. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Separate R/W for indexing Real-time Analytics replicationoffline online Index building service Bulk Write
- 13. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Perf of Scylla on HDD
- 14. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company Expect secondary index
- 15. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company The plan in the future Try to replace Redis with Scylla which is used as a cache service, given that Scylla could supply higher performance and was more convenient in terms of maintenance and scalability. PLAN-1 : Replace Redis with Scylla DC1 DC2 DC3
- 16. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company The plan in the future Combine Scylla with OLAP data warehouse because it performs awesome query throughput. PLAN-2 : Integrate Scylla into OLAP data warehouse OLAP Spark SparkSpark SQL Query CQL Query
- 17. PRESENTATION TITLE ON ONE LINE AND ON TWO LINES First and last name Position, company THANKS