#GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)
•
1 like•2,614 views
Today, if events change the decision model, we wait until the next batch model build for new insights. By extending fast “time-to-decisions” into the world of Big Data Analytics to get fast “time-to-insights”, apps will get what used to be batch insights in near real time. The technology enabling this includes smart in-memory data storage, new storage class memory, and products designed to do one or more parts of an analysis pipeline very well. In this talk we describe how Ampool is building on Apache Geode to allow Big Data analysis solutions to work together with a scalable smart storage class memory layer to allow fast and complex end-to-end pipelines to be built -- closing the loop and providing dramatically lower time to critical insights.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to #GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)
Similar to #GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode) (20)
More from PivotalOpenSourceHub
More from PivotalOpenSourceHub (15)
Recently uploaded
Recently uploaded (20)
#GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)
- 1. Democratizing Fast Analytics with Ampool (Powered by Apache Geode-incubating) Avinash Dongre and Robert Geiger, Ampool Inc.
- 2. " " " Analytics # # # # Apps Multi-Device Testing $ % & | ) * Analytics needs to work in CLOSED LOOP with AppsAnalytics Needs to be Faster!
- 3. " " " Analytics # # # # Apps Multi-Device Testing $ % & | ) * What are the CHALLENGES? ⚠ Many data users/ stakeholders ⚠ Disparate tools & processing needs ⚠ Long time to insights
- 7. AnalyticsIngest App UseETL … FAST OBJECT ACCESS across the pipeline # # # , , - , … . Data Architect Data Developers . . Business Analysts Data Scientists .
- 8. What ENABLERS can help here? In-Memory Fabric Technology • Apache Geode! • Flexible, stable, and proven distributed in-memory technology New memory technologies and fast network fabrics • Storage Class Memory • low latency, high throughput, persistent • Initially exposed via file system interface • Regular or memory mapped
- 9. Emerging Storage Class Memory (SCM) is DISRUPTIVE Challenges the value proposition of in-memory solutions Near DRAM latency and throughput at lower cost Based on one of several types of memory technology • MRAM (magnetic) • ReRAM (resistive) • FRAM (ferroelectric), PCM (phase change) • 3D-XPointTM (Intel/Micron) Accessible via Java and C/C++ libraries • Mnemonic (Java) • Pmem.io (C++)
- 11. In-Memory Technology CHALLENGES Line between memory and storage is blurring File systems getting really fast, so the speed gap is closing • SCM File Systems will also be low latency • File system overhead still limits latency improvements • Before: disk based vs. in-memory • After: file system vs. byte addressable object store Managing multiple layers and types of memory
- 12. Fast Closed Loop Analytics, Powered by a Smart, Distributed In-Memory Fabric… High throughput and large data handling matters • Throughput, latency, and capacity: • each pipeline stage values these differently Common interfaces, multiple region types • Meet the needs of many types of best of breed engines Managing multiple layers of memory and storage • Speed (latency, throughput) differentiator will diminish More classifications for data now • Hot, cold => hot, warm, lukewarm, cold
- 14. What Matters for App, DB, and Compute? The flexibility, suitability, and ease of use of the interfaces Memory & storage are managed transparently to provide QoS The service guarantee abstractions are provided Conflicts are managed and prevented Freeing developersfrom re-inventing the wheel
- 16. Smart Distributed In-Memory Object Store PLUGGABLE distributed memory layer … + 3D XPointTM ......
- 17. Smart Distributed In-Memory Object Store … for MANAGED FLEXIBILITY... + 3D XPointTM ...... ✅ Flexible regions and interfaces for ‘Best of breed’ engines ✅ Extensible Core ✅ Pluggable stores
- 18. AnalyticsIngest App UseETL …and FAST OBJECT ACCESS across the pipeline # # # , , - , … . Data Architect Data Developers . . Business Analysts Data Scientists .
- 21. No change in data application code Config. changes only No change in user experience Performance benefits No added hassles Current mgmt. tools …and deliver VALUE to all Analytics stakeholders . Data Architect Data Developers . . Business Analysts Data Scientists . . Data Admins Infra/ Sys Admins .
- 22. Contributing Back Plan for contributions back to Apache Geode: • Storage plug-ability layer • Off-heap memory plug-ability • SCM plugin (Mnemonic) • Impersonation support for security • Region type plug-ability
- 23. Thank You! Avinash Dongre Architect, Ampool India Pvt. Limited avinash@ampool.io Robert Geiger Chief Architect & VP Engineering, Ampool Inc. robert@ampool.io