The Pacific Research Platform: Building a Distributed Big-Data Machine-Learning Cyberinfrastructure
•Download as PPTX, PDF•
0 likes•368 views
The document summarizes the Pacific Research Platform (PRP) which connects researchers across multiple universities with high-speed networks and computing resources for big data and machine learning applications. Key points: - PRP connects 15 universities with optical networks, distributed storage devices (FIONAs), and over 350 GPUs for data analysis and AI training. - It allows researchers to rapidly share and analyze large datasets, with one example reducing a workflow from 19 days to 52 minutes. - Other projects using PRP resources include climate modeling, astrophysics simulations, and machine learning courses involving thousands of students.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to The Pacific Research Platform: Building a Distributed Big-Data Machine-Learning Cyberinfrastructure
Similar to The Pacific Research Platform: Building a Distributed Big-Data Machine-Learning Cyberinfrastructure (20)
More from Larry Smarr
More from Larry Smarr (20)
Recently uploaded
Recently uploaded (20)
The Pacific Research Platform: Building a Distributed Big-Data Machine-Learning Cyberinfrastructure
- 1. “The Pacific Research Platform: Building a Distributed Big-Data Machine-Learning Cyberinfrastructure” Briefing Jacobs School of Engineering University of California San Diego July 18, 2019 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD http://lsmarr.calit2.net
- 2. UC San Diego’s Calit2 & SDSC Have Pioneered Big-Data Cyberinfrastructure for 17 Years with NSF Grants: OptIPuter, Quartzite, Prism, CHERuB, PRP, CHASE-CI, TNRP OptIPuter PI Smarr, Co-PI DeFanti Co-PI Papadopoulos, Ellisman 2002-2009 Quartzite PI Papadopoulos, Co-PI Smarr, Ford, Fainman
- 3. 2013-2015: Creating a “Big Data” Backplane on Campus: NSF CC-NIE Funded Prism@UCSD and CHERuB Prism@UCSD, Phil Papadopoulos, SDSC, Calit2, PI; Smarr co-PI CHERuB, Mike Norman, SDSC PI CHERuB
- 4. (GDC) 2015-2020: The Pacific Research Platform Connects Campus “Big Data Freeways” to Create a Regional End-to-End Science-Driven “Big Data Superhighway” System NSF CC*DNI Grant $6M 10/2015-10/2020 PI: Larry Smarr, UC San Diego Calit2 Co-PIs: • Camille Crittenden, UC Berkeley CITRIS, • Tom DeFanti, UC San Diego Calit2/QI, • Philip Papadopoulos, UCSD SDSC, • Frank Wuerthwein, UCSD Physics and SDSC Letters of Commitment from: • 50 Researchers from 15 Campuses • 32 IT/Network Organization Leaders Source: John Hess, CENIC UCOP CIO Tom Andriola Provided Funds and ITLC Support for Using Ten UC Campuses For Advanced Technology Testing
- 5. 2017-2020: CHASE-CI Adds Machine-Learning to the Data-Science Community Cyberinfrastructure Caltech UCB UCI UCR UCSD UCSC Stanford MSU UCM SDSU NSF Grant for 256 High Speed “Cloud” GPUs For 32 ML Faculty & Their Students at 10 Campuses To Train AI Algorithms on Big Data
- 6. PRP Engineers Designed and Built Several Generations of Optical-Fiber Big-Data Flash I/O Network Appliances (FIONAs) UCSD-Designed FIONAs Solved the Disk-to-Disk Data Transfer Problem at Near Full Speed on Best-Effort 10G, 40G and 100G Networks FIONAs Designed by UCSD’s Phil Papadopoulos, John Graham, Joe Keefe, and Tom DeFanti FIONette— 1G, $250 Used for Training 50 Engineers in 2018-2019 Two FIONA DTNs at UC Santa Cruz: 40G & 100G Up to 200 TeraByte Rotating Storage Add Up to 8 Nvidia GPUs Per FIONA To Add Machine Learning Capability Over 100 Now Deployed on PRP
- 7. 48 GPUs for OSG Applications UCSD Has Added >350 Game GPUs to Data Sciences Cyberinfrastructure - Devoted to Data Analytics and Machine Learning SunCAVE 70 GPUs WAVE + Vroom 48 GPUs FIONA with 8-Game GPUs 104 GPUs for Students CHASE-CI Grant Provides 96 GPUs at UCSD for Training AI Algorithms on Big Data Plus 288 64-bit GPUs On SDSC’s Comet
- 8. UCSD’s ITS Adapted PRP FIONA8s To Support Data Science Courses Instructional Data Science Machine Learning Platform: Instead of Spending ~$20,000/Quarter/Course on Commercial Clouds: 97 Courses over 6 Quarters $4M vs. $240K over 12 Quarters At least 20,000 Students Adam Tilghman, ITS Source: UCSD ITS
- 9. The Student GPUs Have Supported a Broad Set of Courses Across Campus Source: UCSD ITS
- 10. The ITS GPUs Have Supported Thousands of Students Source: UCSD ITS
- 11. Student GPU Demand Is Variable Allowing for Other Student Uses Available to Support: Independent Study, For-Credit Research, External Barter Source: UCSD ITS
- 12. 2018-2019: PRP Game Changer! Using Kubernetes to Orchestrate Containers Across the PRP “Kubernetes is a way of stitching together a collection of machines into, basically, a big computer,” --Craig Mcluckie, Google and now CEO and Founder of Heptio "Everything at Google runs in a container." --Joe Beda,Google
- 13. 1 FIONA8 1 FIONA8 100G NVMe 6.4TB 100G NVMe 6.4TB Caltech 40G 160TB UCAR 40G 192TB UCSF 40G 160TB HPWREN 40G 160TB 4 FIONA8s Calit2/UCI 35 FIONA2s 12 FIONA8s 2x40G 160TB HPWREN UCSD 100G Epyc NVMe 100G Gold NVMe 8 FIONA8s + 5 FIONA8s SDSC @ UCSD 40G 160TB UCR 40G 160TB USC 2x40G 160TB UCLA 40G 160TB Stanford U 2 FIONA8s 40G 192TB UCSB 4.5 FIONA8s 100G NVMe 6.4TB 40G 160TB UCSC 40G 160TB U Hawaii Nautilus Kubernetes Cluster Connected by CENIC in California 10 FIONA2s 1 FIONA8 40G 160TB UCM 100Gb/s HPR 17 Campus Nautilus Cluster: 3300 CPU Cores 82 Hosts ~4 PB Storage >350 GPUs: >30M core/hrs/day 40G 160TB HPWREN 100G NVMe 6.4TB 1 FIONA8 2 FIONA4s FPGAs + 2PB BeeGFS SDSU 40G FIONA1 UIC CHASE-CI PRP Disks 10G 3TB CSUSB 40G 192TB U Washington Minority Serving Institution
- 14. Major CHASE-CI Usage by UCI Over PRP to UCSD CPUs/GPUs Cognitive Anteater Robotics Laboratory (CARL) supervised by Prof. Jeff Krichmar UCICompVis Group supervised by Prof. Charless Fowlkes #ofCores Demo Last Night From Data Think Tank Lab 2 Months
- 15. Very Cost-Effective for Academic Machine Learning and Data Sharing • Data science researchers need DTNs with lots of storage, encryption and lots of GPUS • One UC spends $40,000 in cloud GPU per published grad student paper • Another spends $20,000 for undergrad ML AWS access in just one course • Instead, add to our Nautilus hypercluster (or clone it & federate): – UCSD ECE Department bought 4 FIONA8s, buying 4 more – UCSD Physics Department. bought 3 FIONA8s, buying 3 more – UCSD CSE researchers bought/are buying FIONA8s to add to Nautilus – UCSD Instructional IT has 13 FIONA8s for Machine Learning/AI class labs • Working Storage on Nautilus FIONAs is – very inexpensive (12TB drives are ~$430 each—16 per FIONA. FISA encrypted drives @ same cost) – and very high speed (most FIONAs are 40/100G and are located in ScienceDMZs) Clemson’s Alex Feltus: “I cannot wait to add a node to the Nautilus compute fabric!” 5/22/2019
- 16. Nautilus Usage April 17, 2019 to July 17, 2019
- 17. Biggest Nautilus GPU Users December – April, 2019 CSE ECE Struc. Eng
- 18. Extra slides
- 19. Original PRP CENIC/PW Link 2018-2019: National-Scale Pilot - Using CENIC & Internet2 to Connect Quilt Regional R&E Networks Announced May 8, 2018 Internet2 Global Summit “Towards The NRP” 3-Year Grant Funded by NSF $2.5M OAC-1826967 PI Smarr Co-PIs Altintas Papadopoulos Wuerthwein Rosing Mgr: DeFanti NRP Pilot NSF CENIC Link
- 20. CENIC/PW Link 40G 3TB U Hawaii 40G 160TB NCAR-WY 40G 192TB UWashington 100G FIONA I2 Chicago 100G FIONA I2 Kansas City 10G FIONA1 40G FIONA UIC 100G FIONA I2 NYC 40G 3TB StarLight United States PRP Nautilus Hypercluster FIONAs We Now Connect 3 More Regionals and 3 Internet2 sites
- 21. Global PRP Nautilus Hypercluster Is Rapidly Increasing Partners Beyond Our Original Partner in Amsterdam—May 2019 PRP PRPv2 Nautilus Transoceanic Nodes Guam Asian Pacific RP Transoceanic Nodes Australia Korea Singapore Netherlands 10G 35TB UvA 40G FIONA6 40G 28TB KISTI 10G (coming) U of Guam 100G 35TB U of Queensland Transoceanic Nodes Show Distance is Not the Barrier to Above 5Gb/s Disk-to-Disk Performance
- 22. PRP is Science-Driven: Connecting Multi-Campus Application Teams and Devices Earth Sciences UC San Diego UCBerkeley
- 23. Director: F. Martin Ralph Big Data Collaboration with: Source: Scott Sellers, PhD CHRS; Postdoc CW3E Collaboration on Atmospheric Water in the West Between UC San Diego and UC Irvine Director, Soroosh Sorooshian, UCSD
- 24. Calit2’s FIONA SDSC’s COMET Calit2’s FIONA Pacific Research Platform (10-100 Gb/s) GPUsGPUs Complete Workflow Time: 19.2 Days52 Minutes! UC, Irvine UC, San Diego PRP Shortened Scott Sellar’s Workflow From 19.2 Days to 52 Minutes - 532 Times Faster! Source: Scott Sellers, US State Dept.
- 25. OSG IceCube Usage on PRP (Purple Segment) 3/9/19: Using 126 GPUs + 142 CPUs + 49 GB RAM GPU Simulations Needed to Improve Ice Model. => Results in Significant Improvement in Pointing Resolution for Multi-Messenger Astrophysics IceCube
- 26. PRP Actively Develops Diversity • Grants – 3 Female co-PIs – 1 Hispanic co-PI • Campuses – 8 Minority-Serving Institutions in PRP/Nautilus • Workshops – NRP’18 Workshop Program Committee 80% Female – Multiple MSI, EPSCoR Focused Workshops Jackson State University PRP MSI Workshop Presenting FIONettes
- 27. Installing FIONAs Across California in Late 2018 and Early 2019 To Enhance User’s CPU and GPU Computing, Data Posting, and Data Transfers UC Merced Stanford UC Santa Barbara UC Riverside UC Santa Cruz UC Irvine