Berkeley cloud computing meetup may 2020
•Download as PPTX, PDF•
0 likes•398 views
The Pacific Research Platform (PRP) is a high-bandwidth global private "cloud" connected to commercial clouds that provides researchers with distributed computing resources. It links Science DMZs at universities across California and beyond using a high-performance network. The PRP utilizes Data Transfer Nodes called FIONAs to transfer data at near full network speeds. It has adopted Kubernetes to orchestrate software containers across its resources. The PRP provides petabytes of distributed storage and hundreds of GPUs for machine learning. It allows researchers to perform data-intensive science across multiple universities much faster than possible individually.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Berkeley cloud computing meetup may 2020
Similar to Berkeley cloud computing meetup may 2020 (20)
More from Larry Smarr
More from Larry Smarr (20)
Recently uploaded
Recently uploaded (20)
Berkeley cloud computing meetup may 2020
- 1. “The Pacific Research Platform- a High-Bandwidth Global-Scale Private ‘Cloud’ Connected to Commercial Clouds” Presentation to the UC Berkeley Cloud Computing MeetUp May 26, 2020 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD http://lsmarr.calit2.net 1
- 2. Before the PRP: ESnet’s ScienceDMZ Accelerates Science Research: DOE & NSF Partnering on Science Engagement and Technology Adoption Science DMZ Data Transfer Nodes (DTN/FIONA) Network Architecture (zero friction) Performance Monitoring (perfSONAR) ScienceDMZ Coined in 2010 by ESnet Basis of PRP Architecture and Design http://fasterdata.es.net/science-dmz/ Slide Adapted From Inder Monga, ESnet DOE NSF NSF Campus Cyberinfrastructure Program Has Made Over 250 Awards 2012 2013 2014 2015 2016 2017 2018
- 3. (GDC) 2015 Vision: The Pacific Research Platform Will Connect Science DMZs Creating a Regional End-to-End Science-Driven Community Cyberinfrastructure NSF CC*DNI Grant $6.3M 10/2015-10/2020 In Year 5 Now PI: Larry Smarr, UC San Diego Calit2 Co-PIs: • Camille Crittenden, UC Berkeley CITRIS, • Philip Papadopoulos, UCI • Tom DeFanti, UC San Diego Calit2/QI, • Frank Wuerthwein, UCSD Physics and SDSC Source: John Hess, CENIC Letters of Commitment from: • 50 Researchers from 15 Campuses • 32 IT/Network Organization Leaders Supercomputer Centers
- 4. PRP Links At-Risk Cultural Heritage and Archaeology Datasets at UCB, UCLA, UCM and UCSD with CAVEkiosks 48 Megapixel CAVEkiosk UCSD Library 48 Megapixel CAVEkiosk UCB CITRIS Tech Museum 24 Megapixel CAVEkiosk UCM Library UC President Napolitano's Research Catalyst Award to UC San Diego (Tom Levy), UC Berkeley (Benjamin Porter), UC Merced (Nicola Lercari) and UCLA (Willeke Wendrich)
- 5. Terminating the Fiber Optics - Data Transfer Nodes (DTNs): Flash I/O Network Appliances (FIONAs) UCSD-Designed FIONAs Solved the Disk-to-Disk Data Transfer Problem at Near Full Speed on Best-Effort 10G, 40G and 100G Networks FIONAs Designed by UCSD’s Phil Papadopoulos, John Graham, Joe Keefe, and Tom DeFanti Two FIONA DTNs at UC Santa Cruz: 40G & 100G Up to 192 TB Rotating Storage Add Up to 8 Nvidia GPUs Per 2U FIONA To Add Machine Learning Capability
- 6. 2017-2020: NSF CHASE-CI Grant Adds a Machine Learning Layer Built on Top of the Pacific Research Platform Caltech UCB UCI UCR UCSD UCSC Stanford MSU UCM SDSU NSF Grant for High Speed “Cloud” of 256 GPUs For 30 ML Faculty & Their Students at 10 Campuses for Training AI Algorithms on Big Data
- 7. Original PRP CENIC/PW Link 2018-2021: Toward the National Research Platform (NRP) - Using CENIC & Internet2 to Connect Quilt Regional R&E Networks “Towards The NRP” 3-Year Grant Funded by NSF $2.5M October 2018 PI Smarr Co-PIs Altintas Papadopoulos Wuerthwein Rosing DeFanti NSF CENIC Link
- 8. 2018/2019: PRP Game Changer! Using Kubernetes to Orchestrate Containers Across the PRP “Kubernetes is a way of stitching together a collection of machines into, basically, a big computer,” --Craig Mcluckie, Google and now CEO and Founder of Heptio "Everything at Google runs in a container." --Joe Beda,Google
- 9. PRP’s Nautilus Hypercluster Adopted Kubernetes to Orchestrate Software Containers and Rook, Which Runs Inside of Kubernetes, to Manage Distributed Storage https://rook.io/ “Kubernetes with Rook/Ceph Allows Us to Manage Petabytes of Distributed Storage and GPUs for Data Science, While We Measure and Monitor Network Use.” --John Graham, Calit2/QI UC San Diego
- 10. 100G NVMe 6.4TB Caltech 40G 192TB UCSF 40G 160TB HPWREN 40G 160TB 4 FIONA8s* Calit2/UCI 35 FIONA2s 17 FIONA8s 2x40G 160TB HPWREN UCSD 100G Epyc NVMe 100G Gold NVMe 8 FIONA8s + 5 FIONA8s SDSC @ UCSD 1 FIONA8 40G 160TB UCR 40G 160TB USC 100G NVMe 6.4TB 2x40G 160TB UCLA 1 FIONA8* 40G 160TB Stanford U 2 FIONA8s* 40G 192TB UCSB 4.5 FIONA8s 100G NVMe 6.4TB 40G 160TB UCSC PRP’s California Nautilus Hypercluster Connected by Use of CENIC 100G Network 10 FIONA2s 2 FIONA8 40G 160TB UCM 15-Campus Nautilus Cluster: 4360 CPU Cores 134 Hosts ~1.7 PB Storage 407 GPUs, ~4000 cores each 40G 160TB HPWREN 100G NVMe 6.4TB 1 FIONA8* 2 FIONA4s FPGAs + 2PB BeeGFS SDSU PRP Disks 10G 3TB CSUSB Minority Serving Institution CHASE-CI 100G 48TB NPS *= July RT 40G 192TB USD
- 11. CENIC/PW Link 40G 3TB U Hawaii 40G 160TB NCAR-WY 40G 192TB UWashington 10G FIONA1 40G FIONA UIC 40G 3TB StarLight PRP/TNRP’s United States Nautilus Hypercluster FIONAs Now Connects 4 More Regionals and 3 Internet2 Storage Sites 100G FIONA I2 Chicago 100G FIONA I2 Kansas City 100G FIONA I2 NYC
- 12. PRP Global Nautilus Hypercluster Is Rapidly Adding International Partners Beyond Our Original Partner in Amsterdam Netherlands 10G 35TB UvA PRP Transoceanic Nodes Show Distance is Not a Barrier to Above 5Gb/s Disk-to-Disk Performance PRP’s Current International Partners Guam Australia Korea Singapore 40G FIONA6 40G 28TB KISTI 10G 96TB U of Guam 100G 35TB U of Queensland GRP Workshop 9/17-18/2019 at Calit2@UCSD
- 13. PRP’s Nautilus Forms a Multi-Application Powerful Distributed “Big Data” Storage and Machine-Learning Computer Source: grafana.nautilus.optiputer.net on 1/27/2020
- 14. Calit2’s FIONA SDSC’s COMET Calit2’s FIONA Pacific Research Platform (10-100 Gb/s) GPUsGPUs Complete workflow time: 19.2 days52 Minutes! 532 Times Faster!UC, Irvine UC, San Diego Collaboration on Distributed Machine Learning for Atmospheric Water in the West Between UC San Diego and UC Irvine Source: Scott Sellers, CW3E
- 15. UCB Science Engagement Workshop: Applying Advanced Astronomy AI to Microscopy Workflows Organized and Coordinated by UCB’s PRP Science Engagement Team
- 16. Co-Existence of Interactive and Non-Interactive Computing on PRP GPU Simulations Needed to Improve Ice Model. Results in Significant Improvement in Pointing Resolution for Multi-Messenger Astrophysics But IceCube Did Not Have Access to GPUs NSF Large-Scale Observatories Asked to Utilize PRP Compute Resources
- 17. IceCube Number of Requested PRP Nautilus GPUs For All Projects Has Gone Up 4X in 2019 Largely Driven By the Unplanned Access by NSF’s IceCube 4X https://grafana.nautilus.optiputer.net/d/fHSeM5Lmk/k8s-compute-resources-cluster- gpus?orgId=1&fullscreen&panelId=2&from=1546329600000&to=1577865599000
- 18. Multi-Messenger Astrophysics with IceCube Across All Available GPUs in the Cloud • Integrate All GPUs Available for Sale Worldwide into a Single HTCondor Pool – Use 28 Regions Across AWS, Azure, and Google Cloud for a Burst of a Couple Hours, or so – Launch From PRP FIONAs • IceCube Submits Their Photon Propagation Workflow to this HTCondor Pool. – The Input, Jobs on the GPUs, and Output are All Part of a Single Globally Distributed System – This Demo Used Just the Standard HTCondor Tools Run a GPU Burst Relevant in-Scale for Future Exascale HPC Systems
- 19. Science with 51,000 GPUs Achieved as Peak Performance 19 Time in Minutes Each Color is a Different Cloud Region in US, EU, or Asia. Total of 28 Regions in Use Peaked at 51,500 GPUs ~380 Petaflops of FP32 Summary of Stats at Peak - 8 Generations of NVIDIA GPUs Used
- 20. Engaging More Scientists: PRP Website http://ucsd-prp.gitlab.io/