Peering The Pacific Research Platform With The Great Plains Network
•Download as PPTX, PDF•
1 like•220 views
The Pacific Research Platform (PRP) connects research institutions across the western United States with high-speed networks to enable data-intensive science collaborations. Key points: - The PRP connects 15 campuses across California and links to the Great Plains Network, allowing researchers to access remote supercomputers, share large datasets, and collaborate on projects like analyzing data from the Large Hadron Collider. - The PRP utilizes Science DMZ architectures with dedicated data transfer nodes called FIONAs to achieve high-speed transfer of large files. Kubernetes is used to manage distributed storage and computing resources. - Early applications include distributed climate modeling, wildfire science, plankton imaging, and cancer genomics. The PR
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Peering The Pacific Research Platform With The Great Plains Network
Similar to Peering The Pacific Research Platform With The Great Plains Network (20)
More from Larry Smarr
More from Larry Smarr (20)
Recently uploaded
Recently uploaded (20)
Peering The Pacific Research Platform With The Great Plains Network
- 1. “Peering The Pacific Research Platform With The Great Plains Network” Keynote Great Plains Network 2018 Annual Meeting Kansas City, MO May 31, 2018 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD http://lsmarr.calit2.net 1
- 2. I Was Born and Raised in the Midwest: Columbia, Missouri My Grandfather, Father, and Me At My MU Graduation My Mother and Me On My First Birthday
- 3. I Earned Three of the Sixteen University of Missouri Degrees in My Family Framing By Brother David Smarr
- 4. 30 Years Ago NSF Brought to University Researchers a DOE HPC Center Model NCSA Was Modeled on LLNL SDSC Was Modeled on MFEnet 1985/6
- 5. Thirty Years After NSF Adopts DOE Supercomputer Center Model NSF Adopts DOE ESnet’s Science DMZ for High Performance Applications • A Science DMZ integrates 4 key concepts into a unified whole: – A network architecture designed for high-performance applications, with the science network distinct from the general-purpose network – The use of dedicated systems as data transfer nodes (DTNs) – Performance measurement and network testing systems that are regularly used to characterize and troubleshoot the network – Security policies and enforcement mechanisms that are tailored for high performance science environments http://fasterdata.es.net/science-dmz/ Science DMZ Coined 2010 The DOE ESnet Science DMZ and the NSF “Campus Bridging” Taskforce Report Formed the Basis for the NSF Campus Cyberinfrastructure Network Infrastructure and Engineering (CC-NIE) Program
- 6. Based on Community Input and on ESnet’s Science DMZ Concept, NSF Has Made Over 200 Campus-Level Awards in 44 States Source: Kevin Thompson, NSF
- 7. (GDC) Logical Next Step: The Pacific Research Platform Networks Campus DMZs to Create a Regional End-to-End Science-Driven “Big Data Superhighway” System NSF CC*DNI Grant $5M 10/2015-10/2020 PI: Larry Smarr, UC San Diego Calit2 Co-PIs: • Camille Crittenden, UC Berkeley CITRIS, • Tom DeFanti, UC San Diego Calit2/QI, • Philip Papadopoulos, UCSD SDSC, • Frank Wuerthwein, UCSD Physics and SDSC Letters of Commitment from: • 50 Researchers from 15 Campuses • 32 IT/Network Organization Leaders NSF Program Officer: Amy Walton Source: John Hess, CENIC
- 8. PRP National-Scale Experimental Distributed Testbed: Using Internet2 to Connect Early-Adopter Quilt Regional R&E Networks Original PRP Extended PRP Testbed Announced May 8, 2018 Internet2 Global Summit
- 9. PRP Science DMZ Data Transfer Nodes (DTNs) - Flash I/O Network Appliances (FIONAs) UCSD Designed FIONAs To Solve the Disk-to-Disk Data Transfer Problem at Full Speed on 10G, 40G and 100G Networks FIONAS—10/40G, $8,000 Phil Papadopoulos, SDSC & Tom DeFanti, Joe Keefe & John Graham, Calit2 FIONette—1G, $250 Five Racked FIONAs at Calit2 • Each Contains: • Dual 12-Core CPUs • 96GB RAM • 1TB SSD • 2 10GbE interfaces • Total ~$10,500 • With 8 GPUs • total ~$18,500
- 10. GPN Becomes the First Multi-State Regional Network to Peer with the PRP Seeing 5Gbs Between the PRP-Contributed PWave DTN in Los Angeles To GPN FIONA in UMC Source: John Hess, CENIC and George Rob III, UMissouri
- 11. Game Changer: Using Kubernetes to Manage Containers Across the PRP “Kubernetes is a way of stitching together a collection of machines into, basically, a big computer,” --Craig Mcluckie, Google and now CEO and Founder of Heptio "Everything at Google runs in a container." --Joe Beda,Google “Kubernetes has emerged as the container orchestration engine of choice for many cloud providers including Google, AWS, Rackspace, and Microsoft, and is now being used in HPC and Science DMZs. --John Graham, Calit2/QI UC San Diego
- 12. Rook is Ceph Cloud-Native Object Storage ‘Inside’ Kubernetes https://rook.io/ Source: John Graham, Calit2/QI
- 13. FIONA8 FIONA8 100G Epyc NVMe 40G 160TB 100G NVMe 6.4T SDSU 100G Gold NVMe March 2018 John Graham, UCSD 100G NVMe 6.4T Caltech 40G 160TB UCAR FIONA8 UCI FIONA8 FIONA8 FIONA8 FIONA8 FIONA8 FIONA8 FIONA8 FIONA8 sdx-controller controller-0 Calit2 100G Gold FIONA8 SDSC 40G 160TB UCR 40G 160TB USC 40G 160TB UCLA 40G 160TB Stanford 40G 160TB UCSB 100G NVMe 6.4T 40G 160TB UCSC 40G 160TB Hawaii Running Kubernetes/Rook/Ceph On PRP Allows Us to Deploy a Distributed PB+ of Storage for Posting Science Data Rook/Ceph - Block/Object/FS Swift API compatible with SDSC, AWS, and Rackspace Kubernetes Centos7
- 14. Operational Metrics: Containerized Trace Route Tool Allows Realtime Visualization of Status of Network Links All Kubernetes Nodes on PRP Source: Dmitry Mishin(SDSC), John Graham (Calit2)Presets This node graph shows UCR as the source of the flow to the mesh
- 15. We Measure Disk-to-Disk Throughput with 10GB File Transfer Using Globus GridFTP 4 Times Per Day in Both Directions for All PRP Sites April 24, 2017 Source: John Graham, Calit2
- 16. PRP’s First 2.5 Years: Connecting Multi-Campus Application Teams and Devices Earth Sciences GPN Is Beginning to Define Its Application Drivers
- 17. PRP Provides High Performance Access to Distributed Data Analysis
- 18. Distributed LHC Data Analysis Running Over PRP Source: Frank Würthwein, OSG, UCSD/SDSC, PRP GPN Can Connect Campus LHC Atlas and CMS Data Analysis
- 19. PRP Distributed Tier-2 Cache Across Caltech & UCSD-Thousands of Flows Sustaining >10Gbps! Cache Server Cache Server… Redirect or Cache Server Cache Server… Redirect or UCSD Caltech Redirector Top Level Cache Global Data Federation of CMS Provisioned pilot systems: PRP UCSD: 9 x 12 SATA Disk of 2TB @ 10Gbps for Each System PRP Caltech: 2 x 30 SATA Disk of 6TB @ 40Gbps for Each System Source: Frank Würthwein, OSG, UCSD/SDSC, PRP
- 20. Collaboration Opportunity with OSG & PRP on Distributed Storage 1.8PB1.2PB1.6PB 210TB Total data volume pulled last year is dominated by 4 caches. OSG Is Operating a Distributed Caching CI. At Present, 4 Caches Provide Significant Use PRP Kubernetes Infrastructure Could Either Grow Existing Caches by Adding Servers, or by Adding Additional Locations StashCache Users include: LIGO DES Source: Frank Würthwein, OSG, UCSD/SDSC, PRP
- 21. PRP Provides High Performance Access to Remote Supercomputers
- 22. 100 Gbps PRP Over CENIC Couples UC Santa Cruz Astrophysics Cluster to LBNL NERSC Supercomputer CENIC 2018 Innovations in Networking Award for Research Applications
- 23. The Great Plains Network Has Many Campuses With Active Projects at SDSC GPN Map Source: James Deaton, GPN Shawn Strande, SDSC
- 24. Dan Cayan USGS Water Resources Discipline Scripps Institution of Oceanography, UC San Diego much support from Mary Tyree, Mike Dettinger, Guido Franco and other colleagues Sponsors: California Energy Commission NOAA RISA program California DWR, DOE, NSF Planning for climate change in California substantial shifts on top of already high climate variability SIO Campus Climate Researchers Need to Download Results from NCAR Remote Supercomputer Simulations to Make Regional Climate Change Forecasts NCAR Upgrading to 100Gbps Link from Wyoming and Boulder to CENIC/PRP GPN Can Connect Campus NCAR Users
- 25. PRP Provides High Performance Access to Multi-Campus Big Data Collaborative Teams
- 26. PRP Will Link the Laboratories of the Pacific Earthquake Engineering Research Center http://peer.berkeley.edu/ PEER Labs: UC Berkeley, Caltech, Stanford, UC Davis, UC San Diego, and UC Los Angeles John Graham Installing FIONette at PEER Feb 10, 2017
- 27. Identifying NSF Multi-Institutional Grants Across and Beyond the PRP Source: GPN Staff; NSF Public Data Next Step: Which Use Big Data?
- 28. PRP Provides High Performance Access to Large Community Data Repositories
- 29. Cancer Genomics Hub (UCSC) Was Housed in SDSC, But NIH Moved Dataset From SDSC to UChicago - So the PRP Deployed a FIONA to Chicago’s MREN 1G 8G Data Source: David Haussler, Brad Smith, UCSC 15G Jan 2016
- 30. In 2011 EROS sent the equivalent of the entire library of congress every 9 days In 2011 SDSU was the 3rd largest user downloading data (GIS) In 2016 EROS sent the equivalent of the entire library of congress every 6 hours Source: Claude Garelik USGS Earth Resources Observation and Science (EROS) Center Is a Natural GPN/PRP Big Data Repository EROS is located ~15 miles north of Sioux Falls, South Dakota
- 31. PRP Provides High Performance Access to Large Scientific Instruments
- 32. 100 Gbps FIONA at UCSC Allows for Downloads to the UCSC Hyades Cluster from the LBNL NERSC Supercomputer for DESI Science Analysis 300 images per night. 100MB per raw image 120GB per night 250 images per night. 530MB per raw image 800GB per night Source: Peter Nugent, LBNL Professor of Astronomy, UC Berkeley Precursors to LSST and NCSA NSF-Funded Cyberengineer Shaw Dong @UCSC Receiving FIONA Feb 7, 2017
- 33. Global Scientific Instruments Will Produce Ultralarge Datasets Continuously Requiring Dedicated Optic Fiber and Supercomputers Large Synoptic Survey Telescope 3.2 Gpixel Camera Tracks ~40B Objects, Creates 1-10M Alerts/Night Within 1 Minute of Observing 1000 Supernovas Discovered/Night 2x100Gb/s “First Light” In 2019
- 34. The Prototype PRP Has Attracted New Application Drivers Scott Sellars, Marty Ralph Center for Western Weather and Water Extremes Frank Vernon, Graham Kent, & Ilkay Altintas, Wildfires Jules Jaffe – Undersea Microscope Tom Levy At-Risk Cultural Heritage
- 35. PRP UC-JupyterHub Backbone Connects FIONAs At UC Berkeley and UC San Diego Source: John Graham, Calit2 Goal: Jupyter Everywhere
- 36. PRP Provides High Performance Access to SensorNets Coupled to Realtime Computing
- 37. Church Fire, San Diego CA Alert SD&ECameras/HPWREN October 21, 2017 New PRP Application: Coupling Wireless Wildfire Sensors to Computing Thomas Fire, Ventura, CA Firemap Tool, WIFIRE December 10, 2017 CENIC 2018 Innovations in Networking Award for Experimental Applications
- 38. Once a Wildfire is Spotted, PRP Brings High-Resolution Weather Data to Fire Modeling Workflows in WIFIRE Real-Time Meteorological Sensors Weather Forecast Landscape data WIFIRE Firemap Fire Perimeter Work Flow PRP Source: Ilkay Altintas, SDSC
- 39. 2014 2015-2017 Grid Spacing 4 km 3 km Domain Size 1163x723x53 1683x1155x53 One Output Time 4.2 GB 9.7 GB Sub-Hourly Interval 10 min 6 min Complete Forecast Size Hourly + Sub-hourly 18h-30h 508 GB 1639 GB For 10 members per day 5.08 TB 16.4 TB For approx 30 days per season 152 TB 492 TB Full CONUS Data Volumes High Resolution Ensemble Weather Forecasts at The Center for Analysis and Prediction of Storms, University of Oklahoma Hazardous Weather Testbed • In 2017, CAPS started testing the next-generation forecasting model FV3 for convective-scale forecasting. • For 2018 HWT CLUE, CAPS is producing 5 ensembles using WRF and FV3 with a total of 52 forecasts of up to 84 hours. Source: Ming Xue and Keith Brewster, CAPS Prime Target for GPN/OneNet Dec, 2013: CAPS has >1 PB of in-house storage capacity
- 40. The Rise of Brain-Inspired Computers: Left & Right Brain Computing: Arithmetic vs. Pattern Recognition Adapted from D-Wave
- 41. UC San Diego Jaffe Lab (SIO) Scripps Plankton Camera Off the SIO Pier with Fiber Optic Network
- 42. Over 1 Billion Images So Far! Requires Machine Learning for Automated Image Analysis and Classification Phytoplankton: Diatoms Zooplankton: Copepods Zooplankton: Larvaceans Source: Jules Jaffe, SIO ”We are using the FIONAs for image processing... this includes doing Particle Tracking Velocimetry that is very computationally intense.”-Jules Jaffe
- 43. New NSF CHASE-CI Grant Creates a Community Cyberinfrastructure: Adding a Machine Learning Layer Built on Top of the Pacific Research Platform Caltech UCB UCI UCR UCSD UCSC Stanford MSU UCM SDSU NSF Grant for High Speed “Cloud” of 256 GPUs For 30 ML Faculty & Their Students at 10 Campuses for Training AI Algorithms on Big Data NSF Program Officer: Mimi McClure
- 44. FIONA8: Adding GPUs to FIONAs Supports Data Science Machine Learning Multi-Tenant Containerized GPU JupyterHub Running Kubernetes / CoreOS Eight Nvidia GTX-1080 Ti GPUs ~$13K 32GB RAM, 3TB SSD, 40G & Dual 10G ports Source: John Graham, Calit2
- 45. 48 GPUs for OSG Applications UCSD Adding >350 Game GPUs to Data Sciences Cyberinfrastructure - Devoted to Data Analytics and Machine Learning SunCAVE 70 GPUs WAVE + Vroom 48 GPUs FIONA with 8-Game GPUs 95 GPUs for Students CHASE-CI Grant Provides 96 GPUs at UCSD for Training AI Algorithms on Big Data Plus 288 64-bit GPUs On SDSC’s Comet
- 46. Next Step: Surrounding the PRP Machine Learning Platform With Clouds of GPUs and Non-Von Neumann Processors Microsoft Installs Altera FPGAs into Bing Servers & 384 into TACC for Academic Access CHASE-CI 64-TrueNorth Cluster 64-bit GPUs 4352x NVIDIA Tesla V100 GPUs GPN Next Step: Add GPUs to FIONAs
- 47. The Second National Research Platform Workshop Bozeman, MT August 6-7, 2018 Announced in I2 Closing Keynote: Larry Smarr “Toward a National Big Data Superhighway” on Wednesday, April 26, 2017 Co-Chairs: Larry Smarr, Calit2 Inder Monga, ESnet Ana Hunsinger, Internet2 Local Host: Jerry Sheehan, MSU
- 48. Expanding to the Global Research Platform Via CENIC/Pacific Wave, Internet2, and International Links PRP PRP’s Current International Partners Korea Shows Distance is Not the Barrier to Above 5Gb/s Disk-to-Disk Performance Netherlands Guam Australia Korea Japan Singapore
- 49. Our Support: • US National Science Foundation (NSF) awards CNS 0821155, CNS-1338192, CNS-1456638, CNS-1730158, ACI-1540112, & ACI-1541349 • University of California Office of the President CIO • UCSD Chancellor’s Integrated Digital Infrastructure Program • UCSD Next Generation Networking initiative • Calit2 and Calit2 Qualcomm Institute • CENIC, PacificWave and StarLight • DOE ESnet