The Rise of Supernetwork Data Intensive Computing

“The Rise of Supernetwork
Data Intensive Computing”
Invited Remote Lecture to SC21
The International Conference for High Performance
Computing, Networking, Storage, and Analysis
St. Louis, Missouri
November 18, 2021
Dr. Larry Smarr
Founding Director Emeritus, California Institute for Telecommunications and Information Technology;
Distinguished Professor Emeritus, Dept. of Computer Science and Engineering
Jacobs School of Engineering, UCSD
http://lsmarr.calit2.net

Abstract
Over the last 35 years, a fundamental architectural transformation in high performance data-intensive
computing has occurred, driven by the rise of optical fiber Supernetworks connecting the globe.
Ironically, this cyberinfrastructure revolution has been led by supercomputer centers, which then became
SuperNodes in this distributed system. I will review key moments, including the birth of the NSF
Supercomputer Centers and NSFnet, the gigabit testbeds, the NSF PACI program, the emergence of
Internet2 and the Regional Optical Networks, all eventually enabling, through a series of NSF grants, the
National and Global Research Platforms. Over this same period a similar cyberinfrastructure architecture
allowed the commercial clouds to develop, which are now interconnected with this academic distributed
system. Critical to this transformation has been the continual exponential rise of data and a new
generation of distributed applications utilizing this connected digital fabric. Throughout this period, the
role of the US Federal Government has been essential, anchored by the 1991 High-Performance
Computing Act, which established the Networking and Information Technology Research and
Development (NITRD) Program. Particularly important to the initiation of this distributed computing
paradigm shift was the continued visionary leadership of Representative, then Senator, then Vice
President Al Gore in the 1990s.

1975-1985: My Early Research was on Computational Astrophysics
Before There Were National Academic Supercomputer Centers
I Spent a Decade Supercomputing at LLNL (with Jim Wilson) and
Then at The Max Planck Institute for Physics and Astrophysics (with Mike Norman and Karl-Heinz Winkler)
Gas Accretion Onto a Black Hole
With Wilson and Hawley
1982
Cosmic Jets Emerging From Galactic Centers
With Norman and Winkler
1981
Gravitational Radiation From Black Hole Collisions
With Eppley
1978

1982-1983: Documenting The Unmet Supercomputing Needs
of a Broad Range of Disciplines Led to the NCSA Proposal to NSF
1982 1983
http://lsmarr.calit2.net/supercomputer_famine_1982.pdf http://lsmarr.calit2.net/Black_Proposal.pdf
1984: NSF Creates Office of Advanced Scientific Computing (John Connolly, Director)
Issues National Competition for Supercomputer Centers

1985: NSF Adopted a DOE High-Performance Computing Model
For Two of the New NSF Supercomputer Centers
NCSA Was Modeled on LLNL SDSC Was Modeled on MFEnet
1985

SuperNetworks Have Co-Evolved
with Supercomputers For 35 Years
“We ought to consider
a national initiative
to build interstate highways
for information
with a fiber optics network
connecting the
major computational centers
in this country”
-Senator Al Gore
“The University of Illinois
will be experimenting with
fiber optic
"information flow pipes,"
which promise to be able
to reach
billions of bits per second.””
-NCSA Director
Larry Smarr
http://lsmarr.calit2.net/hrg-1985-tec-0068_from_1_to_806_s.pdf
1985

Remote Interactive Visual Supercomputing End-to-End Prototype:
Using Analog Communications to Prototype the Fiber Optic Future
“We’re using satellite technology…
to demonstrate
what It might be like to have
high-speed fiber-optic links
between advanced computers
in two different
geographic locations.”
Illinois
Boston
SIGGRAPH 1989
“What we really have to do is eliminate distance
between individuals who want to interact
with other people and computers.”
― Larry Smarr, Director, NCSA
www.youtube.com/watch?v=C3d_6lw8_0M
-Al Gore, Senator
Chair, US Senate
Subcommittee
Cray 2 Driven by
Sun Workstation
AT&T & Sun
Telepresenc
e

1991: Networking Information Technology Research and Development (NITRD)
• NITRID Was Enacted in 1991 by Congress
Through the High-Performance Computing and Communication Act
• Brought Multiple Federal Agencies Together
to Plan and Coordinate Frontier Computing, Networking, Software, and Data
• Bill Was Sponsored and Driven by Senator Al Gore
December 2, 2021

The Bandwidth and Number of Endpoints
on NSFNET Grew Rapidly
Visualization of Inbound Traffic on the NSFNET T1 Backbone
(September 1991) by NCSA’s Donna Cox and Robert Patterson;
Data Collected by Merit Network, Inc.
1994
1991

• The First National 155 Mbps Research Network
– Inter-Connected Telco Networks Via IP/ATM With:
– Supercomputer Centers
– Virtual Reality Research Locations, and
– Applications Development Sites
– Into the San Diego Convention Center
– 65 Science Projects
• I-Way Featured:
– Networked Visualization Applications
– Large-Scale Immersive Displays
– I-Soft Programming Environment
– Led to the Globus Project
Supercomputing ’95:
I-WAY: A Model for Distributed Collaborative Computing
For details see:
“Overview of the I-WAY: Wide Area Visual Supercomputing”
DeFanti, Foster, Papka, Stevens, Kuhfuss
www.globus.org/sites/default/files/iway_overview.pdf SC95 Chair Sid Karin
SC95 Program Chair, Larry Smarr

1990-1996 CNRI’s Gigabit Testbeds
Demonstrated Host I/O Was the Distributed Computing Bottleneck
“Host I/O proved to be
the Achilles' heel
of gigabit networking –
whereas LAN and WAN technologies were
operated in the gigabit regime, many
obstacles impeded
achieving gigabit flows
into and out of
the host computers
used in the testbeds.”
--Final Report
The Gigabit Testbed Initiative
December 1996
Corporation for
National Research Initiatives (CNRI)
Robert Kahn
CNRI Chairman, CEO & President

NSF’s PACI Program was Built on the vBNS
to Prototype America’s 21st Century Information Infrastructure
PACI National Technology Grid
Testbed
National Computational Science
1997
vBNS
led to
Key Role
of Miron Livny
& Condor

The 25 Years From the National Techology Grid
To the National Research Platform
From I-WAY to the National Technology Grid, CACM, 40, 51 (1997)
Rick Stevens, Paul Woodward, Tom DeFanti, and Charlie Catlett

Dave Bader Created the First Linux COTS Supercluster -Roadrunner-
on the National Technology Grid, with the Support of NCSA and NSF
NCSA Director Larry Smarr (left),
UNM President William Gordon,
and U.S. Sen. Pete Domenici
Turn on the Roadrunner Supercomputer
in April 1999
1999
National Computational Science

Illinois’s I-WIRE and Indiana’s I-LIGHT Dark Fiber Networks
Inspired Many Other State and Regional Optical
Source: Larry Smarr, Rick Stevens, Tom DeFanti, Charlie Catlett
1999
Today California’s
CENIC R&E
Backbone Includes
~ 8,000 Miles of
CENIC-Owned and
Managed Fiber

1999: The President’s Information Technology Advisory Committee (PITAC) Report
Led to Funding NSF’s Information Technology Research (ITR) for National Priorities Program
Meeting with Vice President Gore in the White House
To Present Our PITAC Report
PITAC
Co-Chairs:
Ken Kennedy
Bill Joy

The OptIPuter
Exploits a New World
in Which
the Central Architectural Element
is Optical Networking,
Not Computers
to Support
Data-Intensive Scientific Research
and Collaboration
OptIPuter
NSF ITR Grant
$13.5M
PI Smarr,
Co-PIs DeFanti,
Papadopoulos, Ellisman
2002-2009
2002-2009: The NSF OptIPuter ITR Grant-
Can We Make Wide-Area Bandwidth Equal to Cluster Backplane Speeds?

Integrated “OptIPlatform” Cyberinfrastructure System:
A 10Gbps Lightpath Cloud
National LambdaRail
Campus
Optical
Switch
Data Repositories & Clusters
HPC
HD/4k Video Images
HD/4k Video Cams
End User
OptIPortal
10G
Lightpath
HD/4k Telepresence
Instruments
LS 2009
Slide

David Abramson Led OptIPuter Global Workflows and
UCSD/Monash Univ. Co-Mentoring of Undergrads and Graduate Students
First OptIPortal/Kepler
Remote Microscopy Link Feb 2009
Monash U.
UCSD
Monash U.

2010-2020:
NSF Adopted a DOE High-Performance Networking Model
Science
DMZ
Data Transfer
Nodes
(DTN/FIONA)
Network
Architecture
(zero friction)
Performance
Monitoring
(perfSONAR)
ScienceDMZ Coined in 2010 by ESnet
http://fasterdata.es.net/science-dmz/
Slide Adapted From Inder Monga, ESnet
DOE
NSF
NSF Campus Cyberinfrastructure Program
2012-2020
Has Made Over 340 Awards:
Across 50 States and Territories
Slide Adapted From Kevin Thompson, NSF

2013-2015: UCSD as a Laboratory for a “Big Data” 10-100 Gbps ScienceDMZ
NSF-Funded Campus CI Grants: Prism@UCSD and CHeruB
Prism@UCSD, Phil Papadopoulos, SDSC, Calit2, PI (2013-15)
CHERuB, Mike Norman, SDSC PI
CHERuB

(GDC)
2015 Vision: The Pacific Research Platform Will Connect Science DMZs
Creating a Regional End-to-End Science-Driven Community Cyberinfrastructure
NSF CC*DNI Grant
$6.3M 10/2015-10/2020
In Year 6 Now, Year 7 is Funded
Source: John Hess, CENIC
Supercomputer
Centers

PRP Website Has All Details Needed to Get Started
https://pacificresearchplatform.org/

2015-2021: UCSD Designs PRP Data Transfer Nodes (DTNs) --
Flash I/O Network Appliances (FIONAs)
FIONAs Solved the the Gigabit Testbed Disk-to-Disk Data Transfer Problem
at Near Full Speed on Best-Effort 10G, 40G and 100G
FIONAs Designed by UCSD’s Phil Papadopoulos, John Graham,
Joe Keefe, and Tom DeFanti
Up to 192 TB Rotating Storage
www.pacificresearchplatform.org
Today’s
Roadrunner!

2018/2019: PRP Game Changer!
Using Google’s Kubernetes to Orchestrate Containers Across the PRP
User
Applications
Containers
Clouds

PRP’s Nautilus Hypercluster Adopted Kubernetes
to Orchestrate Software Containers and Manage Distributed Storage
“Kubernetes with Rook/Ceph Allows Us to Manage Petabytes of
Distributed Storage and GPUs for Data Science,
While We Measure and Monitor Network Use.”
--John Graham, Calit2/QI UC San Diego
Kubernetes (K8s) is an open-source system for
automating deployment, scaling, and
management of containerized applications.

2017-2020: NSF CHASE-CI Grant Adds a Machine Learning Layer
Built on Top of the Pacific Research Platform
Caltech
UCB
UCI UCR
UCSD
UCSC
Stanford
MSU
UCM
SDSU
NSF Grant for High Speed “Cloud” of 256 GPUs
For 30 ML Faculty & Their Students at 10 Campuses
for Training AI Algorithms on Big Data
PI: Larry Smarr
Co-PIs:
• Tajana Rosing
• Ken Kreutz-Delgado
• Ilkay Altintas
• Tom DeFanti

Original
PRP
CENIC/PW Link
2018-2021: Toward the National Research Platform (TNRP) -
Using CENIC & Internet2 to Connect Quilt Regional R&E Networks
“Towards
The NRP”
3-Year Grant
Funded
by NSF
$2.5M
October 2018
Award #1826967
PI Smarr
Co-PIs Altintas,
Papadopoulos,
Wuerthwein,
Rosing

Rotating Storage
4000 TB
PRP’s Nautilus is a Multi-Institution Hypercluster
Connected by Optical Networks
184 FIONAs on 25 Partner Campuses
Networked Together at 10-100Gbps

PRP’s Nautilus is Global in Scale
FIONAs

PRP’s Nautilus Spans the United States
FIONAs

PRP’s Nautilus is Centered in SoCal
FIONAs
UCSD &
SDSU
UCI
Caltech
UCSB
UCR
CSUSB

We Measure Disk-to-Disk Throughput with 10GB File Transfer
4 Times Per Day in Both Directions for All PRP Sites
January 29,
2016
From Start of Monitoring 12 DTNs
to 24 DTNs Connected at 10-40G
in 1 ½ Years
July 21, 2017
Source: John Graham, Calit2

Operational Metrics: Containerized Trace Route Tool Allows Realtime Visualization
of Status of PRP Network Links on a National and Global Scale
Source: Dima Mishin, SDSC
9/16/2019
Guam
Univ. Queensland
Australia
LIGO
UK
Netherlands
Korea

PRP is Science-Driven:
Connecting Multi-Campus Application Teams and Devices
Earth
Sciences
UC San Diego UCBerkeley UC Merced

Director:
F. Martin Ralph
Big Data Collaboration with:
Scott Sellers, PhD CHRS; Postdoc CW3E
PRP Accelerates Data-Intensive Workflow on Atmospheric Water in the West
Between NASA MERRA Archive, UC San Diego, and UC Irvine
Director:
Soroosh Sorooshian
Complete Workflow Time:
19.2 days🡪52 Minutes!
See Paper by Sellars, et al., IEEE eScience (2019)
http://lsmarr.calit2.net/sellars_accelerating_image_segmentation.pdf

The New Pacific Research Platform Video
Highlights 3 Different Applications Out of 600 Nautilus Namespace Projects
Pacific Research Platform Video:
www.thequilt.net/campus-cyberinfrastructure-program-resource/
www.pacificresearchplatform.org

Co-Existence of Interactive and
Non-Interactive Computing on PRP
GPU Simulations Needed to Improve Ice Model.
=> Results in Significant Improvement
in Pointing Resolution for Multi-Messenger Astrophysics
NSF Large-Scale Observatories Are Using PRP and OSG
as a Cohesive, Federated, National-Scale Research Data Infrastructure
NSF’s IceCube & LIGO Both See Nautilus
as Just Another OSG Resource
IceCube Used Up to Half of
PRP’s 500 GPUs in 2020!

UC President Napolitano's Research Catalyst Award to
UC San Diego (Tom Levy), UC Berkeley (Benjamin Porter), UC Merced (Nicola Lercari) and UCLA (Willeke Wendrich)
PRP Links At-Risk Cultural Heritage and Archaeology Datasets
to Virtual Reality Systems at Multiple Campuses
48 Megapixel CAVEkiosk
UCSD Library
UCB CITRIS Tech Museum
UCM Library

Once a Wildfire is Spotted, PRP Brings High-Resolution Weather Data
to Fire Modeling Workflows in WIFIRE
Real-Time
Meteorological Sensors
Weather Forecast
Landscape data
WIFIRE Firemap
Fire Perimeter
Work Flow
PRP
Source: Ilkay Altintas, SDSC

Community Building Though Inclusion and Diversity
• Grants
– 3 Female co-PIs
– 1 Hispanic co-PI
• Campuses
– 8 Minority-Serving Institutions in PRP/CHASE-CI
• Workshops
– NRPII Workshop Steering Committee 80% Female
– Multiple MSI, EPSCoR Focused Workshops
Jackson State University
PRP MSI Workshop
Presenting
FIONettes

2021-2024 NRP Future I: Proposed Extension of Nautilus
CHASE-CI ENS, Tom DeFanti PI (NSF Award # 2120019)
CHASE-CI ABR, Larry Smarr PI (NSF Award # 2100237)
$2.8M

2021-2026 NRP Future II: PRP Federates with SDSC’s EXPANSE
Using CHASE-CI Developed Composable Systems
~$20M over 5 Years
PI Mike Norman, SDSC

2021-2026 NRP Future III: PRP Federates with
NSF-Funded Prototype National Research Platform
NSF Award OAC #2112167 (June 2021) [$5M Over 5 Years]
PI Frank Wuerthwein (UCSD, SDSC)
Co-PIs Tajana Rosing (UCSD), Thomas DeFanti (UCSD), Mahidhar Tatineni (SDSC), Derek Weitzel (UNL)

PRP/TNRP/CHASE-CI Support and Community:
• US National Science Foundation (NSF) awards to UCSD, NU, and SDSC
⮚ CNS-1456638, CNS-1730158, ACI-1540112, ACI-1541349, & OAC-1826967
⮚ OAC 1450871 (NU) and OAC-1659169 (SDSU)
• UC Office of the President, Calit2 and Calit2’s UCSD Qualcomm Institute
• San Diego Supercomputer Center and UCSD’s Research IT and Instructional IT
• Partner Campuses: UCB, UCSC, UCI, UCR, UCLA, USC, UCD, UCSB, SDSU, Caltech, NU,
UWash UChicago, UIC, UHM, CSUSB, HPWREN, UMo, MSU, NYU, UNeb, UNC,UIUC,
UTA/Texas Advanced Computing Center, FIU, KISTI, UVA, AIST
• CENIC, Pacific Wave/PNWGP, StarLight/MREN, The Quilt, Kinber, Great Plains Network,
NYSERNet, LEARN, Open Science Grid, Internet2, DOE ESnet, NCAR/UCAR & Wyoming
Supercomputing Center, AWS, Google, Microsoft, Cisco

The Rise of Supernetwork Data Intensive Computing

More Related Content

Similar to The Rise of Supernetwork Data Intensive Computing

Similar to The Rise of Supernetwork Data Intensive Computing (20)

More from Larry Smarr

More from Larry Smarr (20)

Recently uploaded

Recently uploaded (20)

The Rise of Supernetwork Data Intensive Computing