poster

Online RL in the programmable dataplane with OPaL

Authors:

Kyle A. Simpson and

Dimitrios P. PezarosAuthors Info & Claims

CoNEXT '21: Proceedings of the 17th International Conference on emerging Networking EXperiments and Technologies

December 2021

Pages 471 - 472

https://doi.org/10.1145/3485983.3493345

Published: 03 December 2021 Publication History

Get Access

Abstract

Reinforcement learning (RL) is a key tool in data-driven networking for learning to control systems online. While recent research has shown how to offload machine learning tasks to the dataplane (reducing processing latency), online learning remains an open challenge unless the model is moved back to a host CPU, harming latency-sensitive applications. Our poster introduces OPaL---On Path Learning---the first work to bring online reinforcement learning to the dataplane. OPaL makes online learning possible in SmartNIC/NPU hardware by returning to classical RL techniques---avoiding neural networks. This simplifies update logic, enabling online learning, and benefits well from the parallelism common to SmartNICs. We show that our implementation on Netronome SmartNIC hardware offers concrete latency improvements over host execution.

References

[1]

Pat Bosshart et al. 'P4: programming protocol-independent packet processors'. In: Computer Communication Review 44.3 (2014), pp. 87--95.

Digital Library

Google Scholar

[2]

Mojgan Ghasemi et al. 'Dapper: Data Plane Performance Diagnosis of TCP'. In: Proceedings of the Symposium on SDN Research, SOSR 2017, Santa Clara, CA, USA, April 3--4, 2017. ACM, 2017, pp. 61--74.

Google Scholar

[3]

Kyle A. Simpson et al. 'Per-Host DDoS Mitigation by Direct-Control Reinforcement Learning'. In: IEEE Trans. Network and Service Management 17.1 (2020), pp. 103--117.

Digital Library

Google Scholar

[4]

Giuseppe Siracusano et al. 'Running Neural Networks on the NIC'. In: CoRR abs/2009.02353 (2020). arXiv: 2009.02353.

Google Scholar

[5]

Richard S. Sutton et al. Reinforcement Learning: An Introduction. 2nd ed. Adaptive Computation and Machine Learning. Cambridge, MA: MIT Press, Nov. 2018. ISBN: 9780262039246.

Google Scholar

[6]

Zhaoqi Xiong et al. 'Do Switches Dream of Machine Learning?: Toward In-Network Classification'. In: Proceedings of the 18th ACM Workshop on Hot Topics in Networks, HotNets 2019, Princeton, NJ, USA, November 13--15, 2019. ACM, 2019, pp. 25--33.

Google Scholar

Cited By

View all

Zheng CHong XDing DVargaftik SBen-Itzhak YZilberman N(2024)In-Network Machine Learning Using Programmable Network Devices: A SurveyIEEE Communications Surveys & Tutorials10.1109/COMST.2023.334435126:2(1171-1200)Online publication date: Oct-2025
https://doi.org/10.1109/COMST.2023.3344351
Xu WZhang ZSong HLiu SFeng YLiu B(2023)ISAC: In-Switch Approximate Cache for IoT Object Detection and RecognitionIEEE INFOCOM 2023 - IEEE Conference on Computer Communications10.1109/INFOCOM53939.2023.10229067(1-10)Online publication date: 17-May-2023
https://doi.org/10.1109/INFOCOM53939.2023.10229067

Index Terms

Online RL in the programmable dataplane with OPaL
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Reinforcement learning
2. Networks
  1. Network algorithms
    1. Data path algorithms
  2. Network services
    1. Programmable networks

Recommendations

Revisiting the Classics: Online RL in the Programmable Dataplane
NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium
Data-driven networking is becoming more capable and widely researched, partly driven by the efficacy of Deep Reinforcement Learning (DRL) algorithms. Yet the complexity of both DRL inference and learning force these tasks to be pushed away from the ...
Read More
Hardware-assisted Isolation in a Multi-tenant Function-based Dataplane
SOSR '18: Proceedings of the Symposium on SDN Research

Existing software dataplanes that run network functions inside VMs or containers can provide either performance (by dedicating CPU cores) or multiplexing (by context switching), but not both at once. Function-based dataplane architectures by replacing ...
Read More
Online RL-based cloud autoscaling for scientific workflows: Evaluation of Q-Learning and SARSA
Abstract
Q-Learning and SARSA are two well-known reinforcement learning (RL) algorithms that have shown promising results in several application domains. However, their approach to build solutions is quite different. For example, SARSA tends to be more ...
Highlights
- A comprehensive comparative study between the Q-Learning and SARSA algorithms.
- Improving performance in learning optimal scaling policies for workflows in Cloud.
- In-depth performance analysis focusing on the online aspect of the ...
Read More

Comments

Information & Contributors

Information

Published In

CoNEXT '21: Proceedings of the 17th International Conference on emerging Networking EXperiments and Technologies

December 2021

507 pages

ISBN:9781450390989

DOI:10.1145/3485983

General Chairs:
Georg Carle
Technical University of Munich, Germany
,
Jörg Ott
Technical University of Munich, Germany

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 December 2021

Check for updates

Qualifiers

Poster

Funding Sources

Engineering and Physical Sciences Research Council

Conference

CoNEXT '21

Sponsor:

SIGCOMM

CoNEXT '21: The 17th International Conference on emerging Networking EXperiments and Technologies

December 7 - 10, 2021

Virtual Event, Germany

Acceptance Rates

Overall Acceptance Rate 198 of 789 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
158
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)4

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zheng CHong XDing DVargaftik SBen-Itzhak YZilberman N(2024)In-Network Machine Learning Using Programmable Network Devices: A SurveyIEEE Communications Surveys & Tutorials10.1109/COMST.2023.334435126:2(1171-1200)Online publication date: Oct-2025
https://doi.org/10.1109/COMST.2023.3344351
Xu WZhang ZSong HLiu SFeng YLiu B(2023)ISAC: In-Switch Approximate Cache for IoT Object Detection and RecognitionIEEE INFOCOM 2023 - IEEE Conference on Computer Communications10.1109/INFOCOM53939.2023.10229067(1-10)Online publication date: 17-May-2023
https://doi.org/10.1109/INFOCOM53939.2023.10229067

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Revisiting the Classics: Online RL in the Programmable Dataplane

Hardware-assisted Isolation in a Multi-tenant Function-based Dataplane

Online RL-based cloud autoscaling for scientific workflows: Evaluation of Q-Learning and SARSA

Comments

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Revisiting the Classics: Online RL in the Programmable Dataplane

Hardware-assisted Isolation in a Multi-tenant Function-based Dataplane

Online RL-based cloud autoscaling for scientific workflows: Evaluation of Q-Learning and SARSA

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations