Implementing Machine Learning at State Departments of Transportation: A Guide (2024)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

NCHRP Research Report 1122 Pre-Publication Draft— Subject to Revision Implementing Machine Learning at State Departments of Transportation A GUIDE Haley Townsend Matthew Samach Kelly Bare Noblis Washington, DC Mecit Cetin Sherif Ishak Old Dominion University Norfolk, VA Kaan Ozbay Cognium LLC, Princeton, NJ Submitted April 2024 DISCLAIMER The opinions and conclusions expressed or implied in this document are those of the researchers who performed the research. They are not necessarily those of the program sponsors; the FHWA; the Transportation Research Board; or the National Academies of Sciences, Engineering, and Medi- cine. The information contained in this document was taken directly from the submission of the au- thors. This material has not been edited by the Transportation Research Board. SPECIAL NOTE: This document IS NOT an official publication of the Transportation Research Board or the National Academies of Sciences, Engineering, and Medicine. A final, edited version of this document will be released at a later date. 

© 2024 by the National Academy of Sciences. National Academies of Sciences, Engineering, and Medicine and the graphical logo are trademarks of the National Academy of Sciences. All rights reserved. NATIONAL COOPERATIVE HIGHWAY RESEARCH PROGRAM Systematic, well-designed, and implementable research is the most effective way to solve many problems facing state departments of transportation (DOTs) administrators and engineers. Often, highway problems are of local or regional interest and can best be studied by state DOTs individually or in cooperation with their state universities and others. However, the accelerating growth of highway transportation results in increasingly complex problems of wide interest to highway authorities. These problems are best studied through a coordinated program of cooperative research. Recognizing this need, the leadership of the American Association of State Highway and Transportation Officials (AASHTO) in 1962 initiated an objective national highway research program using modern scientific techniques— the National Cooperative Highway Research Program (NCHRP). NCHRP is supported on a continuing basis by funds from participating member states of AASHTO and receives the full cooperation and support of the Federal Highway Administration (FHWA), United States Department of Transportation, under Agreement No. 693JJ31950003. COPYRIGHT INFORMATION Authors herein are responsible for the authenticity of their materials and for obtaining written permissions from publishers or persons who own the copyright to any previously published or copyrighted material used herein. Cooperative Research Programs (CRP) grants permission to reproduce material in this publication for classroom and not-for-profit purposes. Permission is given with the understanding that none of the material will be used to imply endorsement by TRB and any of its program sponsors of a particular product, method, or practice. It is expected that those reproducing the material in this document for educational and not-for-profit uses will give appropriate acknowledgment of the source of any reprinted or reproduced material. For other uses of the material, request permission from CRP. DISCLAIMER To facilitate more timely dissemination of research findings, this pre-publication document is taken directly from the submission of the research agency. The material has not been edited by TRB. The opinions and conclusions expressed or implied in this document are those of the researchers who performed the research. They are not necessarily those of the Transportation Research Board; the National Academies of Sciences, Engineering, and Medicine; the FHWA; or the program sponsors. The Transportation Research Board does not develop, issue, or publish standards or specifications. The Transportation Research Board manages applied research projects which provide the scientific foundation that may be used by Transportation Research Board sponsors, industry associations, or other organizations as the basis for revised practices, procedures, or specifications. The Transportation Research Board, the National Academies, and the sponsors of the National Cooperative Highway Research Program do not endorse products or manufacturers. Trade or manufacturers’ names appear herein solely because they are considered essential to the object of the report. This pre-publication document IS NOT an official publication of the Cooperative Research Programs; the Transportation Research Board; or the National Academies of Sciences, Engineering, and Medicine. Recommended citation: Townsend, H., M. Samach, K. Bare, M. Cetin, S. Ishak, and K. Ozbay. 2024. Implementing Machine Learning at State Departments of Transportation: A Guide. Pre-publication draft of NCHRP Research Report 1122. Transportation Research Board, Washington, DC. 

CONTENTS INTRODUCTION ..................................................................................................................................... 1 Target Audience of the Guide ........................................................................................................ 2 Purpose of the Guide ..................................................................................................................... 2 Guide Key Takeaways ..................................................................................................................... 2 ROADMAP TO BUILDING AGENCY MACHINE LEARNING CAPABILITIES............................. 4 Overview of the Roadmap ............................................................................................................. 4 Step 0: Develop Understanding ..................................................................................................... 7 Introduction to ML Approach .................................................................................................... 7 Learning from Data .................................................................................................................... 9 Trends in Machine Learning .................................................................................................... 10 Interpretability of ML Models .................................................................................................. 12 Step 1: Identify Candidate Use Cases .......................................................................................... 13 Decision Gate #1 .......................................................................................................................... 17 Step 2: Assess Gaps ...................................................................................................................... 20 Data .......................................................................................................................................... 21 Data Storage ............................................................................................................................. 26 Cybersecurity ........................................................................................................................... 28 Computing ................................................................................................................................ 29 Workforce and Organizational Considerations ....................................................................... 31 Funding ..................................................................................................................................... 33 Other Considerations ............................................................................................................... 34 Step 3: Build a Business Case ....................................................................................................... 36 Defining the Opportunity ......................................................................................................... 36 Benefit-Cost Analysis................................................................................................................ 36 Communicating the Business Case .......................................................................................... 38 Decision Gate #2 .......................................................................................................................... 40 Step 4: Plan Pilot .......................................................................................................................... 43 

ML Pilot Initial Planning ........................................................................................................... 44 ML Pilot Schedule ..................................................................................................................... 48 ML Pilot Costs ........................................................................................................................... 49 Step 5: Execute Pilot .................................................................................................................... 51 Step 6: Communicate Results ...................................................................................................... 54 Performance Metrics ............................................................................................................... 55 Human Supervision of ML Systems ......................................................................................... 58 Assessment Reporting ............................................................................................................. 59 Lessons Learned ....................................................................................................................... 60 Step 7: Scale Deployment ............................................................................................................ 62 Data Availability and Consistency ............................................................................................ 62 Data Distribution and Bias ....................................................................................................... 63 Generalization/Transferability ................................................................................................. 63 System Integration ................................................................................................................... 63 New Costs ................................................................................................................................. 64 Step 8: Operations & Maintenance ............................................................................................. 66 Step 9: Expand Agency ML Capabilities ....................................................................................... 68 Enterprise AI Strategy .............................................................................................................. 68 Enterprise Data Strategy .......................................................................................................... 70 Workforce ................................................................................................................................ 73 Intra- and Inter-Agency Collaboration ..................................................................................... 74 References.................................................................................................................................... 77 Glossary of Key Terms .................................................................................................................. 80 

Acknowledgments The authors would like to acknowledge individuals who contributed to this research and its final products. From Noblis, Meenakshy Vasudevan and Karl Wunderlich served as advisors and provided critical input on the guide and the project overall. From Old Dominion University, Behrouz Salahshour helped with the literature review and compiling the terms for the glossary. The authors would also like to thank the survey respondents and case study interviewees for providing critical inputs to this guide and the final research report. Their insights and lessons learned will help future machine learning deployers at transportation agencies.