Reconciling Accuracy, Cost, and Latency of Inference Serving Systems

Transcript

1. Reconciling Accuracy, Cost, and Latency of Inference Serving Systems Pooyan

   Jamshidi https://pooyanjamshidi.github.io/ University of South Carolina S
2. None

3. Multi-Objective Optimization with Known Constraints under Uncertainty Solutions: InfAdapter [2023]:

   Autoscaling for ML Inference IPA [2024]: Autoscaling for ML Inference Pipeline Sponge [2024]: Autoscaling for ML Inference Pipeline Dynamic SLO Problem: Different Assumptions

4. InfAdapter [2023]: Autoscaling for ML Model Inference IPA [2024]: Autoscaling

   for ML Inference Pipeline Sponge [2024]: Autoscaling for ML Inference Pipeline with Dynamic SLO

5. Quality adaptation 5 ResNet18: Tiger ResNet152: Dog

6. InfAdapter: Why? Different throughputs with different model variants 6

7. InfAdapter: Why? Higher average accuracy by using multiple model variants

   7

8. InfAdapter: How? 8 Selecting a subset of model variants, each

   having its size meeting latency requirements for the predicted workload while maximizing accuracy and minimizing resource cost

9. InfAdapter: Formulation 9

10. InfAdapter: Formulation 10 Maximizing Average Accuracy

11. InfAdapter: Formulation 11 Maximizing Average Accuracy Minimizing Resource and Loading

    Costs

12. InfAdapter: Formulation 12

13. InfAdapter: Formulation 13 Supporting incoming workload

14. InfAdapter: Formulation 14 Supporting incoming workload Guaranteeing end-to-end latency

15. InfAdapter: Design 15

16. InfAdapter: Experimental evaluation setup Workload: Twitter-trace sample (2022-08) Baselines: Kubernetes

    VPA and Model-Switching Used models: Resnet18, Resnet34, Resnet50, Resnet101, Resnet152 Interval adaptation: 30 seconds Kubernetes cluster: 48 Cores, 192 GiB RAM 16

17. Workload Pattern 17

18. InfAdapter: P99-Latency evaluation 18

19. InfAdapter: P99-Latency evaluation 19

20. InfAdapter: P99-Latency evaluation 20

21. InfAdapter: P99-Latency evaluation 21

22. InfAdapter: P99-Latency evaluation 22

23. InfAdapter: Accuracy evaluation 23

24. 24 InfAdapter: Cost evaluation

25. InfAdapter: Tradeoff Space 25

26. Takeaway 26 Inference Serving Systems should consider accuracy, latency, and

    cost at the same time.

27. Takeaway 27 Model variants provide the opportunity to reduce resource

    costs while adapting to the dynamic workload. Using a set of model variants simultaneously provides higher average accuracy compared to having one variant. Inference Serving Systems should consider accuracy, latency, and cost at the same time.

28. Takeaway 28 Model variants provide the opportunity to reduce resource

    costs while adapting to the dynamic workload. Using a set of model variants simultaneously provides higher average accuracy compared to having one variant. Inference Serving Systems should consider accuracy, latency, and cost at the same time. InfAdapter!

29. InfAdapter [2023]: Autoscaling for ML Model Inference IPA [2024]: Autoscaling

    for ML Inference Pipeline Sponge [2024]: Autoscaling for ML Inference Pipeline with Dynamic SLO

30. Inference Pipeline 30 Video Decoder Stream Muxer Primary Detector Object

    Tracker Secondary Classifier # Configuration Options 55 86 14 44 86

31. Is only scaling enough? ?

32. How to navigate the Accuracy/Latency trade-off space? Previous works, INFaaS

    and Model-Switch, have proven that there is a big latency- accuracy-resource footprint tradeoff of models trained for the same task.

33. 33 Goal: Providing a flexible inference pipeline

34. 34 Snapshot of the System

35. Search Space

36. Problem Formulation Accuracy Objective Resource Objective Batch Control

37. Problem Formulation Latency SLA Throughput Constraint One active model per

    node

38. 38 Evaluation https://github.com/reconfigurable-ml-pipeline/ipa

39. 39 Audio + QA Pipeline

40. 40 Adaptivity to multiple objectives

41. Full replication package is available https://github.com/recon fi gurable-ml-pipeline

42. Model Serving Pipeline Is only scaling enough? ? X Snapshot

    of the System X Adaptivity to multiple objectives

43. InfAdapter [2023]: Autoscaling for ML Model Inference IPA [2024]: Autoscaling

    for ML Inference Pipeline Sponge [2024]: Autoscaling for ML Inference Pipeline with Dynamic SLO

44. Dynamic User -> Dynamic Network Bandwidths ˻ Users move ˻

    Fluctuations in the network bandwidths ˻ Reduced time-budget for processing requests 44 SLO network latency processing latency

45. Dynamic User -> Dynamic Network Bandwidths ˻ Users move ˻

    Fluctuations in the network bandwidths ˻ Reduced time-budget for processing requests 45 SLO network latency processing latency

46. Inference Serving Requirements 46 Highly Responsive! Cost-Efficient! Resource Scaling In-place

    Vertical Scaling Horizontal Scaling (end-to-end latency guarantee) (least resource consumption) (more responsive) (more cost efficient) Sponge!

47. Vertical Scaling DL Model Profiling ˻ How much resource should

    be allocated to a DL model? ˻ Latency/batch size → linear relationship ˻ Latency/CPU allocation → inverse relationship 47

48. Problem Formulation 48

49. Problem Formulation 49 Minimize resource costs

50. Problem Formulation 50 Limit the batch size to grow infinitely!

    Minimize resource costs

51. Problem Formulation 51 Limit the batch size to grow infinitely!

    Minimize resource costs

52. 3 design choices: 1. In-place vertical scaling • Fast response

    time 2. Request reordering • High priority requests 3. Dynamic batching • Increase system utilization 52 System Design

53. Evaluation SLO guarantees (99th percentile) with up to 20% resource

    save up compared to static resource allocation. Sponge source code: https://github.com/saeid93/sponge 53

54. Future Directions 54 Resource Scaling In-place Vertical Scaling Horizontal Scaling

    (more responsive) (more cost efficient) Sponge! How can both scaling mechanisms be used jointly under a dynamic workload to be responsive and cost efficient while guaranteeing SLOs?

55. Performance goals are competing and users have preferences over these

    goals The variability space (design space) of (composed) systems is exponentially increasing Systems operate in uncertain environments with imperfect and incomplete knowledge Goal: Enabling users to f ind the right quality tradeoff Lander Testbed (NASA) Turtlebot 3 (UofSC) Husky UGV (UofSC) CoBot (CMU)

56. Thank you, Saeid Ghafouri! 56