Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Максим Милаков (NVidia)
•
3 likes•1,205 views
Производительность инференса - одна из самых серьезных проблем при внедрении DL приложений, так как она определяет, какое впечатление от сервиса останется у конечного пользователя, а также какова будет цена внедрения этого продукта. Таким образом, для инференса важно быть высокопроизводительным и энергоэффективным. TensorRT автоматически оптимизирует обученную нейронную сеть для максимальной производительности, обеспечивая существенное ускорение по сравнению с обычными часто используемыми фреймворками. Из презентации вы узнаете, какие оптимизации применя��тся в TensorRT, как его использовать и увидите, насколько он быстр в избранных задачах.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Максим Милаков (NVidia)
Similar to Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Максим Милаков (NVidia) (20)
More from Ontico
More from Ontico (20)
Recently uploaded
Recently uploaded (20)
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Максим Милаков (NVidia)
- 1. Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT Максим Милаков, NVIDIA v2
- 2. • You will learn: • How GPUs are used for DL now? • Why do you want to use GPUs for inference? • Why do you want to use TensorRT for inference on GPUs? • This talk is NOT: • An intro to DL • A set of code samples
- 3. NVIDIA The AI Computing Company
- 4. NVIDIA Powering the Deep Learning Ecosystem DEEP LEARNING FRAMEWORKS COMPUTER VISION SPEECH AND AUDIO NATURAL LANGUAGE PROCESSING Object Detection Voice Recognition Language Translation Recommendation Engines Sentiment Analysis Mocha.jl Image Classification NVIDIA DEEP LEARNING SDK NCCLcuDNN cuBLAS cuSPARSE TensorRT
- 5. ML development and deployment cycle
- 6. Training with SGD backpropagation
- 7. ImageNet: results for 2010-2014 15% 83% 95%28% 26% 15% 11% 7% 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0% 5% 10% 15% 20% 25% 30% 2010 2011 2012 2013 2014 % Teams using GPUs Top-5 error
- 8. Deployment scenarios - Hyperscale • Input generated and output used at the client device • Inference is running at data center • High throughput • On-the-fly batching TensorRT for Hyperscale Image Classification Object Detection Image Segmentation ---
- 9. Deployment scenarios - Embedded • On-device inference • Small-batch inference • Low latency GPU Inference Engine for Automotive Pedestrian Detection Lane Tracking Traffic Sign Recognition --- NVIDIA DRIVE PX 2
- 10. TensorRT High-performance deep learning inference for production deployment 0 1 2 3 4 5 6 7 8 1 8 128 CPU-Only Tesla M4 + TensorRT Up to 16x More Inference Efficiency Img/sec/watt Batch Sizes GoogLenet, CPU-only vs Tesla M4 + TensorRT on Single-socket Haswell E5-2698 v3@2.3GHz with HT EMBEDDED Jetson TX1 AUTOMOTIVE Drive PX DATA CENTER Tesla M4
- 11. Comparing to DL frameworks • Particularly effective at small batch-sizes • Improves perf for complex networks the most Jetson TX1 HALF2 column uses fp16 GoogleNet Performance
- 12. TensorRT • Fuse network layers • Eliminate concatenation layers • Kernel specialization • Auto-tuning for target platform • Select optimal tensor layout • Batch size tuningTRAINED NEURAL NETWORK OPTIMIZED INFERENCE RUNTIME
- 13. Layers supported • v1 designed for 2D images • Layers supported • Convolution: 2D • Activation: ReLU, tanh and sigmoid • Pooling: max and average • ElementWise: sum, product or max of two tensors • LRN: cross-channel only • Fully-connected: with or without bias • SoftMax: cross-channel only • Deconvolution • Custom layers possible with sandwich approach now
- 14. Optimizations • Eliminate unused layers • Vertical layer fusion: Fuse convolution, bias, and ReLU layers to form a single layer • Horizontal layer fusion: Combine layers with the same source tensor and the same parameters
- 15. Optimizations: Original network concat max pool input next input 3x3 conv. relu bias 1x1 conv. relu bias 1x1 conv. relu bias 1x1 conv. relu bias concat 1x1 conv. relu bias 5x5 conv. relu bias
- 16. Optimizations: Vertical layer fusion concat max pool input next input concat 1x1 CBR 3x3 CBR 5x5 CBR 1x1 CBR 1x1 CBR 1x1 CBR
- 17. Optimizations: Horizontal layer fusion concat max pool input next input concat 3x3 CBR 5x5 CBR 1x1 CBR 1x1 CBR
- 18. Optimizations: Concat elision max pool input next input 3x3 CBR 5x5 CBR 1x1 CBR 1x1 CBR
- 19. TensorRT – 2 phases deployment • Build • Apply optimizations on the network configuration • Generate an optimized plan for computing the forward pass • Deploy • Forward and output the inference result Build Deploy File Model File Deploy Plan Output I/O Layers Max Batchsize Inputs Batch size
- 20. Pascal GPUs for inference Maximum Efficiency for Scale-out Servers TESLA P4 TESLA P40 Highest Throughput for Scale-up Servers
- 21. P40/P4 – New “Int8” for Inference
- 22. TensorRT v2: int8 accuracy • Almost the same accuracy for major models • Still working on the procedure to make quantization optimal
- 23. TensorRT v2: int8 performance • Up to 3x performance • No FFT and Winograd yet • Constant factors • Beneficial on small batches even on bigger P40
- 24. TensorRT v2: more performance 178 480 1,514 4,121 3,200 6,514 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 E5-2690v4 14 Core M4 (FP32) M40 (FP32) P100 (FP16) P4 (INT8) P40 (INT8) All results are measured, based on GoogLenet with batch size 128 Xeon uses MKL 2017 GOLD with FP32, GPU uses TensorRT internal development ver. >35x 1.4 12.3 10.6 27.9 91.1 56.3 0 20 40 60 80 100 E5-2690v4 14 Core M4 (FP32) M40 (FP32) P100 (FP16) P4 (INT8) P40 (INT8) >60x P40 For Max Inference Throughput, img/sec P4 For Max Inference Efficiency, img/sec/watt
- 25. Deep Learning Everywhere • developer.nvidia.com/tensorrt • developer.nvidia.com/deep-learning • developer.nvidia.com/cuda-zone • mmilakov@nvidia.com
- 26. Backup slides