Discussions

• 1 You must be logged in to vote

  🙏

  Build vllm from source with CPU support

  parkesorgua asked Jul 7, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Does vLLM supports FlashAttention?

  cwzhao asked Jul 24, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Adding custom kernels in vllm

  sjbae1999 asked Jul 23, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Sagemaker support for inference

  Tarun3679 asked Aug 25, 2023 in Q&A · Unanswered

  

  

  

  

  7
• 1 You must be logged in to vote

  🙏

  Set minimum VRAM utilization to work

  cduk asked Jul 21, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Why use manual tensor parallelism implementation instead of using something like deepspeed?

  vikigenius asked Jun 27, 2023 in Q&A · Answered

  

  

  

  

  2
• 1 You must be logged in to vote

  💬

  Build vllm without recompiling pytorch?

  prarit started Jul 19, 2024 in General · Closed

  

  

  0
• 1 You must be logged in to vote

  🙏

  When trying to scale to two nodes, I get "Error ignored in is_in_the_same_node: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] "

  carljones3000 asked Jul 19, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  good

  jiangchengchengark started Jul 19, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  💬

  Running model on multiple GPUs? How to do it? Does vllm allows it do easily?

  martinenkoEduard started Jul 17, 2024 in General

  

  

  1
• 2 You must be logged in to vote

  🙏

  Different from the output of the HF inference

  xcxhy asked Jun 27, 2023 in Q&A · Answered

  

  

  

  

  7
• 1 You must be logged in to vote

  🙏

  What cause the runtime error nvmlDeviceGetNvLinkRemoteDeviceType_

  HanweiLi-cqu asked Jul 18, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How to specify which gpu to use?

  Zhuqln asked Jul 24, 2023 in Q&A · Answered

  

  

  

  

  12
• 1 You must be logged in to vote

  🙏

  How to connect VLLM to Open WebUI?

  Kaszebe asked Jul 16, 2024 in Q&A · Unanswered

  

  

  1
• 2 You must be logged in to vote

  🙏

  gemma2-9b-instrunct how to use vllm?

  ArlanCooper asked Jul 16, 2024 in Q&A · Unanswered

  

  

  0
• 3 You must be logged in to vote

  💬

  Why does vllm flash attention build only support CUDA 12.1 (not 11.8) ?

  thangld201 started May 14, 2024 in General

  

  

  

  

  2
• 1 You must be logged in to vote

  🙏

  How to dispatch LLM across GPUs like device_map="auto" of transformers.AutoModelForCausalLM

  SlyJabiru asked Jul 12, 2024 in Q&A · Closed · Answered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Can vllm serving clients by using multiple model instances?

  aoyulong asked Jun 21, 2023 in Q&A · Answered

  

  

  

  

  5
• 7 You must be logged in to vote

  🙏

  Deploying microsoft/Phi-3-medium-128k-instruct on more than 2 GPUs

  Ala-Na asked Jun 13, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  Add support for embedding input

  Andcircle asked Jul 10, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Why does vLLM use a custom all-reduce method?

  SamKG asked Jul 5, 2024 in Q&A · Answered

  

  

  4
• 1 You must be logged in to vote

  🙏

  How to set up development environment

  vladislavkruglikov asked Jul 10, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  why max_num_batched_tokens must be <= 65528 when LoRA is enabled? can it be extended longer

  junior-zsy asked Jul 9, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  What are we caching only K and V? Why not Q?

  sathyanarays started May 24, 2024 in General · Closed

  

  

  1
• 1 You must be logged in to vote

  🙏

  Implementing Logit Functionality in vLLM

  bardia-mhd asked Nov 7, 2023 in Q&A · Answered

  

  

  

  

  4