使用vllm部署Qwen2-54B-A14B-Instruct时会出现torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.09 GiB. GPU has a total capacity of 22.03 GiB of which 540.88 MiB is free. Including non-PyTorch memory, this process has 21.50 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 197.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) ,硬件是两张A10,有48G显存,请问ModelScope显存够了吗,够了的话怎么解决爆显存的问题?
ModelScope旨在打造下一代开源的模型即服务共享平台,为泛AI开发者提供灵活、易用、低成本的一站式模型服务产品,让模型应用更简单!欢迎加入技术交流群:微信公众号:魔搭ModelScope社区,钉钉群号:44837352