请问ModelScope显存够了吗,够了的话怎么解决爆显存的问题?

使用vllm部署Qwen2-54B-A14B-Instruct时会出现torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.09 GiB. GPU has a total capacity of 22.03 GiB of which 540.88 MiB is free. Including non-PyTorch memory, this process has 21.50 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 197.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) ,硬件是两张A10,有48G显存,请问ModelScope显存够了吗,够了的话怎么解决爆显存的问题?

展开
收起
小小爱吃香菜 2024-06-26 08:30:40 181 分享 版权
1 条回答
写回答
取消 提交回答
  • 不够。换个小一些的模型看看。此回答整理自钉群“魔搭ModelScope开发者联盟群 ①”

    2024-06-28 19:58:50
    赞同 展开评论

ModelScope旨在打造下一代开源的模型即服务共享平台,为泛AI开发者提供灵活、易用、低成本的一站式模型服务产品,让模型应用更简单!欢迎加入技术交流群:微信公众号:魔搭ModelScope社区,钉钉群号:44837352

还有其他疑问?
咨询AI助理