开发者社区 > ModelScope模型即服务 > 正文

请问ModelScope显存够了吗,够了的话怎么解决爆显存的问题?

使用vllm部署Qwen2-54B-A14B-Instruct时会出现torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.09 GiB. GPU has a total capacity of 22.03 GiB of which 540.88 MiB is free. Including non-PyTorch memory, this process has 21.50 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 197.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) ,硬件是两张A10,有48G显存,请问ModelScope显存够了吗,够了的话怎么解决爆显存的问题?

展开
收起
小小爱吃香菜 2024-06-26 08:30:40 9 0
1 条回答
写回答
取消 提交回答
  • 不够。换个小一些的模型看看。此回答整理自钉群“魔搭ModelScope开发者联盟群 ①”

    2024-06-28 19:58:50
    赞同 展开评论 打赏

ModelScope旨在打造下一代开源的模型即服务共享平台,为泛AI开发者提供灵活、易用、低成本的一站式模型服务产品,让模型应用更简单!欢迎加入技术交流群:微信公众号:魔搭ModelScope社区,钉钉群号:44837352

相关电子书

更多
视觉AI能力的开放现状及ModelScope实战 立即下载
ModelScope助力语音AI模型创新与应用 立即下载
低代码开发师(初级)实战教程 立即下载