"ModelScope的11g显存跑千问1.5-1.8怎么也够了，为什么还报错? torch.empty(kv_cache_shape,
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 252.00 MiB. GPU 0 has a total capacty of 10.75 GiB of which 231.94 MiB is free. Including non-PyTorch memory, this process has 9.54 GiB memory in use. Of the allocated memory 9.13 GiB is allocated by PyTorch, and 69.40 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 可复现的代码 #!/bin/bash

Command to run the VLLM OpenAI API server

python3 -m vllm.entrypoints.openai.api_server \
--model=/home/Qwen1.5-7b-chat/Qwen1.5-1.8B \
--served-model-name=Qwen1.5 \
--dtype=half \
--tensor-parallel-size=1 \
--trust-remote-code \
--gpu-memory-utilization=0.90 \
--host=0.0.0.0 \
--port=8001 \
--max-model-len=500 \
--max-num-seqs=1"