OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like Qwen/Qwen2-VL-72B-Instruct is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

OSError: dlopen(/Users/adamsun/Library/Python/3.8/lib/python/site-packages/torch/lib/libtorch_global_deps.dylib, 0x000A): tried: '/Users/adamsun/Library/Python/3.8/lib/python/site-packages/torch/lib/libtorch_global_deps.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64)))

# 查看当前架构
arch
# 修改
arch -x86_64 zsh

# 默认是hg，如果huggingface报超时，set下这个 或者用python环境中的os.environ后再执行LLM类的初始化
export VLLM_USE_MODELSCOPE=True
from vllm import LLM
# modelscope 需要添加 trust_remote_code
llm = LLM(model=..., revision=..., task=..., trust_remote_code=True)
output = llm.generate("...")
print(output)

vllm serve /mnt/sunyf_oss_data/Qwen2-VL-7B-Instruct --limit-mm-per-prompt image=2 --dtype=half --max_model_len=976

python -m vllm.entrypoints.openai.api_server <options>

(myenv) root@dsw-677785-7c89699dd8-trs9f:/mnt/workspace# vllm serve -h
usage: vllm serve <model_tag> [options]
  --model MODEL         Name or path of the huggingface model to use.

vllm serve Qwen/Qwen2-VL-7B-Instruct --model /mnt/sunyf_oss_data/Qwen2-VL-7B-Instruct

Traceback (most recent call last):
  File "/mnt/root/miniconda/envs/myenv/bin/vllm", line 8, in <module>
    sys.exit(main())
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/scripts.py", line 165, in main
    args.dispatch_function(args)
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/scripts.py", line 37, in serve
    asyncio.run(run_server(args))
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 498, in run_server
    async with build_async_engine_client(args) as async_engine_client:
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 110, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 132, in build_async_engine_client_from_engine_args
    if (model_is_embedding(engine_args.model, engine_args.trust_remote_code,
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 73, in model_is_embedding
    return ModelConfig(model=model_name,
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/config.py", line 227, in __init__
    self.max_model_len = _get_and_verify_max_len(
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/config.py", line 1747, in _get_and_verify_max_len
    assert "factor" in rope_scaling
AssertionError

pip install git+https://github.com/huggingface/transformers@21fac7abba2a37fae86106f87fcf9974fd1e3830

(myenv) root@dsw-677785-6dc4f7589d-225hf:~# vllm serve Qwen/Qwen2.5-1.5B-Instruct
Traceback (most recent call last):
  File "/mnt/root/miniconda/envs/myenv/bin/vllm", line 5, in <module>
    from vllm.scripts import main
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/__init__.py", line 3, in <module>
    from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 11, in <module>
    from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/config.py", line 22, in <module>
    from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS,
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/model_executor/__init__.py", line 1, in <module>
    from vllm.model_executor.parameter import (BasevLLMParameter,
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/model_executor/parameter.py", line 7, in <module>
    from vllm.distributed import get_tensor_model_parallel_rank
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/distributed/__init__.py", line 1, in <module>
    from .communication_op import *
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/distributed/communication_op.py", line 6, in <module>
    from .parallel_state import get_tp_group
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 38, in <module>
    import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 15, in <module>
    from vllm.distributed.kv_transfer.kv_connector.factory import (
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 3, in <module>
    from .base import KVConnectorBase
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/distributed/kv_transfer/kv_connector/base.py", line 14, in <module>
    from vllm.sequence import IntermediateTensors
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/sequence.py", line 16, in <module>
    from vllm.inputs import SingletonInputs, SingletonInputsAdapter
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/inputs/__init__.py", line 7, in <module>
    from .registry import (DummyData, InputContext, InputProcessingContext,
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/inputs/registry.py", line 13, in <module>
    from vllm.transformers_utils.tokenizer import AnyTokenizer
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/transformers_utils/tokenizer.py", line 16, in <module>
    from vllm.utils import make_async
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/utils.py", line 46, in <module>
    from vllm.platforms import current_platform
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/platforms/__init__.py", line 100, in <module>
    from .cuda import CudaPlatform
  File "/mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/platforms/cuda.py", line 15, in <module>
    import vllm._C  # noqa
ImportError: /mnt/root/miniconda/envs/myenv/lib/python3.10/site-packages/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled

ImportError('/usr/local/lib/python3.8/dist-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj')

# 查看具体的方法
c++filt _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj
# 查找对应库的路径
pip3 show torch
# 查看对应二进制文件是否有指定方法，实现是否相同
nm -AC ./* | grep torch::Library::Library

nvidia-smi
# 查看当前环境的CUDA版本

# Install vLLM with CUDA 11.8.
export VLLM_VERSION=0.6.1.post1
export PYTHON_VERSION=310
pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118

python3 -c "import torch; print(torch.__version__)"
# 2.4.0+cu118

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/root/miniconda/envs/eas/lib/python3.10/site-packages/modelscope/utils/hf_util.py", line 351, in from_pretrained
    module_obj = module_class.from_pretrained(model_dir, *model_args,
  File "/mnt/root/miniconda/envs/eas/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3589, in from_pretrained
    raise ImportError(
ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install 'accelerate>=0.26.0'`

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/root/miniconda/envs/eas/lib/python3.10/site-packages/transformers/feature_extraction_utils.py", line 244, in to
    new_data[k] = v.to(device=device)
  File "/mnt/root/miniconda/envs/eas/lib/python3.10/site-packages/torch/cuda/__init__.py", line 319, in _lazy_init
    torch._C._cuda_init()
RuntimeError: The NVIDIA driver on your system is too old (found version 11080). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.

FROM ubuntu:22.04
LABEL maintainer="syf373586@alibaba-inc.com"
RUN chsh -s /bin/bash
SHELL ["/bin/bash", "-c"]
RUN mkdir -p /root/opt
RUN mkdir -p /root/service
COPY requirements.txt /root/service/
COPY fast_api_test01.py /root/service/
RUN apt-get update && \
apt-get install -y curl
RUN curl "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" -o /root/opt/miniconda.sh && \
/bin/bash /root/opt/miniconda.sh -b -p /root/miniconda && \
ln -s /root/miniconda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
echo ". /root/miniconda/etc/profile.d/conda.sh" >> ~/.bashrc
ENV PATH /root/miniconda/bin:/root/miniconda/condabin:$PATH
RUN conda create -n qwen2_vl python=3.10 -y
RUN echo "conda activate qwen2_vl" >> ~/.bashrc
ENV PATH /root/miniconda/envs/qwen2_vl/bin:$PATH
RUN python3 -m pip install -r /root/service/requirements.txt

# 构建镜像
# ENTRYPOINT ["python3" ,"/root/service/fast_api_test02.py"]
docker build -t sunyf:1.0 .
# 查看本地镜像
docker image ls -a
# 登录acr镜像库
docker login --username=syf373586@1623027246845796 aes-acr-enterprise-registry.cn-hangzhou.cr.aliyuncs.com
# tag
docker tag [ImageId] aes-acr-enterprise-registry.cn-hangzhou.cr.aliyuncs.com/ns_for_pai/repo_for_pai:[镜像版本号]
# push
docker push aes-acr-enterprise-registry.cn-hangzhou.cr.aliyuncs.com/ns_for_pai/repo_for_pai:[镜像版本号]

from fastapi import FastAPI
from typing import List
app = FastAPI()
from modelscope import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info
from modelscope import snapshot_download
model_dir = "/mnt/sunyf_oss_data/Qwen2-VL-7B-Instruct"
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_dir, torch_dtype="half", device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_dir)
@app.post("/sunyf_post")
def test_post(data: List[dict]):
    # Preparation for inference
    text = processor.apply_chat_template(
        data, tokenize=False, add_generation_prompt=True
    )
    image_inputs, video_inputs = process_vision_info(data)
    inputs = processor(
        text=[text],
        images=image_inputs,
        videos=video_inputs,
        padding=True,
        return_tensors="pt",
    )
    inputs = inputs.to("cuda")
    # Inference: Generation of the output
    generated_ids = model.generate(**inputs, max_new_tokens=128)
    generated_ids_trimmed = [
        out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
    ]
    output_text = processor.batch_decode(
        generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )
    return output_text

[2024-12-05 17:22:21] Uncompressing model[/home/admin/docker_ml/workspace/model] 
[2024-12-05 17:22:21] Uncompressing warm_up file[/home/admin/docker_ml/workspace/warm_up] 
[2024-12-05 17:22:21] Uncompressing processor[/home/admin/docker_ml/workspace/predictor] 
[2024-12-05 17:22:21] ENV dir not exists, please check it!

Failed to pull image "registry-vpc.cn-shanghai.aliyuncs.com/csq-yiwei/csq-yiwei:test-qwen1": [failed to pull and unpack image "registry-vpc.cn-shanghai.aliyuncs.com/csq-yiwei/csq-yiwei:test-qwen1": failed to resolve reference "registry-vpc.cn-shanghai.aliyuncs.com/csq-yiwei/csq-yiwei:test-qwen1": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed, failed to pull and unpack image "registry-vpc.cn-shanghai.aliyuncs.com/csq-yiwei/csq-yiwei:test-qwen1": failed to extract layer sha256:979f8006b89e2599920199ee287311c3ff5b35571289df7543fdb35a4cc7b3a0: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/153/fs/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcusparse.so.12.3.1.170: no space left on device: unknown]: ErrImagePull