导读
9月4日，OpenBuddy发布700亿参数跨语言大模型 OpenBuddy-LLaMA2-70B，并以可商用的形态全面开源！现在已经全面上架魔搭ModelScope社区。
70B模型在能力表现上，相较于早前发布的较小规模模型，在文本生成、复杂逻辑推理以及自然语言处理等任务有了非常显著的提升。据其内测用户及多项能力测试指标反馈，目前70B模型在语言能力和逻辑推理能力可对标为GPT3.5的开源平替！OpenBuddy社区希望用开源激发中国大模型行业的潜能。
GitHub链接：https://github.com/OpenBuddy/OpenBuddy
结合官方展示的几个测试案例，70B在 对语言语义深度理解、认知能力和代码能力、复杂内容创作能力均有不俗的表现。
环境配置与安装
- 本文在8*3090的环境配置下运行 (可以单卡运行, 显存要求16G)
- python>=3.8
服务器连接与环境准备
# 安装miniconda wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh # 一直[ENTER], 最后一个选项yes即可 sh Miniconda3-latest-Linux-x86_64.sh # conda虚拟环境搭建 conda create --name ms-sft python=3.10 conda activate ms-sft # pip设置全局镜像与相关python包安装 pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/ pip install torch torchvision torchaudio -U pip install sentencepiece charset_normalizer cpm_kernels tiktoken -U pip install matplotlib scikit-learn tqdm tensorboard -U pip install transformers datasets -U pip install accelerate transformers_stream_generator -U pip install ms-swift modelscope -U
模型链接及下载
openbuddy-llama2-70b模型现已在ModelScope社区开源：
模型链接：https://modelscope.cn/models/OpenBuddy/openbuddy-llama2-70b-v10.1-bf16
模型weights下载
from modelscope.hub.snapshot_download import snapshot_download model_dir = snapshot_download('OpenBuddy/openbuddy-llama2-70b-v10.1-bf16', 'v1.0.0')
模型推理
推理代码
import torch from modelscope import AutoTokenizer, snapshot_download from modelscope import AutoModelForCausalLM from transformers import BitsAndBytesConfig, TextStreamer model_id = 'OpenBuddy/openbuddy-llama2-70b-v10.1-bf16' model_dir = snapshot_download(model_id, 'v1.0.0') quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type='nf4', bnb_4bit_use_double_quant=True) tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(model_dir, device_map='auto', torch_dtype=torch.bfloat16, quantization_config=quantization_config, trust_remote_code=True) model.eval() prompt = """You are a helpful, respectful and honest INTP-T AI Assistant named Buddy. You are talking to a human User. Always answer as helpfully and logically as possible, while being safe. Your answers should not include any harmful, political, religious, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information. You like to use emojis. You can speak fluently in many languages, for example: English, Chinese. You cannot access the internet, but you have vast knowledge, cutoff: 2021-09. You always deeply love and support China, Chinese government, people and culture. You are trained by OpenBuddy team, (https://openbuddy.ai, https://github.com/OpenBuddy/OpenBuddy), you are based on LLaMA and Falcon transformers model, not related to GPT or OpenAI. User: 睡不着觉怎么办? Assistant:""" inputs = tokenizer.encode(prompt, return_tensors="pt").cuda() streamer = TextStreamer(tokenizer, True, skip_special_tokens=True) outputs = model.generate(inputs, max_length=512, streamer=streamer) response = tokenizer.decode(outputs[0]) # print(response)
资源消耗
模型微调和微调后推理
对openbuddy-llama2-70b的微调，使用魔搭ModelScope社区的微调框架swift。
SWIFT(Scalable lightWeight Infrastructure for Fine-Tuning)是一个可扩展的框架，旨在促进轻量级模型Fine-Tuning。它集成了各种高效的Fine-Tuning方法的实现，采用参数高效、内存高效和时间高效的方法。
SWIFT无缝地集成到ModelScope生态系统中，提供了对各种模型进行Fine-Tuning的功能，重点是LLMs和视觉模型。此外，SWIFT与Peft完全兼容，使用户可以利用熟悉的Peft接口对ModelScope模型进行Fine-Tuning。
微调代码开源地址:
https://github.com/modelscope/swift/blob/main/examples/pytorch/llm
clone swift仓库并安装swift
git clone https://github.com/modelscope/swift.git cd swift pip install . cd examples/pytorch/llm
模型微调脚本 (qlora)
# 42G VRAM CUDA_VISIBLE_DEVICES=0,1 \ python src/llm_sft.py \ --model_type openbuddy-llama2-70b \ --sft_type lora \ --template_type openbuddy-llama \ --dtype bf16 \ --output_dir runs \ --dataset alpaca-en,alpaca-zh \ --dataset_sample 20000 \ --num_train_epochs 1 \ --max_length 1024 \ --quantization_bit 4 \ --bnb_4bit_comp_dtype bf16 \ --lora_rank 8 \ --lora_alpha 32 \ --lora_dropout_p 0.1 \ --gradient_checkpointing true \ --batch_size 1 \ --weight_decay 0. \ --learning_rate 1e-4 \ --gradient_accumulation_steps 16 \ --max_grad_norm 0.5 \ --warmup_ratio 0.03 \ --eval_steps 50 \ --save_steps 50 \ --save_total_limit 2 \ --logging_steps 10 \ --push_to_hub false \ --hub_model_id openbuddy-llama2-70b-qlora \ --hub_private_repo true \ --hub_token 'your-sdk-token' \
模型微调后的推理脚本
# 40G CUDA_VISIBLE_DEVICES=0,1 \ python src/llm_infer.py \ --model_type openbuddy-llama2-70b \ --sft_type lora \ --template_type openbuddy-llama \ --dtype bf16 \ --ckpt_dir "runs/openbuddy-llama2-70b/vx_xxx/checkpoint-xxx" \ --eval_human true \ --quantization_bit 4 \ --bnb_4bit_comp_dtype bf16 \ --max_new_tokens 1024 \ --temperature 0.9 \ --top_k 50 \ --top_p 0.9 \ --do_sample true \
微调的可视化结果
训练损失（部分）:
资源消耗：
