小模型也能有类o1的慢思考能力？使用CAMEL生成CoT数据、Unsloth微调Qwen2.5-1.5B模型并上传至Hugging Face
在本项目中，我们将使用CAMEL的CotDataGenerator生成高质量的问答对数据，结合Unsloth对Qwen模型进行微调，并将生成的数据和微调后的模型上传至Hugging Face平台。本文将详细介绍从数据生成到模型微调的完整流程，并分享一些实用的技巧和注意事项。
1. 项目背景与工具介绍
1.1 CAMEL-AI
CAMEL-AI
是一个开源社区，致力于寻找智能体的扩展规律。我们相信，大规模研究这些智能体可以为了解它们的行为、能力和潜在风险提供有价值的见解。为了促进该领域的研究，我们实现并支持各种类型的智能体、任务、提示、模型和模拟环境。
CAMEL
是一个强大的多智能体系统，本文主要利用它的数据合成能力：生成类似CoT（Chain-of-Thought）的思维链数据。CAMEL支持多种模型，包括Openai、qwen模型等系列，能够高效地生成多样化的训练数据。
本文的完整cookbook在线运行链接：
https://drive.google.com/file/u/1/d/1AnpeyVcYpCTa8oN1Po20fIXya-pv9A1Q/view?usp=sharing
1.3 Qwen模型
Qwen是一个开源的大语言模型，具有多种参数规模（如1.5B、3B等），适合在各种自然语言处理任务中进行微调。我们将使用Unsloth对Qwen模型进行微调，使其能够更好地理解和生成与特定任务相关的内容。
2. 环境准备
首先，我们需要在Google Colab中设置环境，并安装所需的库。以下是安装步骤：
# 安装必要的库
!pip install camel-ai==0.2.15a0
!pip install unsloth
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git
3. 设置API密钥
为了使用CAMEL和OpenAI的GPT模型，我们需要设置OpenAI的API密钥：
from getpass import getpass
import os
openai_api_key = getpass('Enter your OpenAI API key: ')
os.environ["OPENAI_API_KEY"] = openai_api_key
4. 配置ChatAgent
我们使用CAMEL的ChatAgent来生成数据。首先，定义一个系统消息来设置Agent的默认角色和行为：
sys_msg = 'You are a genius at slow-thinking data and code'
接下来，使用ModelFactory设置后端模型。CAMEL支持多种模型，这里我们使用GPT-4 Mini：
from camel.models import ModelFactory
from camel.types import ModelPlatformType, ModelType
from camel.configs import ChatGPTConfig
model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type=ModelType.GPT_4O_MINI,
model_config_dict=ChatGPTConfig().as_dict(),
)
然后，创建ChatAgent实例：
from camel.agents import ChatAgent
chat_agent = ChatAgent(
system_message=sys_msg,
model=model,
message_window_size=10,
)
5. 加载问答数据
我们从一个JSON文件中加载问答数据。首先，从GitHub获取示例数据并保存到本地：
import requests
import json
url = 'https://raw.githubusercontent.com/zjrwtx/alldata/refs/heads/main/qa_data.json'
response = requests.get(url)
if response.status_code == 200:
json_data = response.json()
file_path = 'qa_data.json'
with open(file_path, 'w', encoding='utf-8') as json_file:
json.dump(json_data, json_file, ensure_ascii=False, indent=4)
print(f"JSON data successfully saved to {file_path}")
else:
print(f"Failed to retrieve JSON file. Status code: {response.status_code}")
加载数据：
with open(file_path, 'r', encoding='utf-8') as f:
qa_data = json.load(f)
6. 生成CoT数据
我们使用CAMEL的CotDataGenerator生成CoT数据。首先，创建CotDataGenerator实例：
from camel.datagen.Cotdatagen import CotDataGenerator
testo1 = CotDataGenerator(chat_agent, golden_answers=qa_data)
然后，生成并验证问答对：
generated_answers = {
}
for question in qa_data.keys():
print(f"Question: {question}")
answer = testo1.get_answer(question)
generated_answers[question] = answer
print(f"AI's thought process and answer:\n{answer}")
is_correct = testo1.verify_answer(question, answer)
print(f"Answer verification result: {'Correct' if is_correct else 'Incorrect'}")
print("-" * 50)
7. 数据转换与保存
将生成的问答数据转换为Alpaca训练数据格式，并保存到JSON文件：
def transform_qa_format(input_file):
with open(input_file, 'r', encoding='utf-8') as f:
data = json.load(f)
transformed_data = []
for question, answer in data['qa_pairs'].items():
transformed_pair = {
"instruction": question,
"input": "",
"output": answer
}
transformed_data.append(transformed_pair)
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
output_file = f'transformed_qa_{timestamp}.json'
with open(output_file, 'w', encoding='utf-8') as f:
json.dump(transformed_data, f, ensure_ascii=False, indent=2)
return output_file, transformed_data
output_file, transformed_data = transform_qa_format(simplified_file)
print(f"Transformation complete. Output saved to: {output_file}")
8. 上传数据至Hugging Face
我们使用Hugging Face的API将生成的数据上传至平台。首先，设置Hugging Face的API密钥：
HUGGING_FACE_TOKEN = getpass('Enter your HUGGING_FACE_TOKEN: ')
os.environ["HUGGING_FACE_TOKEN"] = HUGGING_FACE_TOKEN
然后，上传数据：
username = input("Enter your HuggingFace username: ")
dataset_name = input("Enter dataset name (press Enter to use default): ").strip()
if not dataset_name:
dataset_name = None
try:
dataset_url = upload_to_huggingface(transformed_data, username, dataset_name)
print(f"\nData successfully uploaded to HuggingFace!")
print(f"Dataset URL: {dataset_url}")
except Exception as e:
print(f"Error uploading to HuggingFace: {str(e)}")
9. 模型微调
我们使用Unsloth对Qwen模型进行微调。首先，加载模型并配置LoRA：
from unsloth import FastLanguageModel
import torch
max_seq_length = 2048
dtype = None
load_in_4bit = True
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/Qwen2.5-1.5B",
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
)
model = FastLanguageModel.get_peft_model(
model,
r = 16,
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = 16,
lora_dropout = 0,
bias = "none",
use_gradient_checkpointing = "unsloth",
random_state = 3407,
use_rslora = False,
loftq_config = None,
)
然后，将数据转换为训练格式并开始训练：
from datasets import load_dataset
dataset = load_dataset("zjrwtxtechstudio/o1data06", split = "train")
dataset = dataset.map(formatting_prompts_func, batched = True,)
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text",
max_seq_length = max_seq_length,
dataset_num_proc = 2,
packing = False,
args = TrainingArguments(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 4,
warmup_steps = 5,
max_steps = 60,
learning_rate = 2e-4,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
output_dir = "outputs",
report_to = "none",
),
)
trainer_stats = trainer.train()
10. 模型推理与保存
训练完成后，我们可以使用微调后的模型进行推理：
FastLanguageModel.for_inference(model)
inputs = tokenizer(
[
alpaca_prompt.format(
"how many r in strawberry？",
"",
"",
)
],
return_tensors="pt"
).to("cuda")
outputs = model.generate(**inputs, max_new_tokens=4096, use_cache=True)
decoded_outputs = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(decoded_outputs[0])
最后，将微调后的模型保存并上传至Hugging Face：
model.save_pretrained("lora_model")
tokenizer.save_pretrained("lora_model")
model.push_to_hub("zjrwtxtechstudio/qwen2.5-1.5b-math-test", token = "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")
tokenizer.push_to_hub("zjrwtxtechstudio/qwen2.5-1.5b-math-test", token = "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")
11. 总结
通过本项目，我们展示了如何使用CAMEL生成高质量的CoT数据，并使用Unsloth对Qwen模型进行微调。整个过程涵盖了从数据生成、模型微调到模型部署的完整流程，展示了如何利用现代AI工具和平台构建和优化问答系统。
如果你对CAMEL感兴趣，欢迎在GitHub上给我们点个⭐️，并加入我们的社区！
本文的完整cookbook在线运行链接：
https://drive.google.com/file/u/1/d/1AnpeyVcYpCTa8oN1Po20fIXya-pv9A1Q/view?usp=sharing
