[AI MoA] 原来这就是 MoA(Mixture-of-Agents)

今天看到一个项目，Mixture-of-Agents (MoA)，打开了一个新思路。

介绍

Mixture-of-Agents (MoA) 是一种新颖的方法，它利用多个 LLMs 的集体优势来增强性能，实现最先进的结果。通过采用每层包含多个 LLM 代理的分层架构，MoA 仅使用开源模型，在 AlpacaEval 2.0 上的得分为 65.1%，显着优于 GPT-4 Omni 的 57.5%！

我们看下架构图

快速开始

pip install together
export TOGETHER_API_KEY=...
python moa.py

评估

准备

# install requirements
pip install -r requirements.txt
cd alpaca_eval
pip install -e .
cd FastChat
pip install -e ".[model_worker,llm_judge]"
cd ..

# setup api keys
export TOGETHER_API_KEY=<TOGETHER_API_KEY>
export OPENAI_API_KEY=<OPENAI_API_KEY>

AlpacaEval 2

bash run_eval_alpaca_eval.sh

MT-Bench

bash run_eval_mt_bench.sh

FLASK

bash run_eval_flask.sh

结果

从结果上看，效果不错。

总结

MoA

上面的介绍文字，看的不一定清楚。看下代码，就都清楚了。它做的事

• 用户提问，异步调用各种 llm 给结果。
  • 这里用到的模型是 Qwen/Qwen2-72B-Instruct, Qwen/Qwen1.5-72B-Chat, mistralai/Mixtral-8x22B-Instruct-v0.1, databricks/dbrx-instruct ，你可以根据需要自己配
  • 异步并发同时请求各 llm
• 拿到各 llm 的结果后，请求 llm 总结各 llm 的回答，给个好的结果
  • 这里配置的模型是 mistralai/Mixtral-8x22B-Instruct-v0.1 ，同样可以自己改

就这样了，原来这就是 MoA。再去看上面的架构图，很清晰。

这里用的 together api 。通过上面的介绍，思路清晰了，那你自己根据这个思路，写一个，也是很简单了。你可以随便换各种 llm 。

# Mixture-of-Agents in 50 lines of code
import asyncio
import os
from together import AsyncTogether, Together

client = Together(api_key=os.environ.get("TOGETHER_API_KEY"))
async_client = AsyncTogether(api_key=os.environ.get("TOGETHER_API_KEY"))

user_prompt = "What are some fun things to do in SF?"
reference_models = [
    "Qwen/Qwen2-72B-Instruct",
    "Qwen/Qwen1.5-72B-Chat",
    "mistralai/Mixtral-8x22B-Instruct-v0.1",
    "databricks/dbrx-instruct",
]
aggregator_model = "mistralai/Mixtral-8x22B-Instruct-v0.1"
aggreagator_system_prompt = """You have been provided with a set of responses from various open-source models to the latest user query. Your task is to synthesize these responses into a single, high-quality response. It is crucial to critically evaluate the information provided in these responses, recognizing that some of it may be biased or incorrect. Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. Ensure your response is well-structured, coherent, and adheres to the highest standards of accuracy and reliability.

Responses from models:"""


async def run_llm(model):
    """Run a single LLM call with a reference model."""
    response = await async_client.chat.completions.create(
        model=model,
        messages=[{
   "role": "user", "content": user_prompt}],
        temperature=0.7,
        max_tokens=512,
    )
    print(model)
    return response.choices[0].message.content


async def main():
    results = await asyncio.gather(*[run_llm(model) for model in reference_models])

    finalStream = client.chat.completions.create(
        model=aggregator_model,
        messages=[
            {
   "role": "system", "content": aggreagator_system_prompt},
            {
   "role": "user", "content": ",".join(str(element) for element in results)},
        ],
        stream=True,
    )

    for chunk in finalStream:
        print(chunk.choices[0].delta.content or "", end="", flush=True)


asyncio.run(main())

评估

这个项目中还有一点可以说的，就是评估。效果到底好不好，可以通过评估看。

怎么做评估？看下面这个图，从各个指标上去打分，最后看得分。

各个指标计算的 代码在这里

数据集在这里 2.5K 条人工注释

• github
• AI 博客 - 从零开始学AI
• 公众号 - 从零开始学AI