Text2Video Huggingface Pipeline 文生视频接口和文生视频论文API

1. 背景

文生视频是AI领域热点，很多文生视频的大模型都是基于 Huggingface的 diffusers的text to video的pipeline来开发。国内外也有非常多的优秀产品如Runway AI、Pika AI 、可灵King AI、通义千问、智谱的文生视频模型等等。为了方便调用，这篇博客也尝试了使用 PyPI的text2video的python库的Wrapper类进行调用，下面会给大家介绍一下Huggingface Text to Video Pipeline的调用方式以及使用通用的text2video的python库调用方式。

2. Huggingface Text to Video Pipeline 代码

地址: (https://huggingface.co/docs/diffusers/api/pipelines/text_to_video)

  ## code for huggingface diffusion pipeline
  import torch
  from diffusers import DiffusionPipeline
  from diffusers.utils import export_to_video
  pipe = DiffusionPipeline.from_pretrained("damo-vilab/text-to-video-ms-1.7b", torch_dtype=torch.float16, variant="fp16")
  pipe = pipe.to("cuda")
  prompt = "Spiderman is surfing"
  video_frames = pipe(prompt).frames[0]
  video_path = export_to_video(video_frames)
  video_path

3. 使用Python的包Text2Video来下载最新的文本生成领域论文。

3.1 安装 pip3的 text2video的包

pip install text2video

3.2. 使用现有接口从 arxiv程序化下载最新文生视频的论文

定义输入接口，我们使用的是查询 ArxivPaper的API，需要传入 api_name 字段。同时可以设置 查询接口的 额外属性，包含拓展参数有：

可以参考arxiv的官方API

调用python的 text2video包下载最新发布在 Arxiv论文信息

import text2video as t2v
import json 
input_dict = {"text": "Text to Video"}
res = t2v.api(input_dict, model=None, api_name="ArxivPaperAPI", start=0, max_results = 3)
paper_list = json.loads(res["text"])
print ("###### Text to Image Recent Paper List:")
for (i, paper_json) in enumerate(paper_list):
    print ("|" + paper_json["id"] + "|" + paper_json["title"].replace("\n", "") + "|" + paper_json["updated"] )

输出结果

###### Text to Image Recent Paper List:

|http://arxiv.org/abs/2410.08211v1|LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts|2024-10-10T17:59:59Z

|http://arxiv.org/abs/2410.08210v1|PointOBB-v2: Towards Simpler, Faster, and Stronger Single Point  Supervised Oriented Object Detection|2024-10-10T17:59:56Z

|http://arxiv.org/abs/2410.08209v1|Emerging Pixel Grounding in Large Multimodal Models Without Grounding  Supervision|2024-10-10T17:59:55Z

3.3 自定义接口实现text2Video的API Wrapper

继承类 BaseAPI

入参

出参

output_dict| 字典| API输出的结果的dict,包含4个key text,image,audio,video字段

核心逻辑

model继承自 Huggingface的 text_to_video的 pipeline (https://huggingface.co/docs/diffusers/api/pipelines/text_to_video)

4. 相关代码库 Github和Pypi地址

https://github.com/rockingdingo/text2video

https://github.com/rockingdingo/text2audio

https://github.com/rockingdingo/image2video

https://github.com/rockingdingo/SuperAlignment

https://github.com/rockingdingo/SuperIntelligence

http://www.deepnlp.org/blog/introduction-to-text-to-video-generation-huggingface-pipeline-and-pypi-package-text2video

http://www.deepnlp.org/blog/introduction-to-multimodal-generative-models

https://huggingface.co/docs/diffusers/api/pipelines/text_to_video