开发者社区> 1040603911771223> 正文
阿里云
为了无法计算的价值
打开APP
阿里云APP内打开

教你如何玩转Modelscope (一、图片编辑与描述生成)

简介: 本文通过一个有趣的图片处理例子,教你如何利用modelscope强大且多样的模型能力去解锁你专属的图片编辑能力。
+关注继续查看

简介:

本文将会用到2个视觉类相关的模型:达摩卡通化模型达摩人像抠图模型分别对一张图片进行处理,并最终利用一个多模态的模型:达摩图片英文描述生成模型,对处理前后的图片进行描述,让你一键解锁快速图片编辑以及图片标题的能力。


操作:

参考快速开始,里面有一些关于ModelScope的基本介绍


环境准备:

  • 为了更快的体验产品,使用了ModelScope提供的远程环境,即Notebook进行开发,更加便捷。
  • 如果本地测试,也可以先安装依赖:
# 深度学习依赖
pip install torch torchvision torchaudio
pip install --upgrade tensorflow

# modelscope模型依赖
pip install "modelscope[cv]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
pip install "modelscope[multi-modal]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

模型准备:

本文涉及到的模型有3个,包括:

达摩卡通化模型:https://modelscope.cn/#/models/damo/cv_unet_person-image-cartoon_compound-models/summary

达摩人像抠图模型: https://modelscope.cn/#/models/damo/cv_unet_image-matting/summary

以及达摩图片英文描述生成模型: https://modelscope.cn/#/models/damo/ofa_image-caption_coco_large_en/summary


前两个模型是cv模型因此安装 modelscope[cv] ,第三个是涉及到图片和文本的多模态模型因此需要安装 modelscope[multi-modal] 。



模型调试:

首先我们选择一张美丽好看的图片作为这次处理的目标:https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png

image_cartoon.png

达摩卡通化模型,将图片卡通化:

import cv2
from modelscope.hub.snapshot_download import snapshot_download
from modelscope.pipelines import pipeline


model_dir = snapshot_download('damo/cv_unet_person-image-cartoon_compound-models', cache_dir='.')
img_cartoon = pipeline('image-portrait-stylization', model=model_dir)

result = img_cartoon('https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png')

cv2.imwrite('result.png', result['output_img'])
print('finished!')

从notebook中获得如下结果

2022-08-11 15:01:43,132 - modelscope - INFO - File .DS_Store already in cache, skip downloading!
2022-08-11 15:01:43,134 - modelscope - INFO - File README.md already in cache, skip downloading!
2022-08-11 15:01:43,136 - modelscope - INFO - File alpha.jpg already in cache, skip downloading!
2022-08-11 15:01:43,139 - modelscope - INFO - File cartoon_anime_bg.pb already in cache, skip downloading!
2022-08-11 15:01:43,141 - modelscope - INFO - File cartoon_anime_h.pb already in cache, skip downloading!
2022-08-11 15:01:43,143 - modelscope - INFO - File configuration.json already in cache, skip downloading!
2022-08-11 15:01:43,147 - modelscope - INFO - File demo.gif already in cache, skip downloading!
2022-08-11 15:01:43,149 - modelscope - INFO - File detector.pb already in cache, skip downloading!
2022-08-11 15:01:43,151 - modelscope - INFO - File keypoints.pb already in cache, skip downloading!
2022-08-11 15:01:43,155 - modelscope - INFO - initiate model from ./damo/cv_unet_person-image-cartoon_compound-models
2022-08-11 15:01:43,156 - modelscope - INFO - initiate model from location ./damo/cv_unet_person-image-cartoon_compound-models.
2022-08-11 15:01:43,523 - modelscope - INFO - loading model from ./damo/cv_unet_person-image-cartoon_compound-models/cartoon_anime_h.pb
2022-08-11 15:01:43,583 - modelscope - INFO - load model ./damo/cv_unet_person-image-cartoon_compound-models/cartoon_anime_h.pb done.
2022-08-11 15:01:43,584 - modelscope - INFO - loading model from ./damo/cv_unet_person-image-cartoon_compound-models/cartoon_anime_bg.pb
2022-08-11 15:01:43,638 - modelscope - INFO - load model ./damo/cv_unet_person-image-cartoon_compound-models/cartoon_anime_bg.pb done.
finished!

可以在notebook任务栏中查看图片结果

image.png

image.png

是不是很有趣?


达摩人像抠图模型,将人像抠出来

接着上面的步骤,我们获得如下抠出来的结果

img_matting = pipeline(Tasks.portrait_matting,model='damo/cv_unet_image-matting')
result = img_matting('https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png')
cv2.imwrite('result1.png', result[OutputKeys.OUTPUT_IMG])
2022-08-11 15:12:03,172 - modelscope - INFO - File README.md already in cache, skip downloading!
2022-08-11 15:12:03,174 - modelscope - INFO - File configuration.json already in cache, skip downloading!
2022-08-11 15:12:03,176 - modelscope - INFO - File tf_graph.pb already in cache, skip downloading!
2022-08-11 15:12:03,181 - modelscope - INFO - initiate model from /mnt/workspace/.cache/modelscope/damo/cv_unet_image-matting
2022-08-11 15:12:03,181 - modelscope - INFO - initiate model from location /mnt/workspace/.cache/modelscope/damo/cv_unet_image-matting.
2022-08-11 15:12:03,190 - modelscope - INFO - loading model from /mnt/workspace/.cache/modelscope/damo/cv_unet_image-matting/tf_graph.pb
2022-08-11 15:12:03,444 - modelscope - INFO - load model done
True

result1.png


至此,我们有一张图的三种不同的样式了,最后送到我们的英文描述模型里看看效果比较一下吧。

达摩图片英文描述生成模型

注意,由于是多模态模型比较大,所以第一次使用的时候下载速度可能会比较慢,稍等几分钟即可,包括推理时间也较为长,

img_captioning = pipeline(
            Tasks.image_captioning,
            model='damo/ofa_image-caption_coco_large_en')
image_list = [
    {'image': 'result.png'},
    {'image': 'result1.png'},
    {'image': 'https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png'}
]
result = img_captioning(image_list)
print(result)
2022-08-11 15:40:03,932 - modelscope - INFO - File README.md already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File README.md already in cache, skip downloading!
2022-08-11 15:40:03,933 - modelscope - INFO - File config.json already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File config.json already in cache, skip downloading!
2022-08-11 15:40:03,934 - modelscope - INFO - File configuration.json already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File configuration.json already in cache, skip downloading!
2022-08-11 15:40:03,934 - modelscope - INFO - File merges.txt already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File merges.txt already in cache, skip downloading!
2022-08-11 15:40:03,935 - modelscope - INFO - File pytorch_model.bin already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File pytorch_model.bin already in cache, skip downloading!
2022-08-11 15:40:03,936 - modelscope - INFO - File leaderboard.png already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File leaderboard.png already in cache, skip downloading!
2022-08-11 15:40:03,936 - modelscope - INFO - File ofa_caption_score.png already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File ofa_caption_score.png already in cache, skip downloading!
2022-08-11 15:40:03,937 - modelscope - INFO - File ofa_frame.png already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File ofa_frame.png already in cache, skip downloading!
2022-08-11 15:40:03,938 - modelscope - INFO - File ofa_image_captioning_en.png already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File ofa_image_captioning_en.png already in cache, skip downloading!
2022-08-11 15:40:03,938 - modelscope - INFO - File tokenizer.json already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File tokenizer.json already in cache, skip downloading!
2022-08-11 15:40:03,939 - modelscope - INFO - File vocab.json already in cache, skip downloading!
2022-08-11 15:40:03 | INFO | modelscope | File vocab.json already in cache, skip downloading!
2022-08-11 15:40:03,945 - modelscope - INFO - initiate model from /mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en
2022-08-11 15:40:03 | INFO | modelscope | initiate model from /mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en
2022-08-11 15:40:03,946 - modelscope - INFO - initiate model from location /mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en.
2022-08-11 15:40:03 | INFO | modelscope | initiate model from location /mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en.
2022-08-11 15:40:03,951 - modelscope - INFO - initialize model from /mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en
2022-08-11 15:40:03 | INFO | modelscope | initialize model from /mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en
2022-08-11 15:40:12,260 - modelscope - INFO - initialize model from /mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en
2022-08-11 15:40:12 | INFO | modelscope | initialize model from /mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
[{'caption': 'a woman standing in front of a car with her hand out', 'samples': [{'image': 'result.png'}], 'text': None, 'boxes': None, 'labels': None, 'scores': None}, {'caption': 'a woman standing next to a car with her hand outstretched', 'samples': [{'image': 'result1.png'}], 'text': None, 'boxes': None, 'labels': None, 'scores': None}, {'caption': 'a woman standing next to a car with her hand outstretched', 'samples': [{'image': 'https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png'}], 'text': None, 'boxes': None, 'labels': None, 'scores': None}]


结果:

比对下来原图的描述为:

a woman standing next to a car with her hand outstretched

动画图的描述为:

a woman standing in front of a car with her hand out

抠图后的描述为:

a woman standing next to a car with her hand outstretched


惊奇的发现抠图的结果居然和原图的描述一样,这里可以看出抠图确实真真实实的扣出了人像,但是还保留原图的信息。用户甚至可以在图片软件中还原出来原图。


同时,动画效果展示也不错的情况下,描述也惊人的和原图一致,只是手部动作描述略有不同。


看完了这个操作,是不是感觉很有意思?要不要打开你的脑洞一起来玩呦~


版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
【玩转Modelscope】我XXX没有开挂!
本文介绍了一种使用达摩院人体2D关键点检测模型对图片进行处理,识别出图片中人体关键点的方法。此模型可应用于射击游戏中,在达摩院强大图像识别能力的加持下,结合自动截屏和移动鼠标等辅助功能,让你成为射击游戏里的“神枪手”!
28298 0
玩转CSS3基础——过渡与动画
CSS3新增了过渡属性,可以从一个状态变化到另一个状态时,变化更加地平滑。 CSS3的过渡功能就像德芙巧克力那般,如丝一般顺滑,口感很好,哈哈🙆‍♀️
44 0
您有一份数智化商业机密待查收!
关于实体商业数字化和流量运营的秘密,建议偷偷打开......
394 0
1
文章
0
问答
文章排行榜
最热
最新
相关电子书
更多
低代码开发师(初级)实战教程
立即下载
阿里巴巴DevOps 最佳实践手册
立即下载
冬季实战营第三期:MySQL数据库进阶实战
立即下载