简介：
本文将会用到2个视觉类相关的模型：达摩卡通化模型和达摩人像抠图模型分别对一张图片进行处理，并最终利用一个多模态的模型：达摩图片英文描述生成模型，对处理前后的图片进行描述，让你一键解锁快速图片编辑以及图片标题的能力。
操作：
参考快速开始，里面有一些关于ModelScope的基本介绍
环境准备：
- 为了更快的体验产品，使用了ModelScope提供的远程环境，即Notebook进行开发，更加便捷。
- 如果本地测试，也可以先安装依赖：
# 深度学习依赖pip install torch torchvision torchaudio pip install --upgrade tensorflow # modelscope模型依赖pip install "modelscope[cv]"-f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html pip install "modelscope[multi-modal]"-f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
模型准备：
本文涉及到的模型有3个，包括：
达摩卡通化模型：https://modelscope.cn/#/models/damo/cv_unet_person-image-cartoon_compound-models/summary
达摩人像抠图模型： https://modelscope.cn/#/models/damo/cv_unet_image-matting/summary
以及达摩图片英文描述生成模型： https://modelscope.cn/#/models/damo/ofa_image-caption_coco_large_en/summary
前两个模型是cv模型因此安装 modelscope[cv] ，第三个是涉及到图片和文本的多模态模型因此需要安装 modelscope[multi-modal] 。
模型调试：
首先我们选择一张美丽好看的图片作为这次处理的目标：https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png
达摩卡通化模型，将图片卡通化：
importcv2frommodelscope.hub.snapshot_downloadimportsnapshot_downloadfrommodelscope.pipelinesimportpipelinemodel_dir=snapshot_download('damo/cv_unet_person-image-cartoon_compound-models', cache_dir='.') img_cartoon=pipeline('image-portrait-stylization', model=model_dir) result=img_cartoon('https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png') cv2.imwrite('result.png', result['output_img']) print('finished!')
从notebook中获得如下结果
2022-08-1115:01:43,132-modelscope-INFO-File .DS_Storealreadyincache, skipdownloading!2022-08-1115:01:43,134-modelscope-INFO-FileREADME.mdalreadyincache, skipdownloading!2022-08-1115:01:43,136-modelscope-INFO-Filealpha.jpgalreadyincache, skipdownloading!2022-08-1115:01:43,139-modelscope-INFO-Filecartoon_anime_bg.pbalreadyincache, skipdownloading!2022-08-1115:01:43,141-modelscope-INFO-Filecartoon_anime_h.pbalreadyincache, skipdownloading!2022-08-1115:01:43,143-modelscope-INFO-Fileconfiguration.jsonalreadyincache, skipdownloading!2022-08-1115:01:43,147-modelscope-INFO-Filedemo.gifalreadyincache, skipdownloading!2022-08-1115:01:43,149-modelscope-INFO-Filedetector.pbalreadyincache, skipdownloading!2022-08-1115:01:43,151-modelscope-INFO-Filekeypoints.pbalreadyincache, skipdownloading!2022-08-1115:01:43,155-modelscope-INFO-initiatemodelfrom ./damo/cv_unet_person-image-cartoon_compound-models2022-08-1115:01:43,156-modelscope-INFO-initiatemodelfromlocation ./damo/cv_unet_person-image-cartoon_compound-models. 2022-08-1115:01:43,523-modelscope-INFO-loadingmodelfrom ./damo/cv_unet_person-image-cartoon_compound-models/cartoon_anime_h.pb2022-08-1115:01:43,583-modelscope-INFO-loadmodel ./damo/cv_unet_person-image-cartoon_compound-models/cartoon_anime_h.pbdone. 2022-08-1115:01:43,584-modelscope-INFO-loadingmodelfrom ./damo/cv_unet_person-image-cartoon_compound-models/cartoon_anime_bg.pb2022-08-1115:01:43,638-modelscope-INFO-loadmodel ./damo/cv_unet_person-image-cartoon_compound-models/cartoon_anime_bg.pbdone. finished!
可以在notebook任务栏中查看图片结果
是不是很有趣？
达摩人像抠图模型，将人像抠出来
接着上面的步骤，我们获得如下抠出来的结果
img_matting=pipeline(Tasks.portrait_matting,model='damo/cv_unet_image-matting') result=img_matting('https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png') cv2.imwrite('result1.png', result[OutputKeys.OUTPUT_IMG])
2022-08-1115:12:03,172-modelscope-INFO-FileREADME.mdalreadyincache, skipdownloading!2022-08-1115:12:03,174-modelscope-INFO-Fileconfiguration.jsonalreadyincache, skipdownloading!2022-08-1115:12:03,176-modelscope-INFO-Filetf_graph.pbalreadyincache, skipdownloading!2022-08-1115:12:03,181-modelscope-INFO-initiatemodelfrom/mnt/workspace/.cache/modelscope/damo/cv_unet_image-matting2022-08-1115:12:03,181-modelscope-INFO-initiatemodelfromlocation/mnt/workspace/.cache/modelscope/damo/cv_unet_image-matting. 2022-08-1115:12:03,190-modelscope-INFO-loadingmodelfrom/mnt/workspace/.cache/modelscope/damo/cv_unet_image-matting/tf_graph.pb2022-08-1115:12:03,444-modelscope-INFO-loadmodeldoneTrue
至此，我们有一张图的三种不同的样式了，最后送到我们的英文描述模型里看看效果比较一下吧。
达摩图片英文描述生成模型
注意，由于是多模态模型比较大，所以第一次使用的时候下载速度可能会比较慢，稍等几分钟即可，包括推理时间也较为长，
img_captioning=pipeline( Tasks.image_captioning, model='damo/ofa_image-caption_coco_large_en') image_list= [ {'image': 'result.png'}, {'image': 'result1.png'}, {'image': 'https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png'} ] result=img_captioning(image_list) print(result)
2022-08-1115:40:03,932-modelscope-INFO-FileREADME.mdalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|FileREADME.mdalreadyincache, skipdownloading!2022-08-1115:40:03,933-modelscope-INFO-Fileconfig.jsonalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Fileconfig.jsonalreadyincache, skipdownloading!2022-08-1115:40:03,934-modelscope-INFO-Fileconfiguration.jsonalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Fileconfiguration.jsonalreadyincache, skipdownloading!2022-08-1115:40:03,934-modelscope-INFO-Filemerges.txtalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Filemerges.txtalreadyincache, skipdownloading!2022-08-1115:40:03,935-modelscope-INFO-Filepytorch_model.binalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Filepytorch_model.binalreadyincache, skipdownloading!2022-08-1115:40:03,936-modelscope-INFO-Fileleaderboard.pngalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Fileleaderboard.pngalreadyincache, skipdownloading!2022-08-1115:40:03,936-modelscope-INFO-Fileofa_caption_score.pngalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Fileofa_caption_score.pngalreadyincache, skipdownloading!2022-08-1115:40:03,937-modelscope-INFO-Fileofa_frame.pngalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Fileofa_frame.pngalreadyincache, skipdownloading!2022-08-1115:40:03,938-modelscope-INFO-Fileofa_image_captioning_en.pngalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Fileofa_image_captioning_en.pngalreadyincache, skipdownloading!2022-08-1115:40:03,938-modelscope-INFO-Filetokenizer.jsonalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Filetokenizer.jsonalreadyincache, skipdownloading!2022-08-1115:40:03,939-modelscope-INFO-Filevocab.jsonalreadyincache, skipdownloading!2022-08-1115:40:03|INFO|modelscope|Filevocab.jsonalreadyincache, skipdownloading!2022-08-1115:40:03,945-modelscope-INFO-initiatemodelfrom/mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en2022-08-1115:40:03|INFO|modelscope|initiatemodelfrom/mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en2022-08-1115:40:03,946-modelscope-INFO-initiatemodelfromlocation/mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en. 2022-08-1115:40:03|INFO|modelscope|initiatemodelfromlocation/mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en. 2022-08-1115:40:03,951-modelscope-INFO-initializemodelfrom/mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en2022-08-1115:40:03|INFO|modelscope|initializemodelfrom/mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en2022-08-1115:40:12,260-modelscope-INFO-initializemodelfrom/mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_en2022-08-1115:40:12|INFO|modelscope|initializemodelfrom/mnt/workspace/.cache/modelscope/damo/ofa_image-caption_coco_large_enTruncationwasnotexplicitlyactivatedbut `max_length` isprovidedaspecificvalue, pleaseuse `truncation=True` toexplicitlytruncateexamplestomaxlength. Defaultingto'longest_first'truncationstrategy. Ifyouencodepairsofsequences (GLUE-style) withthetokenizeryoucanselectthisstrategymorepreciselybyprovidingaspecificstrategyto `truncation`. [{'caption': 'a woman standing in front of a car with her hand out', 'samples': [{'image': 'result.png'}], 'text': None, 'boxes': None, 'labels': None, 'scores': None}, {'caption': 'a woman standing next to a car with her hand outstretched', 'samples': [{'image': 'result1.png'}], 'text': None, 'boxes': None, 'labels': None, 'scores': None}, {'caption': 'a woman standing next to a car with her hand outstretched', 'samples': [{'image': 'https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_cartoon.png'}], 'text': None, 'boxes': None, 'labels': None, 'scores': None}]
结果：
比对下来原图的描述为：
a woman standing next to a car with her hand outstretched
动画图的描述为：
a woman standing in front of a car with her hand out
抠图后的描述为：
a woman standing next to a car with her hand outstretched
惊奇的发现抠图的结果居然和原图的描述一样，这里可以看出抠图确实真真实实的扣出了人像，但是还保留原图的信息。用户甚至可以在图片软件中还原出来原图。
同时，动画效果展示也不错的情况下，描述也惊人的和原图一致，只是手部动作描述略有不同。
看完了这个操作，是不是感觉很有意思？要不要打开你的脑洞一起来玩呦～
