一、【基于PaddleHub的跳绳AI计数器】智能计数介绍
1.背景介绍
疫情以来，小朋友、大朋友们非必要不外出，都老老实实的画地为牢了。跳绳这类比起跑步等户外活动的优势一下子凸显了出来。值此佳节【六一儿童节】到来之际，我给大家弄一个通过手机视频计数的APP方案，具体如下，我们小学生也会做啦：
2.实现思路
- 1.用户打开手机APP，将手机固定在场地一侧，适当设置手机角度，根据应用的自动语音提示调整身体与手机距离，直到人体完全位于识别框内，即可开始运动。
- 2.通过PaddleHub的human_pose_estimation_resnet50_mpii模型，进行人体关键点检测。
- 3.根据检测的数据，计算是否起跳，并计数（此处选择骨盆关键点进行判断，因为跳绳有多种花式，选择其他关键点不一定能记录到），
- 4.移植该应用到手机app端
另外，【六一】活动大家可以积极相应，具体地址：
精品项目征集方向（一）-青少年主题项目征集 aistudio.baidu.com/paddle/foru…
二、环境准备
1.PaddleHub安装
!pip install -U pip --user >log.log !pip install -U paddlehub >log.log
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. parl 1.4.1 requires pyzmq==18.1.1, but you have pyzmq 22.3.0 which is incompatible.
!pip list |grep paddle
paddle2onnx 0.9.5 paddlehub 2.2.0 paddlenlp 2.0.1 paddlepaddle 2.2.2 tb-paddle 0.3.6
2.human_pose_estimation_resnet50_mpii模型安装
- 模型地址：www.paddlepaddle.org.cn/hubdetail?n…
- 模型概述：人体骨骼关键点检测(Pose Estimation) 是计算机视觉的基础性算法之一，在诸多计算机视觉任务起到了基础性的作用，如行为识别、人物跟踪、步态识别等相关领域。具体应用主要集中在智能视频监控，病人监护系统，人机交互，虚拟现实，人体动画，智能家居，智能安防，运动员辅助训练等等。 该模型的论文《Simple Baselines for Human Pose Estimation and Tracking》由 MSRA 发表于 ECCV18，使用 MPII 数据集训练完成。
!hub install human_pose_estimation_resnet50_mpii >log.log
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/sparse/sputils.py:16: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`. supported_dtypes = [np.typeDict[x] for x in supported_dtypes] /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int, /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/linalg/__init__.py:217: DeprecationWarning: The module numpy.dual is deprecated. Instead of using dual, use the functions directly from numpy or scipy. from numpy.dual import register_func /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/onnx/mapping.py:27: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations int(TensorProto.STRING): np.dtype(np.object) [2022-05-03 23:56:11,218] [ INFO] - Successfully installed human_pose_estimation_resnet50_mpii-1.1.1
!hub list|grep human
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/sparse/sputils.py:16: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`. supported_dtypes = [np.typeDict[x] for x in supported_dtypes] /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int, /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/linalg/__init__.py:217: DeprecationWarning: The module numpy.dual is deprecated. Instead of using dual, use the functions directly from numpy or scipy. from numpy.dual import register_func /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/onnx/mapping.py:27: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations int(TensorProto.STRING): np.dtype(np.object) |human_pose_estimation_res|/home/aistudio/.paddlehub/modules/human_pose_estim|
三、人体关键点检测示例
1.抓取视频
可以通过数据集中的小助手工具，下载抖音、西瓜视频、快手等小视频，特别是可以下载所需要的跳绳视频，具体演示如下：
2.关键点检测演示
针对下面这张图片做关键点检测，具体如下：
import cv2 import paddlehub as hub pose_estimation = hub.Module(name="human_pose_estimation_resnet50_mpii") image=cv2.imread('data/data143990/1.jpg') results = pose_estimation.keypoint_detection(images=[image], visualization=True)
[2022-05-03 23:56:14,417] [ WARNING] - The _initialize method in HubModule will soon be deprecated, you can use the __init__() to handle the initialization of the object W0503 23:56:14.420603 97 analysis_predictor.cc:1350] Deprecated. Please use CreatePredictor instead. image saved in output_pose/ndarray_time=1651593374766039.jpg
# 打印关键点 print(results[0]['data'])
OrderedDict([('left_ankle', [104, 376]), ('left_knee', [107, 312]),
查看output_pose 下输出的图片：
3.寻找转折点
众所周知，跳绳，人体每一帧盆骨点变化的时候算一次，那么可以通过下面方法来计算次数，具体如下：
# -*- coding: utf-8 -*- # __author__:Livingbody # 2022/5/2 21:26 import math from matplotlib import pyplot as plt import numpy as np %matplotlib inline def get_count(y): count = 0 flag = False for i in range(len(y)-1): if y[i] < y[i + 1] and flag == False: continue elif y[i] > y[i + 1] and flag == True: continue else: print(i, y[i]) count = count + 1 flag = not flag return count def get_count(y): count = 0 flag = False for i in range(len(y)-1): if y[i] < y[i + 1] and flag == False: continue elif y[i] > y[i + 1] and flag == True: continue else: print(i, y[i]) count = count + 1 flag = not flag return count if __name__ == '__main__': x = np.linspace(-10, 10, 500) y = np.sin(x * math.pi) # print(x, y) plt.figure(figsize=(12, 12)) count = get_count(y) plt.title(f"point numbers: {math.floor(count/2)}") plt.plot(x, y) plt.show()
12 0.9982119222410882 37 -0.9985684620536466 62 0.9988854219322505 87 -0.999162789313664 112 0.9994005532039664 137 -0.9995987041789873 162 0.999757234384681 187 -0.9998761375374373 212 0.9999554089243304 237 -0.9999950454033061 262 0.9999950454033061 287 -0.9999554089243304 312 0.9998761375374373 337 -0.9997572343846811 362 0.9995987041789874 387 -0.9994005532039665 412 0.999162789313664 437 -0.9988854219322507 462 0.9985684620536466 487 -0.9982119222410885 /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working if isinstance(obj, collections.Iterator): /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working return list(data) if isinstance(data, collections.MappingView) else data
如上图所示，可以算出该曲线里有10次最高点，该方法可运用到跳绳计数中去。
四、跳绳检测
# -*- coding: utf-8 -*- # __author__:Livingbody # 2022/5/2 17:49 import cv2 import paddlehub as hub if __name__ == "__main__": pose_estimation = hub.Module(name="human_pose_estimation_resnet50_mpii") flag = False count = 0 flip_list = [] # 可选择web视频流或者文件 file_name = 'data/data143990/2.mp4' cap = cv2.VideoCapture(file_name) fourcc = cv2.VideoWriter_fourcc(*"mp4v") # out后期可以合成视频返回 out = cv2.VideoWriter( file_name.replace(".mp4", "_output.mp4"), fourcc, 20.0, (int(cap.get(3)), int(cap.get(4))), ) while cap.isOpened(): success, image = cap.read() if not success: break image_height, image_width, _ = image.shape image.flags.writeable = False results = pose_estimation.keypoint_detection(images=[image], visualization=True) # debug打印，可关闭 # print(results) # print(results[0]['data']) # print(results[0]['data']['pelvis']) # print(results[0]['data']['left_hip']) # print(results[0]['data']['right_hip']) flip = results[0]['data']['pelvis'][1] flip_list.append(flip) # 如果超过2个点就开始比较 if len(flip_list) < 2: continue prev_flip = flip_list[len(flip_list) - 2] # 开始进行判断计数 if flip < prev_flip and flag == False: continue elif flip > prev_flip and flag == True: continue else: # print(flip) count = count + 1 flag = not flag print(int(count/2))
输出示例：
image saved in output_pose/ndarray_time=1651593377511581.jpg image saved in output_pose/ndarray_time=1651593378354247.jpg 0 image saved in output_pose/ndarray_time=1651593378987333.jpg 1 image saved in output_pose/ndarray_time=1651593379606857.jpg 1 image saved in output_pose/ndarray_time=1651593380213530.jpg image saved in output_pose/ndarray_time=1651593380818746.jpg image saved in output_pose/ndarray_time=1651593381474900.jpg 2 image saved in output_pose/ndarray_time=1651593382093679.jpg 2 image saved in output_pose/ndarray_time=1651593382712164.jpg 3 image saved in output_pose/ndarray_time=1651593383325455.jpg 3
五、下步打算
该项目抛砖引玉，供大家参考。
- 加入可视化，实时显示关键点检测以及跳绳计数结果；
- 拟采取paddlelite终端部署，或者paddlehub服务器部署调用，结合android客户端、或者电脑加摄像头实现实现方式来验证；
- 数据下载软件已上传数据集，大家也可以动手试试。
