1. 背景
AI在视觉领域最常用的就是人脸检测、人脸识别、活体检测、人体与行为分析、图像识别、图像增强等，而且目前都是比较成熟的技术，不论商业化的Paas平台还是开源的模型，都几乎一抓一大把。一般的，AI开发过程有以下几步：
- 特征分析
- 数据采集
- 数据标注
- 模型训练
- 模型推理
推理可以在云端也可以在客户端，端云各有各的场景，比如一般把人脸检测放到客户端，把人脸识别放到云端。本系列我们主要介绍视觉方向模型推理的工程实践。
2. 项目介绍
我们基于谷歌开源项目mediapipe提供的的模型，在客户端部署运行进行推理，mediapipe提供了一下能力：
- 人脸检测(Face Detection)
- 三维人脸网络模型(Face Mesh)
- 虹膜检测(Iris)
- 手势(Hands)
- 姿态(Pose)
- 全身姿态(Holistic)
- 头发分隔(Hair Segmentation)
- 对象检测(Object Detection)
- 物体追踪(Box Tracking)
- 即时移动检测(Instant Motion Tracking)
- Objectron
- KNIFT
- ...
mediapipe提供了bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu:handtrackinggpu编译出来即可运行。我们这里移动端开发框架我们基于开源项目github.com/terryky/and… NDK运行和测量TensorFlow Lite GPU Delegate的性能。整体基于NativeActivity框架在进行摄像头采集后画面渲染和性能数据渲染。本文我们跑通实时人脸识别模型。移动端开发框架我们基于开源项目github.com/terryky/and… NDK运行和测量TensorFlow Lite GPU Delegate的性能。整体基于NativeActivity框架在进行摄像头采集后画面渲染和性能数据渲染。本文我们跑通实时人脸识别模型。
3. 了解NativeActivity
NativeActivity是为单独使用C|C++开发app提供的基类。纯C++开发Android应用，最后还是需要一个Java层的壳子，在Android提供的开发框架中，已经使用java开发好了一个中间类，我们使用C++开发的Native库之所以能运行，就是因为被这个中间类使用JNI的方式调用了，这个中间类就是NativeActivity。这个NativeActivity类的核心功能，就是在特定事件发生时，调用我们使用C++开发的Native库里的回调函数。比如在我们熟悉的生命周期函数NativeActivity.onStart中，调用C++开发的Native库的onStartNative函数：
protected void onStart() { super.onStart(); onStartNative(mNativeHandle); }
Native层Android为我们提供了两个接口：
- native_activity.h
- android_native_app_glue.h
android_native_app_glue.h封装了native_activity.h，我们直接实现void android_main(struct android_app* state)方法即可。
NativeActivity更多具体信息可以参考Android官方文档：GameActivity | Android 开发者 | Android Developers 。
4. 运行模型
我们选择的模型：storage.googleapis.com/mediapipe-a…
- 加载模型；
- 摄像头预览纹理转换为RGBA
- 将图像数据feed到模型引擎进行推理
- 解析渲染结果
4.1 加载模型
首先我们要将模型文件读取到内存，我们的模型文件放置在Android工程的asset路径下，将文件加载到内存std::vector<uint8_t> m_tflite_model_buf;：
bool asset_read_file (AAssetManager *assetMgr, char *fname, std::vector<uint8_t>&buf) { AAsset* assetDescriptor = AAssetManager_open(assetMgr, fname, AASSET_MODE_BUFFER); if (assetDescriptor == NULL) { return false; } size_t fileLength = AAsset_getLength(assetDescriptor); buf.resize(fileLength); int64_t readSize = AAsset_read(assetDescriptor, buf.data(), buf.size()); AAsset_close(assetDescriptor); return (readSize == buf.size()); } asset_read_file (m_app->activity->assetManager, (char *)BLAZEFACE_MODEL_PATH, m_tflite_model_buf);
tflite提供了FlatBufferModel::BuildFromBuffer加载模型，返回tflite::FlatBufferModel类型的指针：
std::unique_ptr<tflite::FlatBufferModel> model = FlatBufferModel::BuildFromBuffer(model_buf, model_size)
加载完模型，通过模型创建推理引擎解释器tflite::Interpreter,tflite提供了InterpreterBuilder工具来构建tflite::Interpreter：
class InterpreterBuilder { public: InterpreterBuilder(const FlatBufferModel& model, const OpResolver& op_resolver);
需要传入模型model及OpResolver，OpResolver是个抽象接口，返回给定操作码或自定义操作名的tflite注册器。这是将flatbuffer模型中引用的操作被映射到可执行函数指针(TfLiteRegistrations)的机制。InterpreterBuilder重载了括号操作符：
TfLiteStatus operator()(std::unique_ptr<Interpreter>* interpreter); TfLiteStatus operator()(std::unique_ptr<Interpreter>* interpreter, int num_threads);
构建完InterpreterBuilder后创建tflite::Interpreter:
std::unique_ptr<tflite::FlatBufferModel> model; std::unique_ptr<tflite::Interpreter> interpreter; tflite::ops::builtin::BuiltinOpResolver resolver; InterpreterBuilder(*model, resolver)(&interpreter)
InterpreterBuilder重载的括号操作符有两个，第二个有个线程数量的参数，我们也可以通过tflite::Interpreter的SetNumThreads手动设置：
int num_threads = std::thread::hardware_concurrency(); char *env_tflite_num_threads = getenv ("FORCE_TFLITE_NUM_THREADS"); if (env_tflite_num_threads) { num_threads = atoi (env_tflite_num_threads); DBG_LOGI ("@@@@@@ FORCE_TFLITE_NUM_THREADS=%d\n", num_threads); } DBG_LOG ("@@@@@@ TFLITE_NUM_THREADS=%d\n", num_threads); interpreter->SetNumThreads(num_threads);
接下来分配tensor空间：
// Update allocations for all tensors. This will redim dependent tensors // using the input tensor dimensionality as given. This is relatively // expensive. This *must be* called after the interpreter has been created // and before running inference (and accessing tensor buffers), and *must be* // called again if (and only if) an input tensor is resized. Returns status of // success or failure. TfLiteStatus AllocateTensors();
接下来解析引擎获取模型配置（主要是输入输出张量）：
int tflite_get_tensor_by_name (std::unique_ptr<tflite::Interpreter> interpreter, int io, const char *name, tflite_tensor_t *ptensor) { memset (ptensor, 0, sizeof (*ptensor)); int tensor_idx; int io_idx = -1; int num_tensor = (io == 0) ? interpreter->inputs ().size() : interpreter->outputs().size(); for (int i = 0; i < num_tensor; i ++) { tensor_idx = (io == 0) ? interpreter->inputs ()[i] : interpreter->outputs()[i]; const char *tensor_name = interpreter->tensor(tensor_idx)->name; if (strcmp (tensor_name, name) == 0) { io_idx = i; break; } } if (io_idx < 0) { DBG_LOGE ("can't find tensor: "%s"\n", name); return -1; } void *ptr = NULL; TfLiteTensor *tensor = interpreter->tensor(tensor_idx); switch (tensor->type) { case kTfLiteUInt8: ptr = (io == 0) ? interpreter->typed_input_tensor <uint8_t>(io_idx) : interpreter->typed_output_tensor<uint8_t>(io_idx); break; case kTfLiteFloat32: ptr = (io == 0) ? interpreter->typed_input_tensor <float>(io_idx) : interpreter->typed_output_tensor<float>(io_idx); break; case kTfLiteInt64: ptr = (io == 0) ? interpreter->typed_input_tensor <int64_t>(io_idx) : interpreter->typed_output_tensor<int64_t>(io_idx); break; default: DBG_LOGE ("ERR: %s(%d)\n", __FILE__, __LINE__); return -1; } ptensor->idx = tensor_idx; ptensor->io = io; ptensor->io_idx = io_idx; ptensor->type = tensor->type; ptensor->ptr = ptr; ptensor->quant_scale = tensor->params.scale; ptensor->quant_zerop = tensor->params.zero_point; for (int i = 0; (i < 4) && (i < tensor->dims->size); i ++) { ptensor->dims[i] = tensor->dims->data[i]; } return 0; } static tflite_tensor_t s_detect_tensor_input; static tflite_tensor_t s_detect_tensor_scores; static tflite_tensor_t s_detect_tensor_bboxes; tflite_get_tensor_by_name (&s_detect_interpreter, 0, "input", &s_detect_tensor_input); tflite_get_tensor_by_name (&s_detect_interpreter, 1, "regressors", &s_detect_tensor_bboxes); tflite_get_tensor_by_name (&s_detect_interpreter, 1, "classificators", &s_detect_tensor_scores);
根据模型配置可以读取支持输入图片宽高：
int det_input_w = s_detect_tensor_input.dims[2]; int det_input_h = s_detect_tensor_input.dims[1];
4.2 摄像头预览纹理转换为RGBA
将摄像头读取的纹理数据转换成RGBA模型才能识别，我们将纹理转换为内存数据：
unsigned char *buf_ui8 = NULL; static unsigned char *pui8 = NULL; if (pui8 == NULL) pui8 = (unsigned char *)malloc(w * h * 4); buf_ui8 = pui8; draw_2d_texture_ex (srctex, 0, win_h - h, w, h, RENDER2D_FLIP_V); glPixelStorei (GL_PACK_ALIGNMENT, 4); glReadPixels (0, 0, w, h, GL_RGBA, GL_UNSIGNED_BYTE, buf_ui8);
需要想将摄像头读取的纹理绘制到帧缓存区，再通过OpenGL函数glReadPixels将纹理读取到内存缓存。
注意：glReadPixels是耗时操作
4.3 将图像数据feed到模型引擎进行推理
先通过上面获取的引起输入张量s_detect_tensor_input获取引起分配的输入缓存：
void * get_blazeface_input_buf (int *w, int *h) { *w = s_detect_tensor_input.dims[2]; *h = s_detect_tensor_input.dims[1]; return s_detect_tensor_input.ptr; }
将上面获取的图片内容转换成float，赋给输入张量：
float mean = 128.0f; float std = 128.0f; for (y = 0; y < h; y ++) { for (x = 0; x < w; x ++) { int r = *buf_ui8 ++; int g = *buf_ui8 ++; int b = *buf_ui8 ++; buf_ui8 ++; /* skip alpha */ *buf_fp32 ++ = (float)(r - mean) / std; *buf_fp32 ++ = (float)(g - mean) / std; *buf_fp32 ++ = (float)(b - mean) / std; } }
4.4 解析渲染结果
接下来调用解释器的Invoke()方法执行推理：
if (interpreter->Invoke() != kTfLiteOk) { DBG_LOGE ("ERR: %s(%d)\n", __FILE__, __LINE__); return -1; }
接下来解析检测结果：
static int decode_bounds (std::list<face_t> &face_list, float score_thresh, int input_img_w, int input_img_h) { face_t face_item; float *scores_ptr = (float *)s_detect_tensor_scores.ptr; int i = 0; for (auto itr = s_anchors.begin(); itr != s_anchors.end(); i ++, itr ++) { fvec2 anchor = *itr; float score0 = scores_ptr[i]; float score = 1.0f / (1.0f + exp(-score0)); if (score > score_thresh) { float *p = get_bbox_ptr (i); /* boundary box */ float sx = p[0]; float sy = p[1]; float w = p[2]; float h = p[3]; float cx = sx + anchor.x; float cy = sy + anchor.y; cx /= (float)input_img_w; cy /= (float)input_img_h; w /= (float)input_img_w; h /= (float)input_img_h; fvec2 topleft, btmright; topleft.x = cx - w * 0.5f; topleft.y = cy - h * 0.5f; btmright.x = cx + w * 0.5f; btmright.y = cy + h * 0.5f; face_item.score = score; face_item.topleft = topleft; face_item.btmright = btmright; /* landmark positions (6 keys) */ for (int j = 0; j < kFaceKeyNum; j ++) { float lx = p[4 + (2 * j) + 0]; float ly = p[4 + (2 * j) + 1]; lx += anchor.x; ly += anchor.y; lx /= (float)input_img_w; ly /= (float)input_img_h; face_item.keys[j].x = lx; face_item.keys[j].y = ly; } face_list.push_back (face_item); } } return 0; }
face_t封装了识别结果中的得分、左上、右下坐标：
typedef struct _face_t { float score; fvec2 topleft; fvec2 btmright; fvec2 keys[kFaceKeyNum]; } face_t;
通过坐标我们可以在识别到的“人脸”上绘制一个框：
5. 总结
本文介绍了常见的AI开发步骤，以及常用的AI视觉应用。通过人脸检测功能，了解了tensorflow lite加载模型、输入数据、执行推理、获取结果等常用接口。
