技术背景
我们在对接GB28181设备接入模块的时候，遇到这样的技术诉求，好多开发者期望能提供编码后（H.264/H.265、AAC/PCMA）数据对接，确保外部采集设备，比如无人机类似回调过来的数据，直接通过模块，对接到GB28181平台侧，此外，还期望不支持或者内网没有外部网络权限的RTSP设备，也能间接接入到国标平台。
技术实现
编码后音视频数据
本文以Android平台为例，基于上述诉求，我们设计的接口如下，简单来说，GB28181交互流程不变，只要提供数据接入接口即可：
/** * 设置编码后视频数据(H.264) * * @param codec_id, H.264对应 1 * * @param data 编码后的video数据 * * @param size data length * * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0. * * @param timestamp video timestamp * * @param pts Presentation Time Stamp, 显示时间戳 * * @return {0} if successful */ public native int SmartPublisherPostVideoEncodedData(long handle, int codec_id, ByteBuffer data, int size, int is_key_frame, long timestamp, long pts); /** * 设置编码后视频数据(H.264) * * @param codec_id, H.264对应 1 * * @param data 编码后的video数据 * *@param offset data的偏移 * * @param size data length * * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0. * * @param timestamp video timestamp * * @param pts Presentation Time Stamp, 显示时间戳 * * @return {0} if successful */ public native int SmartPublisherPostVideoEncodedDataV2(long handle, int codec_id, ByteBuffer data, int offset, int size, int is_key_frame, long timestamp, long pts, byte[] sps, int sps_len, byte[] pps, int pps_len); /** * 设置编码后视频数据(H.264)，如需录制编码后的数据，用此接口，且设置实际宽高 * * @param codec_id, H.264对应 1 * * @param data 编码后的video数据 * *@param offset data的偏移 * * @param size data length * * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0. * * @param timestamp video timestamp * * @param pts Presentation Time Stamp, 显示时间戳 * * @param width, height: 编码后视频宽高 * * @return {0} if successful */ public native int SmartPublisherPostVideoEncodedDataV3(long handle, int codec_id, ByteBuffer data, int offset, int size, int is_key_frame, long timestamp, long pts, byte[] sps, int sps_len, byte[] pps, int pps_len, int width, int height); /** * 设置音频数据(AAC/PCMA/PCMU/SPEEX) * * @param codec_id: * * NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000, * NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE, * NT_MEDIA_CODEC_ID_PCMU, * NT_MEDIA_CODEC_ID_AAC, * NT_MEDIA_CODEC_ID_SPEEX, * NT_MEDIA_CODEC_ID_SPEEX_NB, * NT_MEDIA_CODEC_ID_SPEEX_WB, * NT_MEDIA_CODEC_ID_SPEEX_UWB, * * @param data audio数据 * * @param size data length * * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略 * * @param timestamp video timestamp * * @param parameter_info 用于AAC special config信息填充 * * @param parameter_info_size parameter info size * * @return {0} if successful */ public native int SmartPublisherPostAudioEncodedData(long handle, int codec_id, ByteBuffer data, int size, int is_key_frame, long timestamp,ByteBuffer parameter_info, int parameter_info_size); /** * 设置音频数据(AAC/PCMA/PCMU/SPEEX) * * @param codec_id: * * NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000, * NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE, * NT_MEDIA_CODEC_ID_PCMU, * NT_MEDIA_CODEC_ID_AAC, * NT_MEDIA_CODEC_ID_SPEEX, * NT_MEDIA_CODEC_ID_SPEEX_NB, * NT_MEDIA_CODEC_ID_SPEEX_WB, * NT_MEDIA_CODEC_ID_SPEEX_UWB, * * @param data audio数据 * * @param offset data的偏移 * * @param size data length * * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略 * * @param timestamp video timestamp * * @param parameter_info 用于AAC special config信息填充 * * @param parameter_info_size parameter info size * * @return {0} if successful */ public native int SmartPublisherPostAudioEncodedDataV2(long handle, int codec_id, ByteBuffer data, int offset, int size, int is_key_frame, long timestamp, byte[] parameter_info, int parameter_info_size); /** * 设置音频数据(AAC/PCMA/PCMU/SPEEX) * * @param codec_id: * * NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000, * NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE, * NT_MEDIA_CODEC_ID_PCMU, * NT_MEDIA_CODEC_ID_AAC, * NT_MEDIA_CODEC_ID_SPEEX, * NT_MEDIA_CODEC_ID_SPEEX_NB, * NT_MEDIA_CODEC_ID_SPEEX_WB, * NT_MEDIA_CODEC_ID_SPEEX_UWB, * * @param data audio数据 * * @param offset data的偏移 * * @param size data length * * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略 * * @param timestamp video timestamp * * @param parameter_info 用于AAC special config信息填充 * * @param parameter_info_size parameter info size * * @param sample_rate 采样率,如果需要录像的话必须传正确的值 * *@param channels 通道数, 如果需要录像的话必须传正确的值, 一般是1或者2 * * @return {0} if successful */ public native int SmartPublisherPostAudioEncodedDataV3(long handle, int codec_id, ByteBuffer data, int offset, int size, int is_key_frame, long timestamp, byte[] parameter_info, int parameter_info_size, int sample_rate, int channels);
拉取RTSP流接入到GB28181平台
简单那来说，把摄像机的RTSP流数据拉下来，然后回调编码后的数据到上层，上层根据GB28181数据格式要求，实现PS打包，然后通过对接GB28181平台信令和数据交互，国标平台侧需要预览的时候，信令交互后，拉RTSP即可。
如何拉流：
private boolean StartPull() { if ( isPulling ) return false; if (!OpenPullHandle()) return false; libPlayer.SmartPlayerSetAudioDataCallback(playerHandle, new PlayerAudioDataCallback()); libPlayer.SmartPlayerSetVideoDataCallback(playerHandle, new PlayerVideoDataCallback()); int is_pull_trans_code = 1; libPlayer.SmartPlayerSetPullStreamAudioTranscodeAAC(playerHandle, is_pull_trans_code); int startRet = libPlayer.SmartPlayerStartPullStream(playerHandle); if (startRet != 0) { Log.e(TAG, "Failed to start pull stream!"); if(!isPlaying && !isRecording && isPushing && !isRTSPPublisherRunning) { libPlayer.SmartPlayerClose(playerHandle); playerHandle = 0; } return false; } isPulling = true; return true; } private void StopPull() { if ( !isPulling ) return; libPlayer.SmartPlayerStopPullStream(playerHandle); if ( !isPlaying && !isRecording && !isPushing && !isRTSPPublisherRunning) { libPlayer.SmartPlayerClose(playerHandle); playerHandle = 0; } isPulling = false; }
拉到的音视频数据，投递到GB28181接入模块：
class PlayerAudioDataCallback implements NTAudioDataCallback { private int audio_buffer_size = 0; private int param_info_size = 0; private ByteBuffer audio_buffer_ = null; private ByteBuffer parameter_info_ = null; @Override public ByteBuffer getAudioByteBuffer(int size) { //Log.i("getAudioByteBuffer", "size: " + size); if( size < 1 ) { return null; } if ( size <= audio_buffer_size && audio_buffer_ != null ) { return audio_buffer_; } audio_buffer_size = size + 512; audio_buffer_size = (audio_buffer_size+0xf) & (~0xf); audio_buffer_ = ByteBuffer.allocateDirect(audio_buffer_size); // Log.i("getAudioByteBuffer", "size: " + size + " buffer_size:" + audio_buffer_size); return audio_buffer_; } @Override public ByteBuffer getAudioParameterInfo(int size) { //Log.i("getAudioParameterInfo", "size: " + size); if(size < 1) { return null; } if ( size <= param_info_size && parameter_info_ != null ) { return parameter_info_; } param_info_size = size + 32; param_info_size = (param_info_size+0xf) & (~0xf); parameter_info_ = ByteBuffer.allocateDirect(param_info_size); //Log.i("getAudioParameterInfo", "size: " + size + " buffer_size:" + param_info_size); return parameter_info_; } public void onAudioDataCallback(int ret, int audio_codec_id, int sample_size, int is_key_frame, long timestamp, int sample_rate, int channel, int parameter_info_size, long reserve) { if ( audio_buffer_ == null) return; audio_buffer_.rewind(); if ( ret == 0 && (isPushing || isRTSPPublisherRunning || isGB28181StreamRunning)) { libPublisher.SmartPublisherPostAudioEncodedData(publisherHandle, audio_codec_id, audio_buffer_, sample_size, is_key_frame, timestamp, parameter_info_, parameter_info_size); } } } class PlayerVideoDataCallback implements NTVideoDataCallback { private int video_buffer_size = 0; private ByteBuffer video_buffer_ = null; @Override public ByteBuffer getVideoByteBuffer(int size) { if( size < 1 ) { return null; } if ( size <= video_buffer_size && video_buffer_ != null ) { return video_buffer_; } video_buffer_size = size + 1024; video_buffer_size = (video_buffer_size+0xf) & (~0xf); video_buffer_ = ByteBuffer.allocateDirect(video_buffer_size); return video_buffer_; } public void onVideoDataCallback(int ret, int video_codec_id, int sample_size, int is_key_frame, long timestamp, int width, int height, long presentation_timestamp) { if ( video_buffer_ == null) return; video_buffer_.rewind(); if ( ret == 0 && (isPushing || isRTSPPublisherRunning || isGB28181StreamRunning) ) { libPublisher.SmartPublisherPostVideoEncodedData(publisherHandle, video_codec_id, video_buffer_, sample_size, is_key_frame, timestamp, presentation_timestamp); } } }
如何预览播放外部音视频数据？
除了想把编码后的音视频数据转至GB28181外，有些场景下，还需要本地预览甚至对数据做二次处理（视频分析、实时水印字符叠加等，然后二次编码），基于这样的场景诉求，我们实现了Android平台外部编码数据实时预览播放模块。
外部（H.264/H.265）投递接口设计如下：
// SmartPlayerJniV2.java // Author: daniusdk.com /** * 投递视频包给外部Live Source * * @param codec_id: 编码id, 当前仅支持H264和H265, 1:H264, 2:H265 * * @param packet: 视频数据, ByteBuffer必须是DirectBuffer, 包格式请参考H264/H265 Annex B Byte stream format, 例如: * 0x00000001 nal_unit 0x00000001 ... * H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit .... * H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit .... * * @param offset: 偏移量 * @param size: packet size * @param timestamp_ms: 时间戳, 单位毫秒 * @param is_timestamp_discontinuity: 是否时间戳间断，0:未间断，1:间断 * @param is_key: 是否是关键帧, 0:非关键帧, 1:关键帧 * @param extra_data: 可选参数，可传null, 对于H264关键帧包, 如果packet不含sps和pps, 可传0x00000001 sps 0x00000001 pps * ,对于H265关键帧包, 如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps * @param extra_data_size: extra_data size * @param width: 图像宽, 可传0 * @param height: 图像高, 可传0 * * @return {0} if successful */ public native int PostVideoPacketByteBuffer(long handle, int codec_id, java.nio.ByteBuffer packet, int offset, int size, long timestamp_ms, int is_timestamp_discontinuity, int is_key, byte[] extra_data, int extra_data_size, int width, int height); /* * 请参考 PostVideoPacketByteBuffer说明 */ public native int PostVideoPacketByteArray(long handle, int codec_id, byte[] packet, int offset, int size, long timestamp_ms, int is_timestamp_discontinuity, int is_key, byte[] extra_data, int extra_data_size, int width, int height);
PostVideoPacketByteBuffer()和PostVideoPacketByteArray()接口设计基本类似，唯一的区别在于，一个数据类型是ByteBuffer，一个是byte数组。
其中codec_id，系编码id，目前仅支持H.264和H.265类型。
packet视频数据，需要注意的是，ByteBuffer必须是DirectBuffer, 包格式请参考H264/H265 Annex B Byte stream format, 例如:
0x00000001 nal_unit 0x00000001 ... H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit .... H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
extra_data: 可选参数，可传null, 对于H264关键帧包，如果packet不含sps和pps，可传0x00000001 sps 0x00000001 pps，对于H265关键帧包，如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps
音频（AAC/PCMA/PCMU）投递接口设计如下：
/** * 投递音频包给外部Live source, 注意ByteBuffer对象必须是DirectBuffer * * @param handle: return value from SmartPlayerOpen() * * @param codec_id: 编码id, 当前支持PCMA、PCMU和AAC, 65536:PCMA, 65537:PCMU, 65538:AAC * @param packet: 音频数据 * @param offset：packet偏移量 * @param size: packet size * @param pts_ms: 时间戳, 单位毫秒 * @param is_pts_discontinuity: 是否时间戳间断，false:未间断，true:间断 * @param extra_data: 如果是AAC的话，需要传 Audio Specific Configuration * @param extra_data_offset: extra_data 偏移量 * @param extra_data_size: extra_data size * @param sample_rate: 采样率 * @param channels: 通道数 * * @return {0} if successful */ public native int PostAudioPacket(long handle, int codec_id, java.nio.ByteBuffer packet, int offset, int size, long pts_ms, boolean is_pts_discontinuity, java.nio.ByteBuffer extra_data, int extra_data_offset, int extra_data_size, int sample_rate, int channels); /* * 投递音频包给外部Live source, byte数组版本, 具体请参考PostAudioPacket * * @param is_pts_discontinuity: 是否时间戳间断，0:未间断，1:间断 * @return {0} if successful */ public native int PostAudioPacketByteArray(long handle, int codec_id, byte[] packet, int offset, int size, long pts_ms, int is_pts_discontinuity, byte[] extra_data, int extra_data_size, int sample_rate, int channels);
总结
通过以上描述，大家可以看到，GB/T 28181音视频数据源接入，无论是编码前还是编码后数据，或外部RTSP流数据，包括数据预览，如果有技术积累的话，实现起来也没那么麻烦，感兴趣的开发者，可以尝试看。
