开发者社区> daniulive> 正文
阿里云
为了无法计算的价值
打开APP
阿里云APP内打开

如何在Unity3d平台下低延迟播放RTMP或RTSP流

简介:
+关注继续查看

随着VR类、游戏类场景的快速发展,开发者对Unity3d低延迟的直播需求量越来越大,前两年,大牛直播SDK发布了Windows平台、Android平台和iOS平台的Unity3d RTMP和RTSP的播放,好多公司用起来体验都非常好,以下介绍大概实现流程。

本文以Android平台为例,我们的实现:基于大牛直播SDK现有非常成熟的native RTMP和RTSP播放模块,回调解码后的原始数据,传递给Unity3d,实现相应的绘制即可,对应demo,可以参考 Github。

具体步骤如下:

  1. Native RTSP或RTSP直播播放SDK回调RGB/YUV420/NV12等其中的一种未压缩的图像格式;
  2. Unity3D创建相应的RGB/YUV420等Shader;

3.Unity3D从各个平台获取图像数据来填充纹理即可。

以Android平台为例,我们在原有接口基础上,做了桥接接口,供Unity平台使用:

/// <summary>
/// Init
/// </summary>
public int NT_U3D_Init()
{
    return DANIULIVE_RETURN_OK;
}

/// <summary>
/// 开始
/// 返回播放句柄
/// </summary>
public long NT_U3D_Open()
{
    if ( 0 != player_obj_.Call<int>("Init", java_obj_cur_activity_) )
    {
        return 0;
    }

    return player_obj_.Call<long>("Open");
}

/// <summary>
/// Register Game Object,用于消息传递
/// </summary>
public int NT_U3D_Set_Game_Object(long handle, string gameObjectName)
{
    return player_obj_.Call<int>("SetGameObject", handle, gameObjectName);
}

/// <summary>
/// 设置H.264解码方式 false 软件解码 true 硬件解码 默认为false
/// </summary>
/// <param name="isHwDecoder"></param>
public int NT_U3D_SetVideoDecoderMode(long handle, int isHwDecoder)
{
    return player_obj_.Call<int>("SetPlayerVideoHWDecoder", handle, isHwDecoder);
}

/// <summary>
/// 设置H.265 解码方式 false 软件解码 true 硬件解码 默认为false
/// </summary>
/// <param name="isHevcHwDecoder"></param>
public int NT_U3D_SetVideoHevcDecoderMode(long handle, int isHevcHwDecoder)
{
    return player_obj_.Call<int>("SetPlayerVideoHevcHWDecoder", handle, isHevcHwDecoder);
}

/// <summary>
/// 设置音频输出模式: if 0: 自动选择; if with 1: audiotrack模式
/// </summary>
/// <param name="use_audiotrack"></param>
public int NT_U3D_SetAudioOutputType(long handle, int use_audiotrack)
{
    return player_obj_.Call<int>("SetAudioOutputType", handle, use_audiotrack);
}

/// <summary>
/// 设置播放端缓存大小, 默认200毫秒
/// </summary>
/// <param name="buffer"></param>
public int NT_U3D_SetBuffer(long handle, int buffer)
{
    return player_obj_.Call<int>("SetBuffer", handle, buffer);
}

/// <summary>
/// 接口可实时调用:设置是否实时静音,1:静音; 0: 取消静音
/// </summary>
/// <param name="is_mute"></param>
public int NT_U3D_SetMute(long handle, int is_mute)
{
    return player_obj_.Call<int>("SetMute", handle, is_mute);
}

/// <summary>
/// 设置RTSP TCP模式, 1: TCP; 0: UDP
/// </summary>
/// <param name="is_using_tcp"></param>
public int NT_U3D_SetRTSPTcpMode(long handle, int is_using_tcp)
{
    return player_obj_.Call<int>("SetRTSPTcpMode", handle, is_using_tcp);
}

/// <summary>
/// 设置RTSP超时时间, timeout单位为秒,必须大于0
/// </summary>
/// <param name="timeout"></param>
public int NT_U3D_SetRTSPTimeout(long handle, int timeout)
{
    return player_obj_.Call<int>("SetRTSPTimeout", handle, timeout);
}

/// <summary>
/// 设置RTSP TCP/UDP自动切换
/// NOTE: 对于RTSP来说,有些可能支持rtp over udp方式,有些可能支持使用rtp over tcp方式.
/// 为了方便使用,有些场景下可以开启自动尝试切换开关, 打开后如果udp无法播放,sdk会自动尝试tcp, 如果tcp方式播放不了,sdk会自动尝试udp.
/// </summary>
/// <param name="timeout"></param>
/// timeout:如果设置1的话, sdk将在tcp和udp之间尝试切换播放,如果设置为0,则不尝试切换.
public int NT_U3D_SetRTSPAutoSwitchTcpUdp(long handle, int is_auto_switch_tcp_udp)
{
    return player_obj_.Call<int>("SetRTSPAutoSwitchTcpUdp", handle, is_auto_switch_tcp_udp);
}

/// <summary>
/// 设置快速启动该模式,
/// </summary>
/// <param name="is_fast_startup"></param>
public int NT_U3D_SetFastStartup(long handle, int is_fast_startup)
{
    return player_obj_.Call<int>("SetFastStartup", handle, is_fast_startup);
}

/// <summary>
/// 设置超低延迟模式 false不开启 true开启 默认false
/// </summary>
/// <param name="mode"></param>
public int NT_U3D_SetPlayerLowLatencyMode(long handle, int mode)
{
    return player_obj_.Call<int>("SetPlayerLowLatencyMode", handle, mode);
}

/// <summary>
/// 设置视频垂直反转
/// is_flip: 0: 不反转, 1: 反转
/// </summary>
/// <param name="is_flip"></param>
public int NT_U3D_SetFlipVertical(long handle, int is_flip)
{
    return player_obj_.Call<int>("SetFlipVertical", handle, is_flip);
}

/// <summary>
/// 设置视频水平反转
/// is_flip: 0: 不反转, 1: 反转
/// </summary>
/// <param name="is_flip"></param>
public int NT_U3D_SetFlipHorizontal(long handle, int is_flip)
{
    return player_obj_.Call<int>("SetFlipHorizontal", handle, is_flip);
}

/// <summary>
/// 设置顺时针旋转, 注意除了0度之外, 其他角度都会额外消耗性能
/// degress: 当前支持 0度,90度, 180度, 270度 旋转
/// </summary>
/// <param name="degress"></param>
public int NT_U3D_SetRotation(long handle, int degress)
{
    return player_obj_.Call<int>("SetRotation", handle, degress);
}

/// <summary>
/// 设置是否回调下载速度
/// is_report: if 1: 上报下载速度, 0: 不上报.
/// report_interval: 上报间隔,以秒为单位,>0.
/// </summary>
/// <param name="is_report"></param>
/// <param name="report_interval"></param>
public int NT_U3D_SetReportDownloadSpeed(long handle, int is_report, int report_interval)
{
    return player_obj_.Call<int>("SetReportDownloadSpeed", handle, is_report, report_interval);
}

/// <summary>
/// 设置是否需要在播放或录像过程中快照
/// </summary>
/// <param name="is_save_image"></param>
public int NT_U3D_SetSaveImageFlag(long handle, int is_save_image)
{
    return player_obj_.Call<int>("SetSaveImageFlag", handle, is_save_image);
}

/// <summary>
/// 播放或录像过程中快照
/// </summary>
/// <param name="imageName"></param>
public int NT_U3D_SaveCurImage(long handle, string imageName)
{
    return player_obj_.Call<int>("SaveCurImage", handle, imageName);
}

/// <summary>
/// 播放或录像过程中,快速切换url
/// </summary>
/// <param name="uri"></param>
public int NT_U3D_SwitchPlaybackUrl(long handle, string uri)
{
    return player_obj_.Call<int>("SwitchPlaybackUrl", handle, uri);
}

/// <summary>
/// 创建录像存储路径
/// </summary>
/// <param name="path"></param>
public int NT_U3D_CreateFileDirectory(string path)
{
    return player_obj_.Call<int>("CreateFileDirectory", path);
}

/// <summary>
/// 设置录像存储路径
/// </summary>
/// <param name="path"></param>
public int NT_U3D_SetRecorderDirectory(long handle, string path)
{
    return player_obj_.Call<int>("SetRecorderDirectory", handle, path);
}

/// <summary>
/// 设置单个录像文件大小
/// </summary>
/// <param name="size"></param>
public int NT_U3D_SetRecorderFileMaxSize(long handle, int size)
{
    return player_obj_.Call<int>("SetRecorderFileMaxSize", handle, size);
}

/// <summary>
/// 设置录像时音频转AAC编码的开关
/// aac比较通用,sdk增加其他音频编码(比如speex, pcmu, pcma等)转aac的功能.
/// 注意: 转码会增加性能消耗
/// </summary>
/// <param name="is_transcode"></param>
/// is_transcode:设置为1的话,如果音频编码不是aac,则转成aac,如果是aac,则不做转换. 设置为0的话,则不做任何转换. 默认是0.
public int NT_U3D_SetRecorderAudioTranscodeAAC(long handle, int is_transcode)
{
    return player_obj_.Call<int>("SetRecorderAudioTranscodeAAC", handle, is_transcode);
}

/// <summary>
/// 设置播放路径
/// </summary>
public int NT_U3D_SetUrl(long handle, string url)
{
    return player_obj_.Call<int>("SetUrl", handle, url);
}

/// <summary>
/// 开始播放
/// </summary>
public int NT_U3D_StartPlay(long handle)
{
    return player_obj_.Call<int>("StartPlay", handle);
}

/// <summary>
/// 获取YUV数据
/// </summary>
public AndroidJavaObject NT_U3D_GetVideoFrame(long handle)
{
    return player_obj_.Call<AndroidJavaObject>("GetVideoFrame", handle);
}

/// <summary>
/// 停止播放
/// </summary>
public int NT_U3D_StopPlay(long handle)
{
    return player_obj_.Call<int>("StopPlay", handle);
}

/// <summary>
/// 开始录像
/// </summary>
public int NT_U3D_StartRecorder(long handle)
{
    return player_obj_.Call<int>("StartRecorder", handle);
}

/// <summary>
/// 停止录像
/// </summary>
public int NT_U3D_StopRecorder(long handle)
{
    return player_obj_.Call<int>("StopRecorder", handle);
}

/// <summary>
/// 关闭播放
/// </summary>
public int NT_U3D_Close(long handle)
{
    return player_obj_.Call<int>("Close", handle);
}

/// <summary>
/// UnInit Player
/// </summary>
public int NT_U3D_UnInit()
{
    return DANIULIVE_RETURN_OK;
}

Event传递,我们在unity3d里面解析相应的字串即可:

///

/// android 传递过来 code
/// </summary>
/// <param name="code"></param>
public void onNTSmartEvent(string param)
{
    if (!param.Contains(","))
    {
        Debug.Log("[onNTSmartEvent] android传递参数错误");
        return;
    }

   string[] strs = param.Split(',');

   string player_handle =strs[0];
   string code = strs[1];
   string param1 = strs[2];
   string param2 = strs[3];
   string param3 = strs[4];
   string param4 = strs[5];
    
   Debug.Log("[onNTSmartEvent] code: 0x" + Convert.ToString(Convert.ToInt32(code), 16));

    switch (Convert.ToInt32(code))
    {
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_STARTED:
            Debug.Log("开始。。");
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_CONNECTING:
            Debug.Log("连接中。。");
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_CONNECTION_FAILED:
            Debug.Log("连接失败。。");
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_CONNECTED:
            Debug.Log("连接成功。。");
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_DISCONNECTED:
            Debug.Log("连接断开。。");
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_STOP:
            Debug.Log("停止播放。。");
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_RESOLUTION_INFO:
            Debug.Log("分辨率信息: width: " + Convert.ToInt32(param1) + ", height: " + Convert.ToInt32(param2));
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_NO_MEDIADATA_RECEIVED:
            Debug.Log("收不到媒体数据,可能是url错误。。");
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_SWITCH_URL:
            Debug.Log("切换播放URL。。");
            break;

        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_CAPTURE_IMAGE:
            Debug.Log("快照: " + param1 + " 路径:" + param3);

            if (Convert.ToInt32(param1) == 0)
            {
                Debug.Log("截取快照成功。.");
            }
            else
            {
                Debug.Log("截取快照失败。.");
            }
            break;

        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_RECORDER_START_NEW_FILE:
            Debug.Log("[record]开始一个新的录像文件 : " + param3);
            break;
        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_ONE_RECORDER_FILE_FINISHED:
            Debug.Log("[record]已生成一个录像文件 : " + param3);
            break;

        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_START_BUFFERING:
            Debug.Log("Start_Buffering");
            break;

        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_BUFFERING:
            Debug.Log("Buffering: " + Convert.ToInt32(param1));
            break;

        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_STOP_BUFFERING:
            Debug.Log("Stop_Buffering");
            break;

        case EVENTID.EVENT_DANIULIVE_ERC_PLAYER_DOWNLOAD_SPEED:
            Debug.Log("download_speed:" + param1 + "Byte/s" + ", "
                    + (Convert.ToInt32(param1) * 8 / 1000) + "kbps" + ", " + (Convert.ToInt32(param1) / 1024)
                    + "KB/s");
            break;
    }

}

开始播放:

public void Play()
{
    if (is_running)
    {
        Debug.Log("已经在播放。。");   
        return;
    }

    //获取输入框的url
    string url = input_url_.text.Trim();

    if (!url.StartsWith("rtmp://") && !url.StartsWith("rtsp://"))
    {
        videoUrl = "rtmp://202.69.69.180:443/webcast/bshdlive-pc";
    }
    else
    {
        videoUrl = url;
    }

    OpenPlayer();

    if ( player_handle_ == 0 )
        return;

    NT_U3D_Set_Game_Object(player_handle_, game_object_);

    NT_U3D_SetUrl(player_handle_, videoUrl);

    /* ++ 播放前参数配置可加在此处 ++ */
    int is_using_tcp = 0;        //TCP/UDP模式设置
    NT_U3D_SetRTSPTcpMode(player_handle_, is_using_tcp);

    int is_report = 0;
    int report_interval = 1;
    NT_U3D_SetReportDownloadSpeed(player_handle_, is_report, report_interval);  //下载速度回调

    NT_U3D_SetBuffer(player_handle_, play_buffer_time_);                        //设置buffer time

    NT_U3D_SetPlayerLowLatencyMode(player_handle_, is_low_latency_ ? 1 : 0);    //设置是否启用低延迟模式

    NT_U3D_SetMute(player_handle_, is_mute_ ? 1 : 0);                           //是否启动播放的时候静音

    NT_U3D_SetVideoDecoderMode(player_handle_, is_hw_decode_ ? 1 : 0);          //设置H.264软硬解模式

    NT_U3D_SetVideoHevcDecoderMode(player_handle_, is_hw_decode_ ? 1 : 0);          //设置H.265软硬解模式

    int is_fast_startup = 1;
    NT_U3D_SetFastStartup(player_handle_, is_fast_startup);                     //设置快速启动模式

    int rtsp_timeout = 10;
    NT_U3D_SetRTSPTimeout(player_handle_, rtsp_timeout);                        //设置RTSP超时时间

    int is_auto_switch_tcp_udp = 1;
    NT_U3D_SetRTSPAutoSwitchTcpUdp(player_handle_, is_auto_switch_tcp_udp);    //设置TCP/UDP模式自动切换
    /* -- 播放前参数配置可加在此处 -- */

    int flag = NT_U3D_StartPlay(player_handle_);

    if (flag  == DANIULIVE_RETURN_OK)
    {
        is_need_get_frame_ = true;
        Debug.Log("播放成功");
    }
    else
    {
        is_need_get_frame_ = false;
        Debug.LogError("播放失败");
    }

    is_running = true;  
}

开始播放后,native模块,回调yuv/rgb数据,unity3d模块,做相应的绘制处理即可。

private void Update()
{
    if (!is_need_get_frame_)
        return;

    if (player_handle_ == 0)
        return;

    AndroidJavaObject u3d_video_frame_obj = NT_U3D_GetVideoFrame(player_handle_);

    if (u3d_video_frame_obj == null)
        return;

    VideoFrame converted_video_frame = ConvertToVideoFrame(u3d_video_frame_obj);

    if (converted_video_frame == null)
        return;

    if ( !is_need_init_texture_)
    {
        if (converted_video_frame.width_ != video_width_
            || converted_video_frame.height_ != video_height_
            || converted_video_frame.y_stride_ != y_row_bytes_
            || converted_video_frame.u_stride_ != u_row_bytes_
            || converted_video_frame.v_stride_ != v_row_bytes_)
        {
            is_need_init_texture_ = true;
        }
    }

    if (is_need_init_texture_)
    {
        if (InitYUVTexture(converted_video_frame))
        {
            is_need_init_texture_ = false;
        }
    }

    UpdateYUVTexture(converted_video_frame);

    converted_video_frame = null;
}

关闭播放:

private void ClosePlayer()
{
    is_need_get_frame_ = false;
    is_need_init_texture_ = false;

    int flag = NT_U3D_StopPlay(player_handle_);
    if (flag == DANIULIVE_RETURN_OK)
    {
        Debug.Log("停止成功");
    }
    else
    {
        Debug.LogError("停止失败");
    }

    flag = NT_U3D_Close(player_handle_);
    if (flag == DANIULIVE_RETURN_OK)
    {
        Debug.Log("关闭成功");
    }
    else
    {
        Debug.LogError("关闭失败");
    }

    player_handle_ = 0;

    NT_U3D_UnInit();

    is_running = false;
}

以上是大概的实现流程,Unity环境下,实际测试时延,和 native直播模块相差无几,都是毫秒级延迟。Windows平台和iOS平台也是如此。

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
[Unity3d]控制物体的旋转以及缩放
我们在用unity做产品展示的功能的时候,旋转和缩放是必不可少的功能,如果是大型物品,比如汽车或者其他的东西,可以使用旋转摄像机的思路来控制达到物体旋转的效果,如果是这种情况可以参考我之前的文章:http://blog.csdn.net/dingxiaowei2013/article/details/16337411;如果是小型物品的展示,就可以通过通过控制物体本身的旋转和位置的变化来达到缩放和旋转物体的效果。
1256 0
[unity3d]鼠标拖动and旋转缩放
using UnityEngine; using System.Collections; public class mouseRotateAndScale : MonoBehaviour { GameObject car; //public GameOb...
871 0
[Unity3d]鼠标旋转缩放拖动模型
实现鼠标旋转拖动缩放模型,非常不错的代码: using UnityEngine; using System.Collections; public class MouseControl : MonoBehaviour { //目标物体 ...
842 0
+关注
文章
问答
文章排行榜
最热
最新
相关电子书
更多
全景视频的播放及优化
立即下载
从 SDK 到编解码:视频直播架构解析
立即下载
低代码开发师(初级)实战教程
立即下载