Chapter 3. The complex multi-layer input

简介: The idea behind the input module is to treat packets, without knowing at all what is in it. It only takes a packet, reads its ID, and delivers...

The idea behind the input module is to treat packets, without knowing at all what is in it. It only takes a packet, reads its ID, and delivers it to the decoder at the right time indicated wps_clip_image1 in the packet header (SCR and PCR fields in MPEG). All the basic browsing operations are implemented wps_clip_image2 without peeking wps_clip_image3 at the content wps_clip_image4 of the elementary stream.

Thus it remains very generic. This also means you can't do stuff like "play 3 frames now" or "move forward 10 frames" or "play as fast as you can but play all frames". It doesn't even know what a "frame" is. There is no privilegedwps_clip_image5 elementary stream, like the video one could be (for the simple reason that, according to MPEG, a stream may contain several video ES).

What happens to a file

An input thread is spawnedwps_clip_image6 for every file read. Indeed, input structures and decoders need to be reinitialized because the specificities of the stream may be different. input_CreateThread is called by the interface thread (playlist module).

At first, an input plug-in capable of reading the plugin itemwps_clip_image7 is looked for [this is inappropriate wps_clip_image8: we should first open the socket, and then probe the beginning of the stream to see which plug-in can read it]. The socket is opened by either input_FileOpen, input_NetworkOpen, or input_DvdOpen. This function sets two very important parameters : b_pace_control and b_seekable (see next section).

Note

We could use so-called "access" plugins for this whole mechanism wps_clip_image9of opening the input socket. This is not the case because we thought only those three methods were to be used at present, and if we need others we can still build them in.

Now we can launch the input plugin's pf_init function, and an endless loop doing pf_read and pf_demux. The plugin is responsiblewps_clip_image10 for initializing the stream structures (p_input->stream), managing packet buffers, reading packets and demultiplex them. But in most tasks it will be assisted by functions from the advanced input API (c). That is what we will study in the coming sections !

Stream Management

The function which has opened the input socket must specify two properties wps_clip_image11about it :

1. p_input->stream.b_pace_control : Whether or not the stream can be read at our own pace (determined by the stream's frequency and the host computer's system clock). For instance a file or a pipe (including TCP/IP connections) can be read at our pacewps_clip_image12, if we don't read fast enough, the other end of the pipe will just block on a write() operation. On the contrarywps_clip_image13, UDP streaming (such as the one used by VideoLAN Server) is done at the server's pace, and if we don't read fast enough, packets will simply be lost when the kernel's buffer is full. So the driftwps_clip_image14 introduced by the server's clock must be regularly compensated. This property controls the clock management, and whether or not fast forward and slow motion wps_clip_image15can be done.

Subtilities in the clock management

With a UDP socket and a distant server, the drift is not negligible wps_clip_image16because on a whole movie it can account for seconds if one of the clocks is slightlywps_clip_image17 fucked up. That means that presentation dates given by the input thread may be out of sync, to some extent, with the frequencies given in every Elementary Stream. Output threads (and, anecdotically, decoder threads) must deal with it.

The same kind of problems may happen when reading from a device (like video4linux's /dev/video ) connected for instance to a video encoding board. There is no way we could differentiate wps_clip_image18it from a simple cat foo.mpg | vlc - , which doesn't imply any clock problem. So the Right Thing (c) would be to ask the user about the value of b_pace_control , but nobody would understand what it means (you are not the dumbest person on Earth, and obviouslywps_clip_image19 you have read this paragraphwps_clip_image20 several times to understand it :-). Anyway, the drift should be negligiblewps_clip_image21 since the board would share the same clock as the CPU, so we chose to neglectwps_clip_image22 it.

2. p_input->stream.b_seekable : Whether we can do lseek() calls on the file descriptor or not. Basically whether we can jump anywhere in the stream (and thus display a scrollbar) or if we can only read one byte after the other. This has less impactwps_clip_image23 on the stream management than the previous item, but it is not redundantwps_clip_image24, because for instance cat foo.mpg | vlc - is b_pace_control = 1 but b_seekable = 0. On the contrary, you cannot have b_pace_control = 0 along with b_seekable = 1. If a stream is seekable, p_input->stream.p_selected_area->i_size must be set (in an arbitrary unit, for instance bytes, but it must be the same as p_input->i_tell which indicates the byte we are currently reading from the stream).

Offset to time conversions wps_clip_image25

Functions managing clocks are located in src/input/input_clock.c. All we know about a file is its start offset and its end offset (p_input->stream.p_selected_area->i_size), currently in bytes, but it could be plugin-dependant. So how the hell can we display in the interface a time in seconds ? Well, we cheat. PS streams have a mux_rate property wps_clip_image26which indicates how many bytes we should read in a second. This is subject to change at any time, but practicallywps_clip_image27 it is a constant for all streams we know. So we use it to determine time offsets.

Structures exported to the interface

Let's focus on the communication API between the input module and the interface. The most important file is include/input_ext-intf.h, which you should know almost by heart. This file defines the input_thread_t structure, the stream_descriptor_t and all programs and ES descriptors included (you can view it as a tree).

First, note that the input_thread_t structure features two void * pointers, p_method_data and p_plugin_data, which you can respectivly wps_clip_image28use for buffer management data and plugin data.

Second, a stream description is stored in a tree featuring program descriptors, which themselves contain several elementary stream descriptors. For those of you who don't know all MPEG concepts, an elementary stream, aka ES, is a continuous stream of video or (exclusive) audio data, directly readable by a decoder, without decapsulation.

This tree structure is illustrated by the following figure, where one stream holds two programs. In most cases there will only be one program (to my knowledge only TS streams can carry several programs, for instance a movie and a football game at the same time - this is adequate for satellite and cable broadcasting).

p_input->stream : The stream, programs and elementary streams can be viewed as a tree.

Warning

For all modifications wps_clip_image29and accesses to the p_input->stream structure, you must hold the p_input->stream.stream_lock.

ES are described by an ID (the ID the appropriate demultiplexer will look for), a stream_id (the real MPEG stream ID), a type (defined in ISO/IEC 13818-1 table 2-29) and a litteral description. It also contains context information for the demultiplexer, and decoder information p_decoder_fifo we will talk about in the next chapter. If the stream you want to read is not an MPEG system layer (for instance AVI or RTP), a specific demultiplexer will have to be written. In that case, if you need to carry additional information, you can use void * p_demux_data at your convenience. It will be automatically freed on shutdown. wps_clip_image30

Why ID and not use the plain MPEG stream_id ?

When a packet (be it a TS packet, PS packet, or whatever) is read, the appropriatewps_clip_image31 demultiplexer will look for an ID in the packet, find the relevantwps_clip_image32 elementary stream, and demultiplex it if the user selected it. In case of TS packets, the only information we have is the ES PID, so the reference ID we keep is the PID. PID don't exist in PS streams, so we have to invent one. It is of course based on the stream_id found in all PS packets, but it is not enough, since private streams (ie. AC3, SPU and LPCM) all share the same stream_id (0xBD). In that case the first byte of the PES payload is a stream private ID, so we combine wps_clip_image33this with the stream_id to get our ID (if you did not understand everything, it isn't very important - just remember we used our brains before writing the code :-).

The stream, program and ES structures are filled in by the plugin's pf_init() using functions in src/input/input_programs.c, but are subject to change at any time. The DVD plugin parses .ifo files to know which ES are in the stream; the TS plugin reads the PAT and PMT structures in the stream; the PS plugin can either parse the PSM structure (but it is rarely present), or build the tree "on the fly" by pre-parsing the first megabyte of data.

Warning

In most cases we need to pre-parse (that is, read the first MB of data, and go back to the beginning) a PS stream, because the PSM (Program Stream Map) structure is almost never present. This is not appropriate, though, but we don't have the choice. A few problems will arise. First, non-seekable streams cannot be pre-parsed, so the ES tree will be built on the fly. Second, if a new elementary stream starts after the first MB of data (for instance a subtitle track won't show up during the credits), it won't appear in the menu before we encounter the first packet. We cannot pre-parse the entire wps_clip_image34stream because it would take hours (even without decoding it).

It is currently the responsibility of the input plugin to spawn the necessary decoder threads. It must call input_SelectES ( input_thread_t * p_input, es_descriptor_t * p_es ) on the selected ES.

The stream descriptor also contains a list of areas. Areas are logical discontinuities in the stream, for instance chapters and titles in a DVD. There is only one area in TS and PS streams, though we could use them when the PSM (or PAT/PMT) version changes. The goal is that when you seek to another area, the input plugin loads the new stream descriptor tree (otherwise the selected ID may be wrong).

Methods used by the interface

Besides, input_ext-intf.c provides a few functions to control the reading of the stream :

· input_SetStatus ( input_thread_t * p_input, int i_mode ) : Changes the pace of reading. i_mode can be one of INPUT_STATUS_END, INPUT_STATUS_PLAY, INPUT_STATUS_PAUSE, INPUT_STATUS_FASTER, INPUT_STATUS_SLOWER.

Note

Internally,wps_clip_image35 the pace of reading is determined by the variable p_input->stream.control.i_rate. The default value is DEFAULT_RATE. The lower the value, the faster the pace is. Rate changes are taken into account wps_clip_image36in input_ClockManageRef. Pause is accomplished by simply stopping the input thread (it is then awaken by a pthread signal). In that case, decoders will be stopped too. Please remember this if you do statistics on decoding times (like src/video_parser/vpar_synchro.c does). Don't call this function if p_input->b_pace_control == 0.

· input_Seek ( input_thread_t * p_input, off_t i_position ) : Changes the offset of reading. Used to jump to another place in a file. You mustn't call this function if p_input->stream.b_seekable == 0. The position is a number (usually long long, depends on your libc) between p_input->p_selected_area->i_start and p_input->p_selected_area->i_size (current value is in p_input->p_selected_area->i_tell).

Note

Multimedia files can be very large, especially when we read a device like /dev/dvd, so offsets must be 64 bits large. Under a lot of systems, like FreeBSD, off_t are 64 bits by default, but it is not the case under GNU libc 2.x. That is why we need to compile VLC with -D_FILE_OFFSET_BITS=64 -D__USE_UNIX98.

Escaping wps_clip_image37 stream discontinuities

Changing the reading position at random can result in a messed wps_clip_image38up stream, and the decoder which reads it may segfault. To avoid this, we send several NULL packets (ie. packets containing nothing but zeros) before changing the reading position. Indeed, under most video and audio formats, a long enough stream of zeros is an escape sequence and the decoder can exit cleanly.

· input_OffsetToTime ( input_thread_t * p_input, char * psz_buffer, off_t i_offset ) : Converts an offset value to a time coordinate (used for interface display). [currently it is broken with MPEG-2 files]

· input_ChangeES ( input_thread_t * p_input, es_descriptor_t * p_es, u8 i_cat ) : Unselects all elementary streams of type i_cat and selects p_es. Used for instance to change language or subtitle track.

· input_ToggleES ( input_thread_t * p_input, es_descriptor_t * p_es, boolean_t b_select ) : This is the clean way to select or unselect a particular elementary stream from the interface.

Buffers management

Input plugins must implementwps_clip_image39 a way to allocate and deallocatewps_clip_image40 packets (whose structures will be described in the next chapter). We basically need four functions :

· pf_new_packet ( void * p_private_data, size_t i_buffer_size ) : Allocates a new data_packet_t and an associated buffer of i_buffer_size bytes.

· pf_new_pes ( void * p_private_data ) : Allocates a new pes_packet_t.

· pf_delete_packet ( void * p_private_data, data_packet_t * p_data )  : Deallocates p_data.

· pf_delete_pes ( void * p_private_data, pes_packet_t * p_pes ) : Deallocates p_pes.

All functions are given p_input->p_method_data as first parameter, so that you can keep records of allocated and freed packets.

Buffers management strategies

Buffers management can be done in three ways :

1. Traditional libc allocation : For a long time we have used in the PS plugin malloc() and free() every time we needed to allocate or deallocate a packet. Contrary to a popular beliefwps_clip_image41, it is not that slow.

2. Netlist : In this method we allocate a very big buffer at the beginning of the problem, and then manage a list of pointers to free packets (the "netlist"). This only works well if all packets have the same size. It is used for long for the TS input. The DVD plugin also uses it, but adds a refcount flag because buffers (2048 bytes) can be shared among several packets. It is now deprecated wps_clip_image42and won't be documented.

3. Buffer cache : We are currently developing a new method. It is already in use in the PS plugin. The idea is to call malloc() and free() to absorbwps_clip_image43 stream irregularitieswps_clip_image44, but re-use all allocated buffers via a cache system. We are extending it so that it can be used in any plugin without performance hit, but it is currentlywps_clip_image45 left undocumented.

Demultiplexing the stream

After being read by pf_read , your plugin must give a function pointer to the demultiplexer function. The demultiplexer is responsible for parsing the packet, gathering PES, and feeding decoders.

Demultiplexers for standard MPEG structures (PS and TS) have already been written. You just need to indicate input_DemuxPS and input_DemuxTS for pf_demux. You can also write your own demultiplexer.

It is not the purpose of this document to describe the different levels of encapsulationwps_clip_image46 in an MPEG stream. Please refer to your MPEG specificationwps_clip_image47 for that.

相关文章
|
4月前
|
TensorFlow 算法框架/工具
【Tensorflow】解决A `Concatenate` layer should be called on a list of at least 2 inputs
在TensorFlow 2.0中,使用Concatenate函数时出现错误,可以通过替换为tf.concat 来解决。
44 4
A+B for Input-Output Practice
A+B for Input-Output Practice
Re1:读论文 C&S (Correct and Smooth) Combining Label Propagation and Simple Models Out-performs Graph Ne
Re1:读论文 C&S (Correct and Smooth) Combining Label Propagation and Simple Models Out-performs Graph Ne
Re1:读论文 C&S (Correct and Smooth) Combining Label Propagation and Simple Models Out-performs Graph Ne
|
算法框架/工具 Windows
|
并行计算
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the
528 0