CLIENT/SERVER
STREAMING SERVER
CAPTURE
PLAYBACK
PROTOCOLS
RTMP
RTP/RTSP
HLS
MPEG-DASH
WEBRTC
SRT
MORE
FILES
MP4
TS
TOOLS
PROCESSING
VIDEO
AUDIO
CODEC
This chapter describes some basic elements constituting VASTreaming libraries.
VASTreaming state machine objects are not re-usable! Once object reaches a terminal state e.g., Closed or Error, it must be disposed. A new object instance must be created if there is a necessity to keep using it.
It describes one and only media stream, for example video, audio or subtitle stream. MediaType includes full information about a media stream received or sent. For video, it can contain width, frame rate, etc.; for audio - sample rate, number of channels, etc.; for any stream - it contains codec type, bitrate, etc. MediaType is widely used throughout the VASTreaming library and can be received by a user, for example in NewStream event handler.
Later, in the documentation, the term media descriptor may be seen. Media descriptor is a set of MediaType objects describing all streams of a particular media source, media sink, or publishing point.
IMediaSource is a standard interface, implemented in streaming library by numerous source objects, for example (the list is not exhaustive):
IMediaSource exposes the following properties and methods (the list is not exhaustive):
And the following events:
See Reference section for detailed explanation of each property, method and event.
Classes implementing IMediaSource interface are heavily asynchronous, and event controlled. It is not guaranteed that calls to functions Open(), Start() and Stop() change object state immediately. Instead, a user has to rely on event handlers to control the object.
One of important features of IMediaSource inherited object is that it can be shared among multiple sinks, e.g. only one camera source can produce media data for concurrent file writing and streaming.
IMediaSource is a finite-state machine, see the below diagrams for state transitions. Paused and Stopped states are optional and can be in interactive sources only, such as files.
Initialized - a state that object gets initially upon its creation.
In order to launch an object, a user needs to call method Open(). The method turns an object into the state Opening and starts connecting to a server, file opening or any other logic necessary to be done to obtain media descriptor of a source.
While opening a source, as soon as media streams are completely detected, event NewStream is invoked. If a source contains N streams, then NewStream event is invoked N times.
When opening procedure finishes, an object gets the state Opened. At this moment, we know descriptions of all media streams, so we have MediaType objects for each stream.
If opening fails, an object gets the state Error.
After getting the state Opened, we can start an object by calling the method Start(). The attempt to call method Start() when an object is in Initialized or Opening state results in exception.
Upon successful finishing of method Start(), an object gets the state Started and starts pushing media samples and invoking NewSample event.
If any critical error occurs during or after starting, an object state turns to Error.
The state Started is the main state of any object, in this state it remains most of the time.
For an interactive source, state Paused is also possible. An object gets it upon calling method Pause(). In this state, an object stops receiving and pushing media samples but does not disconnect from a server or does not close a file. In order to restart pushing media samples, a user needs to call method Start(). After that, an object turns to state Started again.
When an object is not needed, it can be stopped by calling method Stop(). Upon successful stopping, an object state turns to Stopped or Closed.
State Stopped is not intended for all objects, sometimes an object can be directly turned into state Closed.
There are several more specialized interfaces inherited from IMediaSource.
IInteractiveMediaSource - an interactive media source with certain duration, that allows pausing and seeking.
IAudioCaptureSource - an audio capture source with additional parameters for capturing audio.
IVideoCaptureSource - a video capture source with additional parameters for capturing video.
INetworkSource - a network source with network related additional parameters e.g., remote end point, socket error etc.
Sample IMediaSource usage:
VAST.Media.IMediaSource mediaSource = VAST.Media.SourceFactory.Create("source_uri"); mediaSource.StateChanged += (object sender, VAST.Media.MediaState e) => { switch (e) { case VAST.Media.MediaState.Opened: // source successfully opened, start it mediaSource.Start(); break; case VAST.Media.MediaState.Started: // source has been started, samples will come soon break; case VAST.Media.MediaState.Closed: // source has been disconnected and closed break; case VAST.Media.MediaState.Error: // critical error occurred break; } }; mediaSource.NewStream += (object sender, VAST.Media.NewStreamEventArgs e) => { // source stream has been detected }; mediaSource.NewSample += (object sender, VAST.Media.NewSampleEventArgs e) => { // new sample arrived }; mediaSource.Error += (object sender, VAST.Media.ErrorEventArgs e) => { if (e.IsCritical) { // critical error occurred, show it to a user } }; // start opening mediaSource.Open();
Notes:
IMediaSink is a standard interface that is implemented in the streaming library by numerous sink objects, for example (the list is not exhaustive):
IMediaSink exposes the following properties and methods (the list is not exhaustive):
IMediaSink is heavily asynchronous, and event controlled. It is not guaranteed that calls to functions Open(), Start() and Stop() change object state immediately. Instead, a user has to rely on event handlers to control the object.
IMediaSink is a finite-state machine. The diagram for state transitions is just the same as for IMediaSource (see Fig. 5).
Interactive interface is not supported, so Fig. 4 is not applicable to IMediaSink.
Sample IMediaSink usage:
VAST.Media.IMediaSink sink = new VAST.Media.SinkFactory.Create("sink-uri"); // add video media type sink.AddStream(videoStreamIndex, videoMediaType); // add audio media type sink.AddStream(audioStreamIndex, audioMediaType); sink.StateChanged += (object sender, Media.MediaState e) => { lock (this) { switch (e) { case VAST.Media.MediaState.Opened: // sink successfully opened, start it ((VAST.Media.IMediaSink)sender).Start(); Task.Run(() => { // run endless loop to push media data to sink }); break; case VAST.Media.MediaState.Started: // sink has been started break; case VAST.Media.MediaState.Closed: // sink has been disconnected break; } } }; sink.Error += (object sender, VAST.Media.ErrorEventArgs e) => { if (e.IsCritical) { // critical error occurred, show it to a user } }; // start opening sink.Open();
A user can handle IMediaSource and IMediaSink objects manually, but MediaSession helps to do it in the simplest and most optimal way.
MediaSession is an object allowing a client to automate simultaneous processing of several IMediaSinks and IMediaSources. Every MediaSession may have from 1 to N IMediaSources and from 1 to M IMediaSinks. Media data from all sources is re-directed to each IMediaSink, so each IMediaSink receives a frame copy from each IMediaSource, which allows re-directing media data from sources to several sinks.
MediaSession is a state machine without explicit states (a user cannot receive value of an object state). Although, in fact, we have 3 implicit states:
Sample MediaSession usage:
var source = VAST.Media.SourceFactory.Create("source-uri"); var sink = new VAST.File.ISO.IsoSink(); sink.Uri = "file-path"; var recordingSession = new VAST.Media.MediaSession(); recordingSession.AddSource(source); recordingSession.AddSink(sink); recordingSession.Error += (object sender, VAST.Media.ErrorEventArgs e) => { if (e.IsCritical) { // critical error occurred, show it to a user } }; recordingSession.Start();
IDecoder is a standard interface that is implemented by various platform-dependent decoders. This object is made stateless in order to provide a user with a simple interface to be easily used in any place of user code.
Since all decoders are platform specific, user should not instantiate decoder object directly. It is highly recommended to use special class DecoderFactory, which creates an optimal decoder for a certain platform. If necessary, user may set additional parameters in class DecoderParameters such as media framework, hardware acceleration support, etc.
Class DecoderParameters allows user to set media framework, which will be used for creating a decoder. By default, VASTreaming library uses built-in media framework of a platform, but user can set another framework e.g., FFmpeg. Besides, a user can choose MediaFramework.Unknown. In the last case the library will try to create a decoder using all frameworks available on this platform.
Sample IDecoder usage:
// decoder creation VAST.Media.IDecoder decoder = VAST.Media.DecoderFactory.Create(encodedMediaType, decodedMediaType); ... // frame decoding procedure decoder.Write(encodedSample); while (true) { VAST.Common.VersatileBuffer decodedSample = decoder.Read(); if (decodedSample == null) { break; } // process decoded sample if necessary decodedSample.Release(); }
IEncoder is a standard interface that is implemented by various platform-dependent encoders. IEncoder is also stateless due to the same reason as it is mentioned above for IDecoder.
Since all encoders are platform specific, user should not instantiate encoder object directly. It is better to use special class EncoderFactory, which creates an optimal encoder for a certain platform. If necessary, user may set additional parameters in class EncoderParameters such as media framework, hardware acceleration support, etc.
Class EncoderParameters allows user to set media framework, which will be used for creating an encoder. By default, VASTreaming library uses built-in media framework of a platform, but user can set another framework e.g., FFmpeg. Besides, a user can choose MediaFramework.Unknown. In the last case the library will try to create an encoder using all frameworks available on this platform.
Sample IEncoder usage:
// encoder creation VAST.Media.IEncoder encoder = VAST.Media.EncoderFactory.Create( uncompressedMediaType, encodedMediaType); ... // encoding procedure encoder.Write(uncompressedSample); while (true) { VAST.Common.VersatileBuffer encodedSample = encoder.Read(); if (encodedSample == null) { break; } // process encoded sample if necessary encodedSample.Release(); }
Publishing point is a virtual entity on a streaming server answering clients requests with some content. The term was introduced by Microsoft to translate client requests for content into a physical path on the server hosting the content. One streaming server can have multiple publishing points. Each publishing point is bound to one and only source on a particular streaming server, thus a source defines the content of a publishing point delivered to clients via different protocols.
PublishingPoint object implements publishing point in VASTreaming library. This object can work only with multi-protocol server; there is no implementation for single-protocol servers.
Every publishing point is characterized by unique publishing path, which is used for its identification and creation of its URI in order to ensure client access to this publishing point. Moreover, every publishing point has its unique id (type System.Guid), which equals to id of a source object. So, a publishing point can be obtained either by its id or by publishing path. User may use both methods.
PublishingPoint object is fully controlled by StreamingServer object. It supports only one source and multiple sinks of different protocols. Meanwhile, VASTreaming library provides sources that may be aggregation of other sources, e.g., AggregatedNetworkSource, MixingSource, etc. See Reference section for more details.
PublishingPoint controls media data flow between sources and sinks automatically, in the similar way MediaSession does.
PublishingPoint object is rarely used directly. Usually, user code needs it to add manual sinks.
A user can tweak publishing point creation by providing optional PublishingPointParameters object with various settings. See reference section for more details.
StreamingServer object implements a multi-protocol server shown in Fig. 3.
StreamingServer is also a state machine without explicit states (a user cannot receive value of an object state). Although, in fact, we have 3 implicit states:
Before starting, StreamingServer should be initialized by setting parameters of protocol specific servers, which will run as a part of multi-protocol server (the list is not exhaustive):
Setting particular port value to a non-zero enables corresponding protocol.
HTTP server is common for many HTTP-based protocols. Each HTTP-based server specifies a path to distinguish them from each other.
During runtime, StreamingServer can be controlled by user code via utilizing of the following methods (the list is not exhaustive):
Multi-protocol server creates a publishing point automatically once it gets an incoming connection when a publisher connects a server. But a user may need a publishing point which pulls media data from a local file or from an external server. In this case, a user can create this publishing point manually by calling CreatePublishingPoint method.
All server logic is also controlled with events which are handled by user code. The events are as follows (the list is not exhaustive):
Authorize. This event is invoked when a remote peer establishes a new connection. When it is invoked, a server already knows whether it is a publisher or a client connection, protocol by which a remote peer has been connected, inbound URI, etc. This information is passed to a user in the event argument.
User code should recognize whether to permit a connection or deny it. What algorithm is used for authentication/authorization is up to a user. It can be done by adding any parameters to URI, via an additional header, etc.
If a user authorizes a certain connection, and if it has come from a publisher, then a user can set additional parameters for the publishing point creation in ConnectionInfo.AdditionalParameters property.
PublisherConnected. This event is invoked when a connection with publisher has been established, and publishing point has been created. The created PublishingPoint object is passed as a parameter of this event.
User can utilize this event handler to add sinks to a publishing point, e.g., forward to an external server or record to a file.
Every publisher gets a unique connection id (type System.Guid) which is equal to a publishing point id. This id can be used later to obtain a PublishingPoint object.
ClientConnected. The event is invoked when a client connects to a server, and all client network settings are being passed in event parameter. Every client gets a unique connection id which can be used later to get information about the client.
Disconnected. The event is invoked when a remote peer (publisher or client) disconnects. User can recognize what peer exactly has been disconnected by comparing connection id with the saved data. If it is a publisher who disconnects, then it will be publisher connection id. If it is a client, it will be a client connection id.
Error. The event is invoked when an unrecoverable error occurs while a server is being connected by a publisher or a client. It provides detailed description of the error. If error occurred in a publisher connection, then it means that the corresponding publishing point will be deleted shortly.
PublishingPointRequested. Streaming server provides a possibility to create publishing point upon a client demand. Once a client connects to a server and a server cannot find a requested publishing path in the list of active publishing points, then a server invokes this event so that the user can create a publishing point in event handler. This logic is optional. If user finds unnecessary to support on-demand publishing point creation, a client just receives a reply with an error from a server. If user handles this event, then a client receives a normal response from a server after user code creates new publishing point on the fly.
StreamingServer object can also be controlled via JSON API. The API functionality does not cover all functions available to user code, but it is constantly enhancing.
When a streaming server creates a new publishing point, it automatically adds several sinks. These automatically created sinks are defined on the base of active single protocol servers e.g., if streaming server is configured as an RTMP and HLS server, then while creating every publishing point, sinks are being created to access the publishing point via RTMP and HLS protocols. However, user can override the default server logic and explicitly set allowed protocols in ConnectionInfo.AdditionalParameters property in Authorize event and then a server will create sinks only for the specified protocols.
While creating a publishing point, it gets unique publishing path as it has been mentioned above. This publishing path is used to create unique URI to provide a client with publishing point access.
Streaming server supports two streaming types - live and VOD. Live streaming refers to any media delivered and played back simultaneously and endlessly. Whereas VOD is used to access previously created content, it is usually stored as a file. One more fundamental difference between these two is that VOD allows pausing and seeking content at any point of a recorded video. Accordingly, streaming server gives a possibility to created VOD publishing point, but access to such interactive contents is possible only via a limited number of protocols - in fact, they are HLS and MPEG-DASH.
Streaming server is optimized to latency minimization in those protocols where applicable.
Streaming is supposed to use memory buffers of significant size. For example, one uncompressed frame in full HD resolution requires 8 Mb. Creating even one buffer of this kind takes essential CPU resources.
Moreover, allocation and de-allocation of big buffers in the heap may lead to memory fragmentation. In spite of built-in heap optimization within .NET, some probability of fragmentation still remains.
To solve the problems described above, VASTreaming library uses buffers allocated during a library initialization at an application startup. These buffers are in operation and are not disposed while an application is running.
Besides, VASTreaming library implements mechanism of allocators, which returns object VersatileBuffer upon user request. Object VersatileBuffer is also an abstraction layer over internal implementation different for each platform and dependent on methods of data storing.
User do not have direct access to data storage, which is used in VersatileBuffer, instead of that one should use methods Append, Insert, and CopyTo for indirect copying data into VersatileBuffer and from it.
VersatileBuffer also implements mechanism of references on top of the standard object reference mechanism of .NET. It can be difficult to grasp for users who has little knowledge of low-level programming languages like C++, but user should just follow the described below practice while using VersatileBuffer:
VASTreaming library provides special class MediaGlobal, which contains several methods and properties to access allocators and buffers. In the very beginning of user application operation, MediaGlobal should be initialized depending on the expected load the application produces. Later, user can utilize method MediaGlobal.LockBufer() for getting VersatileBufer.
The following code demonstrates sample usage of VersatileBuffer received from an external code. Pay attention to AddRef() called when you store a sample and Release() called when sample is not needed anymore.
private Queue<VAST.Common.VersatileBuffer> inputQueue = new Queue<VAST.Common.VersatileBuffer>(); private void MediaSource_NewSample(object sender, VAST.Media.NewSampleEventArgs e) { lock (this.inputQueue) { // add sample reference e.Sample.AddRef(); this.inputQueue.Enqueue(e.Sample); } } private void processingRoutine() { while (true) { VAST.Common.VersatileBuffer inputSample = null; lock (this.inputQueue) { if (this.inputQueue.Count > 0) { inputSample = this.inputQueue.Dequeue(); } } if (inputSample != null) { // process sample data // release sample when it's not needed anymore inputSample.Release(); } } }
The following code demonstrates sample usage of VersatileBuffer received from an allocator. Pay attention that AddRef() must not be called in this case, but Release() still must be called when sample is not needed anymore.
VAST.Common.VersatileBuffer sample = VAST.Media.MediaGlobal.LockBuffer(1024 * 1024); // perform any necessary actions with allocated buffer // release sample when it's not needed anymore sample.Release();
VAST.Common.Log object is used for logging across all VASTreaming libraries. Since most of the library objects are not visual, using the log is the only way to debug and investigate possible issues. Therefore, user should initialize the log correctly and provide written logs to VASTreaming support in case of any problem.
Sample log initialization:
VAST.Common.Log.LogFileName = System.IO.Path.Combine(logFolderName, logFileName); VAST.Common.Log.LogLevel = VAST.Common.Log.Level.Debug;
Log level VAST.Common.Log.Level.Debug shall be used whenever a user needs to report an issue or request a help from VASTreaming support.
User is free to use any other logging framework in the code but in case of using the VASTreaming log, sample log line should look like this:
VAST.Common.Log.ErrorFormat("Error occurred: {0}", e.ErrorDescription);