Send Sample 2

The SendSample2 extends Send Sample 1 by demonstrating how to send both video and audio streams simultaneously using pre-encoded H.264 video and AAC audio. It shows how to parse raw bitstreams to extract codec configuration, find frame boundaries, and push individual frames with correct timestamps.

Overview

The SendSample2 class performs the following:

Loads pre-encoded H.264 video and AAC audio bitstreams from embedded resources
Parses the H.264 bitstream to extract resolution, framerate, and codec private data
Generates AAC codec private data from the audio media type
Creates an RtmpPublisherSink targeting a remote RTMP server
Registers both streams with the sink and opens it asynchronously
Paces video and audio frames against real-time on a background task, looping both streams continuously

Parsing Codec Configuration

H.264 Video

VAST.Codecs.H264.ConfigurationParser h264Parser = new Codecs.H264.ConfigurationParser();
h264Parser.Parse(this.videoBuffer);

this.videoMediaType = new VAST.Common.MediaType
{
    ContentType = VAST.Common.ContentType.Video,
    CodecId = VAST.Common.Codec.H264,
    Bitrate = 800000,
    Width = h264Parser.Width,
    Height = h264Parser.Height,
    Framerate = h264Parser.FrameRate,
    PixelAspectRatio = h264Parser.AspectRatio,
};

VAST.Codecs.H264.ConfigurationParser.GenerateCodecPrivateData(this.videoMediaType, this.videoBuffer);

ConfigurationParser parses the raw H.264 Annex B bitstream to extract SPS/PPS parameters — resolution, framerate, and pixel aspect ratio. GenerateCodecPrivateData stores the SPS and PPS NAL units in the media type's CodecPrivateData field, which is required by the sink to set up the stream correctly.

AAC Audio

this.audioMediaType = new VAST.Common.MediaType
{
    ContentType = VAST.Common.ContentType.Audio,
    CodecId = VAST.Common.Codec.AAC,
    SampleRate = 44100,
    Channels = 2,
    Bitrate = 128000,
};

VAST.Codecs.AAC.ConfigurationParser.GenerateCodecPrivateData(this.audioMediaType);

ConfigurationParser generates AAC codec private data (AudioSpecificConfig) from the sample rate and channel count.

Creating the Sink

this.sink = new VAST.RTMP.RtmpPublisherSink();
this.sink.Uri = "rtmp://127.0.0.1/live/test";
this.sink.Error += sink_Error;
this.sink.StateChanged += sink_StateChanged;

this.sink.AddStream(this.videoStreamIndex, this.videoMediaType);
this.sink.AddStream(this.audioStreamIndex, this.audioMediaType);

this.sink.Open();

RtmpPublisherSink implements IMediaSink and publishes media to a remote RTMP server. Both streams are registered via AddStream before opening — video on stream index 0, audio on stream index 1.

Sink State Management

private void sink_StateChanged(object sender, Media.MediaState e)
{
    lock (this)
    {
        switch (e)
        {
            case VAST.Media.MediaState.Opened:
                ((VAST.Media.IMediaSink)sender).Start();
                this.pushingTask = Task.Run(() => pushingRoutine());
                break;
            case VAST.Media.MediaState.Started:
                // sink has been started
                break;
            case VAST.Media.MediaState.Closed:
                // sink has been disconnected
                break;
        }
    }
}

When the sink reaches the Opened state, Start() is called and the pushing routine is launched on a background task.

Pushing Video and Audio

The pushing routine maintains two independent timelines — one for video and one for audio — both paced against the wall clock.

Video Frame Extraction

int bitstreamPosition = h264BitstreamPosition + 4;
int startCodeSize = 0;
int nextNalPosition = 0;
bool keyFrame = false;

while ((nextNalPosition = VAST.Codecs.H264.ConfigurationParser.FindNextStartCode(
    videoBuffer, bitstreamPosition,
    videoBuffer.Length - bitstreamPosition, out startCodeSize)) >= 0)
{
    VAST.Codecs.H264.NalUnitTypes nalUnit =
        (Codecs.H264.NalUnitTypes)(videoBuffer[nextNalPosition + startCodeSize] & 0x1F);

    if (nalUnit == Codecs.H264.NalUnitTypes.AccessUnitDelimiter)
    {
        break; // found next frame
    }
    else
    {
        if (nalUnit == Codecs.H264.NalUnitTypes.SliceIdr)
        {
            keyFrame = true;
        }
        bitstreamPosition = nextNalPosition + startCodeSize;
    }
}

H.264 frames are delimited by scanning for Access Unit Delimiter NAL units using FindNextStartCode. Each frame may contain multiple NAL units (SPS, PPS, slice data). IDR slices are detected to mark key frames.

Pushing a Video Frame

VAST.Common.VersatileBuffer packet = VAST.Media.MediaGlobal.LockBuffer(frameSize);
packet.Append(videoBuffer, h264BitstreamPosition, frameSize);
packet.Pts = packet.Dts = videoFileTime;
packet.KeyFrame = packet.CleanPoint = keyFrame;
packet.StreamIndex = this.videoStreamIndex;

this.sink.PushMedia(this.videoStreamIndex, packet);
packet.Release();

Video timestamps are calculated from the frame count and framerate:

long videoFileTime = videoFrameCount * 10000000L
    * videoMediaType.Framerate.Den / videoMediaType.Framerate.Num;

Audio Frame Extraction

int frameSize = ((audioBuffer[audioBitstreamPosition + 3] & 0x03) << 11)
    | (audioBuffer[audioBitstreamPosition + 4] << 3)
    | ((audioBuffer[audioBitstreamPosition + 5] & 0xE0) >> 5);

VAST.Common.VersatileBuffer packet = VAST.Media.MediaGlobal.LockBuffer(frameSize - 7);
packet.Append(audioBuffer, audioBitstreamPosition + 7, frameSize - 7);
packet.Pts = packet.Dts = audioFileTime;
packet.KeyFrame = packet.CleanPoint = true;
packet.StreamIndex = this.audioStreamIndex;

this.sink.PushMedia(this.audioStreamIndex, packet);
packet.Release();

AAC frames are extracted from an ADTS bitstream. The frame size is read from the ADTS header (bytes 3–5), and the 7-byte ADTS header is stripped before pushing the raw AAC frame data. Audio timestamps are calculated from the frame count and sample rate:

long audioFileTime = audioFrameCount * 10000000L * 1024 / audioMediaType.SampleRate;

The constant 1024 is the number of PCM samples per AAC frame.

Real-Time Pacing

do
{
    // push video frame if due
    // push audio frame if due
}
while (videoFileTime <= playbackTime || audioFileTime <= playbackTime);

Task.Delay(10).Wait();

The inner loop pushes all video and audio frames whose timestamps have fallen behind wall-clock time, then sleeps for 10 ms before checking again. Both streams loop independently when they reach the end of their respective buffers.

Table of Contents