MP4 Reader Sample

The FileReaderSample demonstrates how to use IsoSource to read an MP4 media file, decode its video and audio streams into raw pixel data (BGRA) and PCM audio (S16), and process decoded frames in application code. It also shows interactive features such as seeking and variable playback rate.

Overview

The FileReaderSample class performs the following:

Creates an IsoSource with parsing parameters
Opens the file asynchronously and seeks to the midpoint on the Opened state
Detects streams via NewStream and creates decoders for video and audio
Decodes incoming samples directly in the NewSample handler
Delivers decoded raw data to processData() for application-specific processing

Creating the Source

this.source = new VAST.File.ISO.IsoSource(
    new File.ISO.ParsingParameters { PreserveNals = true });

this.source.Uri = filePath;
this.source.PlaybackRate = double.MaxValue;

this.source.NewStream += source_NewStream;
this.source.NewSample += source_NewSample;
this.source.Error += source_Error;
this.source.StateChanged += source_StateChanged;

this.source.Open();

IsoSource implements IInteractiveMediaSource — an extended media source interface that adds seeking, pausing, looping, and variable playback rate on top of the standard IMediaSource.

Property	Value	Description
`PreserveNals`	`true`	Preserve all NAL units from the original H.264/H.265 bitstream (by default, delimiter NAL units are stripped per ISO standard)
`PlaybackRate`	`double.MaxValue`	Read content as fast as possible instead of real-time pacing

Setting PlaybackRate to double.MaxValue is useful for batch processing scenarios (thumbnail generation, transcoding, analysis). For real-time playback, omit this property or set it to 1.0.

Source State Management

private void source_StateChanged(object sender, Media.MediaState e)
{
    lock (this)
    {
        switch (e)
        {
            case VAST.Media.MediaState.Opened:
                this.source.Position = new TimeSpan(this.source.Duration.Ticks / 2);
                this.source.Start();
                break;
            case VAST.Media.MediaState.Started:
                // source has been started, samples will come soon
                break;
            case VAST.Media.MediaState.Closed:
                // source has been disconnected
                break;
        }
    }
}

When the source reaches the Opened state, the sample seeks to the midpoint of the file using the Position property and then calls Start() to begin reading. The Duration property returns the total file duration.

IInteractiveMediaSource Properties

Property	Type	Description
`Position`	`TimeSpan`	Gets or sets the current playback position (seeking)
`Duration`	`TimeSpan`	Gets the total duration of the media file
`PlaybackRate`	`double`	Gets or sets the playback speed multiplier
`CanSeek`	`bool`	Whether the source supports seeking
`CanPause`	`bool`	Whether the source supports pausing
`Loop`	`bool`	Whether to loop playback when reaching end of file

Stream Detection and Decoder Creation

When a new stream is detected, a decoder is created based on the content type.

Video Decoder

case VAST.Common.ContentType.Video:
    if (e.MediaType.CodecId == VAST.Common.Codec.Uncompressed
        && e.MediaType.PixelFormat == VAST.Common.PixelFormat.BGRA)
    {
        streamContext.PerformDecoding = false;
    }
    else
    {
        VAST.Common.MediaType desiredVideoMediaType = new VAST.Common.MediaType
        {
            ContentType = VAST.Common.ContentType.Video,
            CodecId = VAST.Common.Codec.Uncompressed,
            PixelFormat = VAST.Common.PixelFormat.BGRA,
            Width = e.MediaType.Width,
            Height = e.MediaType.Height,
        };

        streamContext.Decoder = VAST.Media.DecoderFactory.Create(
            e.MediaType, desiredVideoMediaType, decoderParameters);
        streamContext.PerformDecoding = true;
    }
    break;

Video is decoded to uncompressed BGRA pixel data at the source resolution. If the source already provides uncompressed BGRA, decoding is skipped.

Audio Decoder

case VAST.Common.ContentType.Audio:
    if (e.MediaType.CodecId == VAST.Common.Codec.PCM
        && e.MediaType.SampleFormat == VAST.Common.SampleFormat.S16)
    {
        streamContext.PerformDecoding = false;
    }
    else
    {
        VAST.Common.MediaType desiredAudioMediaType = new VAST.Common.MediaType
        {
            ContentType = VAST.Common.ContentType.Audio,
            CodecId = VAST.Common.Codec.PCM,
            SampleFormat = VAST.Common.SampleFormat.S16,
            SampleRate = e.MediaType.SampleRate,
            Channels = e.MediaType.Channels,
        };

        streamContext.Decoder = VAST.Media.DecoderFactory.Create(
            e.MediaType, desiredAudioMediaType, decoderParameters);
        streamContext.PerformDecoding = true;
    }
    break;

Audio is decoded to PCM signed 16-bit samples at the source sample rate and channel count.

Decoder Parameters

VAST.Media.DecoderParameters decoderParameters = new VAST.Media.DecoderParameters
{
    PreferredMediaFramework = useFFmpeg
        ? VAST.Common.MediaFramework.FFmpeg
        : VAST.Common.MediaFramework.Builtin,
    AllowHardwareAcceleration = allowHardwareAcceleration,
};

Field	Default	Description
`useFFmpeg`	`false`	Use FFmpeg for decoding instead of the built-in framework
`allowHardwareAcceleration`	`true`	Allow GPU-accelerated decoding

Decoding Samples

Unlike the Receive Sample which uses a background decoding task with a sample queue, this sample decodes directly in the NewSample event handler since IsoSource delivers samples from its own internal thread:

private void source_NewSample(object sender, VAST.Media.NewSampleEventArgs e)
{
    StreamContext streamContext = this.streams[inputSample.StreamIndex];

    if (streamContext.PerformDecoding)
    {
        if (streamContext.FirstFrame)
        {
            if (inputSample.KeyFrame)
            {
                streamContext.FirstFrame = false;
            }
            else
            {
                return; // drop until first keyframe
            }
        }

        streamContext.Decoder.Write(inputSample);

        while (true)
        {
            VAST.Common.VersatileBuffer decodedSample = streamContext.Decoder.Read();
            if (decodedSample == null) break;

            this.processData(decodedSample);
            decodedSample.Release();
        }
    }
    else
    {
        this.processData(inputSample);
    }
}

After a seek, the first frames may not be key frames. The sample drops non-key frames until the first key frame arrives, ensuring the decoder starts with a clean reference point. The decoder follows the standard write/read pattern — write a compressed sample, then read decoded frames in a loop.

Processing Decoded Data

private void processData(VAST.Common.VersatileBuffer sample)
{
    // TODO: process decoded, uncompressed data
}

This is where application-specific processing goes:

Video — sample contains a raw BGRA pixel array that can be used to create a bitmap, fed into a computer vision pipeline, or rendered to screen
Audio — sample contains a raw PCM signed 16-bit buffer for audio analysis, playback, or further processing

Optional: Thumbnail Extraction

The sample includes a commented-out example of extracting a single video frame as a JPEG thumbnail using ImageExtractor:

var extractor = new VAST.File.Utility.ImageExtractor();
var image = await extractor.Execute(filePath, TimeSpan.FromSeconds(30));
if (image != null)
{
    using (var stream = System.IO.File.OpenWrite("thumbnail.jpg"))
    {
        image.Encode(SkiaSharp.SKEncodedImageFormat.Jpeg, 80).SaveTo(stream);
    }
}

ImageExtractor.Execute opens the file, seeks to the specified position, decodes a single video frame, and returns it as a SkiaSharp SKImage. The decoder framework can be configured via extractor.DecoderParameters.

Table of Contents