POKE 53280,0: Pete Brown's Blog

Silverlight 3 – Creating Video from Raw Bits using a MediaStreamSource

There are a number of ways to get pixels on the screen in Silverlight 3. One you may not have considered, especially with the new Bitmap API being an otherwise obvious choice, is the newly enhanced MediaStreamSource API

The MediaStreamSource API was available in Silverlight 2, but it required you to transcode into a supported format. The new version of the API lets you create raw video images much more easily. There are a number of use-cases for this including complete bitmap-based games, creating chroma-keyed/alpha video, and in my case, using it to show the display of my Commodore 64 emulator at 50 frames per second (PAL standard)

(You can also use the MediaStreamSource to create audio from raw bits. Look for that in another upcoming post.)

So how hard is this to do? Well, once you get the basic setup down, it’s actually really easy. The thing to remember is your code will be called by Silverlight, not the other way around. Therefore you just need to respond appropriately when, for example, Silverlight requests the next frame.

First, you want to create a class that inherits from MediaStreamSource

public class VideoMediaStreamSource : MediaStreamSource

Pixel Buffers

The emulated MOSS 6569 VIC-II chip (the equivalent of a video card today) in my code pushes pixels out to my media stream source until it reaches the end of the current frame. Once that happens, it calls Flip(). I keep two pixel buffers around to eliminate any tearing or half-updates on the screen where Silverlight requests a frame that is not yet complete. One buffer is what the VIC is writing to, the other is what Silverlight is reading from. When the VIC calls Flip(), I swap the two.

private byte[][] _frames = new byte[2][];

public VideoMediaStreamSource(int frameWidth, int frameHeight)
{
    _frameWidth = frameWidth;
    _frameHeight = frameHeight;

    _framePixelSize = frameWidth * frameHeight;
    _frameBufferSize = _framePixelSize * BytesPerPixel;

    // PAL is 50 frames per second
    _frameTime = (int)TimeSpan.FromSeconds((double)1 / 50).Ticks;

    _frames[0] = new byte[_frameBufferSize];
    _frames[1] = new byte[_frameBufferSize];

    _currentBufferFrame = 0;
    _currentReadyFrame = 1;
}

(yes, I could have done something clever with xor here)

public void Flip()
{
    int f = _currentBufferFrame;
    _currentBufferFrame = _currentReadyFrame;
    _currentReadyFrame = f;
}

Writing a pixel is pretty easy, as long as you get the format correct. Notice the order of the component R, G, B and A parts. BytesPerPixel is defined as 4 as this is a 32 bit color value.

public void WritePixel(int position, Color color)
{
    int offset = position * BytesPerPixel;

    _frames[_currentBufferFrame][offset++] = color.B;
    _frames[_currentBufferFrame][offset++] = color.G;
    _frames[_currentBufferFrame][offset++] = color.R;
    _frames[_currentBufferFrame][offset++] = color.A;

}

Opening your Video 

Surprisingly, this is much easier for video than it is for audio. For audio, you need to fill out a whole WaveFormatEx structure. For video, you simply need to set the appropriate FourCC code (like RGBA or YV12), the frame width and the frame height.

protected override void OpenMediaAsync()
{
    // Init
    Dictionary<MediaSourceAttributesKeys, string> sourceAttributes = 
        new Dictionary<MediaSourceAttributesKeys, string>();
    List<MediaStreamDescription> availableStreams = 
        new List<MediaStreamDescription>();

    PrepareVideo();

    availableStreams.Add(_videoDesc);

    // a zero timespan is an infinite video
    sourceAttributes[MediaSourceAttributesKeys.Duration] = 
        TimeSpan.FromSeconds(0).Ticks.ToString(CultureInfo.InvariantCulture);

    sourceAttributes[MediaSourceAttributesKeys.CanSeek] = false.ToString();

    // tell Silverlight that we've prepared and opened our video
    ReportOpenMediaCompleted(sourceAttributes, availableStreams);
}

private void PrepareVideo()
{
    _frameStream = new MemoryStream();

    // Stream Description 
    Dictionary<MediaStreamAttributeKeys, string> streamAttributes = 
        new Dictionary<MediaStreamAttributeKeys, string>();

    streamAttributes[MediaStreamAttributeKeys.VideoFourCC] = "RGBA";
    streamAttributes[MediaStreamAttributeKeys.Height] = _frameHeight.ToString();
    streamAttributes[MediaStreamAttributeKeys.Width] = _frameWidth.ToString();
    
    MediaStreamDescription msd = 
        new MediaStreamDescription(MediaStreamType.Video, streamAttributes);

    _videoDesc = msd;
}

Requesting Frames

Ignore for a moment that the code checks for both audio and video here. While I have snipped some of that from the other examples above, the code is actually set up to eventually serve up the audio stream from the Silverlight SID chip as well as the video from the VIC chip.

protected override void GetSampleAsync(MediaStreamType mediaStreamType)
{
    if (mediaStreamType == MediaStreamType.Audio)
    {
        GetAudioSample();
    }
    else if (mediaStreamType == MediaStreamType.Video)
    {
        GetVideoSample();
    }
}

Now in the GetVideoSample() method, I’m creating a new stream with each request. I don’t like the approach, but I was getting out of memory errors with early Silverlight 3 builds if I tried to reuse the stream and simply provide offsets into it. I’ll debug this some more before posting the final code.

private void GetVideoSample()
{
    _frameStream = new MemoryStream();
    _frameStream.Write(_frames[_currentReadyFrame], 0, _frameBufferSize);

    // Send out the next sample
    MediaStreamSample msSamp = new MediaStreamSample(
        _videoDesc,
        _frameStream,
        0,
        _frameBufferSize,
        _currentVideoTimeStamp,
        _emptySampleDict);

    _currentVideoTimeStamp += _frameTime;

    ReportGetSampleCompleted(msSamp);
}

One key part of the code above is the _currentVideoTimeStamp += _frameTime. Frame Time is a constant equal to 1/50 of a second in 100 nanosecond intervals. Setting the frametime there tells Silverlight that we’re serving up 50 frames per second. I haven’t tried varying that number, but you could probably get some interesting performance characteristics by playing around with that in real-time.

Wiring it Up

Hooking it up to Silverlight couldn’t be easier:

<MediaElement x:Name="VideoDisplay"
              Grid.Row="0"
              Grid.Column="0"
              Stretch="Uniform"
              IsHitTestVisible="False"
              Margin="4" />

VideoDisplay.SetSource(_c64.Display.Video);

_c64.Display.Video is an instance of my MediaStreamSource class.

Conclusion

So there you have it, runtime-generated video inside a Silverlight 3 application. Pretty cool if you ask me!

I’ll make all of this code available as part of various projects on codeplex later. Stay tuned. In the mean time, the complete source code for this one class is available here.