Welcome to the very first Kinecting for Windows tutorial.

Introduction

The goal of this application is to create a television where you are in the picture!
It starts with a static tv untill you plug your Kinect in & it is connected, that’s where you jump in.
You’ll be able to see yourself and change the angle or output type in the settings panel.
Last but least, when you disconnect your sensor it will go back to static and vice versa so that our application is more stable as well.

Pre-requisites

• Basic C# skills
• A Kinect for Xbox/Windows sensor
• Installed Kinect SDK v1.7 (more info)
• You’ve read my intro to the SDK (post)

What you will learn

• Connect a Kinect sensor
• Enable Color-data
• Show camera feed
• Tilt the camera
• Selecting camera output at runtime
• Supporting different Kinect states

Screenshots

Start of the application where you have a static television.

I connected my Kinect and see myself. Note the settings pane on the right.

Settings panel where you can change the angle of the sensor or the output format of the camera.

Fasten your seatbelt

Before we can start we’ll need to install the latest version of the Kinect for Windows SDK v1.7 as I explained in this blogpost.
I’ll use Visual Studio Express 2012 for Windows Desktop with C# & XAML for this tutorial.

In order to create this application I prepared a template with my UI in it.
You will notice that my Image-control is mirrored since our the data we will get will also be mirrored. We will only need MainPage.xaml.cs where we will code.

You can download my template here or Visual Studio Express 2012 for Windows Desktop here.

Connecting you sensor

First things first, we need to reference the SDK first since the ‘Microsoft.Kinect’ isn’t a built-in namespace by right clicking on References > Add reference > Browse to C:/Program Files/Microsoft SDKs/Kinect/v1.6/Assemblies/Microsoft.Kinect.dll or type it in the search bar and include a ‘using’ in MainWindow.xaml.cs.

using Microsoft.Kinect;

Now it’s time to get our KinectSensor-object that represents a Kinect sensor but we only want one if it’s KinectStatus is Connected.
We can do this by querying KinectSensors, a static field that holds all KinectSensor objects, with LINQ.

        public MainWindow()
        {
            InitializeComponent();

            KinectSensor currentSensor = KinectSensor.KinectSensors.FirstOrDefault(sensor => sensor.Status == KinectStatus.Connected);
        }

Next step is to start our sensor by enabling the ColorStream & start listening when new frames are coming in so we can process them.
It’s not a bad idea to check if the provided KinectSensor is null to avoid crashed.

private KinectSensor _currentSensor = null; 

private void StartSensor(KinectSensor sensor)
{
     // Avoid crashes
     if (_currentSensor != null)
         return;

     // Show our Image-control for the output
     Output.Visibility = Visibility.Visible;

     // Save into global
     _currentSensor = sensor;

     // Enable color-stream
     _currentSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

     // Listen to colorFrameReady-event
     _currentSensor.ColorFrameReady += OnColorFrameReadyHandler;

     // Start the sensor
     _currentSensor.Start();
}

Now it’s time to pass our sensor-object to our StartSensor-method in the constructor –

public MainWindow()
{
    InitializeComponent();

    KinectSensor currentSensor = KinectSensor.KinectSensors.FirstOrDefault(sensor => sensor.Status == KinectStatus.Connected);

    StartSensor(currentSensor);
}

Show our camera feed

Our sensor is started with an enabled ColorStream and ready to be processed in our OnColorFrameReadyHandler.

Next step is to pull out the colordata from a ColorImageFrame and show it in our television.
To do that we need to allocate a byte array that will temporary hold our image data because we will write it to a WriteableBitmap later on.
Last but not least, we will link the WriteableBitmap to the Image-control the first time we receive a ColorImageFrame.

I’m using a WriteableBitmap because I only need to create a new instance of it once and write a new set of pixels to it every frame. By doing that performance will be a bit better.

Every thrown ColorFrameReady-event we’ll be able to grab our ColorImageFrame, that contains our data, from the event arguments by using the method OpenColorImageFrame().
It’s possible that our frame is null because we are too late to process this frame or something went wrong so we need to check this to avoid crashes.

First time we get a ColorImageFrame we need to create a byte array with to correct size, initiate a new WriteableBitmap & link it to our Image-control.
We can do this by checking if the size of our byte array is still 0, if so we can allocate a byte array based on the PixelDataLength of our frame.
Our WriteableBitmap needs to be as wide & tall as our frame with a standard DPI of 96 and we need to specify a PixelFormat which is in this case Bgr32. This value is based on the ColorImageFormat you are using when you enable the ColorStream.
We’re not going to use a BitmapPallette so we pass in a null-value.

Once all this is set we are ready to process our data, which is actually really easy!
We copy all the pixel data to our array by calling the CopyPixelDataTo-method of our frame and write those pixels to our WriteableImage.
This requires a Int32Rect that specifies how big the image needs to be along with the byte array with our pixel data, how big our stride is and where we want to start reading.

private WriteableBitmap _outputBitmap = null;
private byte[] _pixelData = new byte[0];

private void OnColorFrameReadyHandler(object sender, ColorImageFrameReadyEventArgs e)
{
    using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
    {
         // The frame will be null if we are too late to process it
         if (colorFrame == null)
            return;

         // Checks if the length is 0 (first time we receive a frame)
         if (_pixelData.Length == 0)
         {
             // Creates a buffer long enough to receive all the data of a frame
             _pixelData = new byte[colorFrame.PixelDataLength];

             // Initialize new WriteableBitmap
             _outputBitmap = new WriteableBitmap(colorFrame.Width,
                                                 colorFrame.Height,

                                                 // Standard DPI
                                                 96, 96,

                                                 // Current format for the ColorImageFormat
                                                 PixelFormats.Bgr32,

                                                 // BitmapPalette
                                                 null);
                    
             // Assign the writeablebitmap to the imagecontrol
             Output.Source = _outputBitmap;
         }

         // Copies the data from the frame to the pixelData array
         colorFrame.CopyPixelDataTo(_pixelData);

         // Update the writeable bitmap
         _outputBitmap.WritePixels(

             // Represents the size of our image
             new Int32Rect(0, 0, colorFrame.Width, colorFrame.Height),

             // Our image data
             _pixelData,

             // Stride - How much bytes are there in a single row?
             colorFrame.Width * colorFrame.BytesPerPixel,

             // Offset for the buffer, where does he need to start
             0);
    }
}

When you run your current application you should get something like this, yes it already works!

Tilting the camera

As you can see on the screenshots there is a slider in the settings panel where you can change the angle of the sensor.
Before we can change the angle of the sensor we should adjust our slider, called ‘ElevationSlider’, with the current value when we start the sensor.
We can do this by getting the value of the ElevationAngle-value of our current KinectSensor-object.

private void SetBasicElevationLevel()
{
    if (_currentSensor != null)
        ElevationSlider.Value = _currentSensor.ElevationAngle;
}

Changing the angle of the sensor is as easy as getting the value, we just need to listen to our click-event and put the slider value in the ElevationAngle-property of our KinectSensor.

private void OnChangeElevationClick(object sender, RoutedEventArgs e)
{
    if (_currentSensor != null)
        _currentSensor.ElevationAngle = (int)ElevationSlider.Value;
}

NOTE - It is not recommended to use databinding for this since you shouldn’t change the angle too much. Changing the angle should only be used to set a correct angle when for example launching the application because if you change it too often your motor could break.

Selecting camera output at runtime

There are different camera output types called ColorImageFormat that define what color format you want when you enable your ColorStream, in our case RgbResolution640x480Fps30.
You can change the output at runtime in the settings panel, well that’s what we’re going to do now.

We need to populate our combobox with some formats when we launch the application before we query for our sensor.

private void LoadFormats()
{
    IList<ColorImageFormat> colorImageFormat = new List<ColorImageFormat> 
                                                {
                                                    ColorImageFormat.RgbResolution640x480Fps30,
                                                    ColorImageFormat.RgbResolution1280x960Fps12,
                                                    ColorImageFormat.YuvResolution640x480Fps15,
                                                    ColorImageFormat.InfraredResolution640x480Fps30,
                                                    ColorImageFormat.Undefined
                                                };

    FormatComboBox.ItemsSource = colorImageFormat;
    FormatComboBox.SelectedIndex = 0;
}

Now need to listen to the SelectionChanged-event of the combobox so we can disable the ColorStream and enable the ColorStream with the new selected format or collapse the output when the selected item is Undefined since this won’t give you any frames.

private void OnSelectedFormatChanged(object sender, SelectionChangedEventArgs e)
{
   if (_currentSensor != null && _currentSensor.Status == KinectStatus.Connected && e.AddedItems.Count > 0)
   {
        // Disable the color stream for a second
        _currentSensor.ColorStream.Disable();

        // Retrieve the selected format
        ColorImageFormat selectedFormat = (ColorImageFormat)e.AddedItems[0];

        // Check if the ColorImageFormat is different from Undefined, if so, show that it is unsupported
        if (selectedFormat == ColorImageFormat.Undefined)
        {
            Output.Visibility = Visibility.Collapsed;
        }
        else
        {
             Output.Visibility = Visibility.Visible;

             // Enable the stream with the new stream
             _currentSensor.ColorStream.Enable(selectedFormat);
        }
    }
}

Now that our format is changing at runtime we need to change our code in OnColorFrameReadyHandler since the PixelDataLength & PixelFormat of one format can be different from another.
We need to reallocate our byte array everytime its length is different than PixelDataLength instead of checking if it’s still 0.
Also instead of passing ‘PixelFormats.Bgr32′ to our WriteableBitmap we can determine a PixelFormat based on the FramebytesPerPixel property of our frame. In our case it can Bgr32 when there are 4 bytes or Gray16 in other cases based on the formats in the combobox.

private void OnColorFrameReadyHandler(object sender, ColorImageFrameReadyEventArgs e)
{
    using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
    {
        if (colorFrame == null)
            return;

        // Checks if the length has changed (means the format has changed)
        if (_pixelData.Length != colorFrame.PixelDataLength)
        {
            // Creates a buffer long enough to receive all the data of a frame
            _pixelData = new byte[colorFrame.PixelDataLength];

            /* Most of the formats provided by the kinect are 4-bytes per pixel
             * The infrared stream only has 2 bytes so we'll have to check that*/
            PixelFormat currentFormat = _currentSensor.ColorStream.FrameBytesPerPixel == 4
                                        ? PixelFormats.Bgr32
                                        : PixelFormats.Gray16;

            _outputBitmap = new WriteableBitmap(colorFrame.Width,
                                                colorFrame.Height,
                                                96, 96,

                                                 // Current format for the ColorImageFormat
                                                 currentFormat,

                                                 null);
                    
            Output.Source = _outputBitmap;
        }

        colorFrame.CopyPixelDataTo(_pixelData);

        _outputBitmap.WritePixels(
               new Int32Rect(0, 0, colorFrame.Width, colorFrame.Height),
               _pixelData,
               colorFrame.Width * colorFrame.BytesPerPixel,
               0);
     }
}

Supporting different Kinect states

Our application is not stable at the moment.
Imagine that your Kinect wasn’t plugged in when you started the application or your connection to the sensor drops, it will crash your application.

We need to support adding, removing or changing states of our sensor so that our application is a fully functional television and shutdown our sensor when the program is closing.

Let’s start by disabling our sensor when our application throws an Closing-event, that’s as easy as pie. We just need to check if the our sensor is not null & if it is connected so we can stop it.

void OnClosing(object sender, System.ComponentModel.CancelEventArgs e)
{
    if (_currentSensor != null && _currentSensor.Status == KinectStatus.Connected)
        _currentSensor.Stop();
}

Now when a sensor is changing his state, called KinectStatus, KinectSensor.KinectSensors will throw an event called StatusChanged where we will need to start our Kinect or stop it, based on the new state.
First of all, we need to start listening to the event.

KinectSensor.KinectSensors.StatusChanged += OnKinectSensorChanged;

Every time the status of our sensor changes we want to check if it is our current sensor that changed it status or if it is a new one.
We only want to start the new instance of the sensor when it is our current sensor that is Connected or if our _currentSensor is null and the new instance is Connected.
This is because if it’s not connected we will pass a null value which means we want to show our static image since the sensor isn’t connected and we won’t be able to get any color data from it.

private void OnKinectSensorChanged(object sender, StatusChangedEventArgs e)
{
    if (e.Sensor == _currentSensor)
    {
        if (e.Status != KinectStatus.Connected)
        {
            StartSensor(null);
        }
        else StartSensor(e.Sensor);
    }
    else if ((_currentSensor == null) && (e.Sensor.Status == KinectStatus.Connected))
        StartSensor(e.Sensor);
}

Last but not least, we need to change our StartSensor-method to support null values by stopping the current sensor and hiding the output image so that we see the static image.
It also needs to get the selected format from the combobox so that when we connect a new sensor it will use that format instead of our default format we used before.
Once again, if it is Undefined we hide our output image to show the static image.

private void StartSensor(KinectSensor sensor)
{
    // Stop if still running
    if (_currentSensor != null)
        _currentSensor.Stop();

    // Get currently selected format
    ColorImageFormat selectedFormat = (ColorImageFormat)FormatComboBox.SelectedItem;

    if (sensor == null)
    {
        // Hide output window
        Output.Visibility = Visibility.Collapsed;

        // Save 'null'
        _currentSensor = null;

        return;
    }

    // Save into global
    _currentSensor = sensor;
            
    if (selectedFormat == ColorImageFormat.Undefined)
    {
        // Hide output window
        Output.Visibility = Visibility.Collapsed;
    }
    else
    {
        // Show output window
        Output.Visibility = Visibility.Visible;

        // Enable color-stream
        _currentSensor.ColorStream.Enable(selectedFormat);

        // Listen to colorFrameReady-event
        _currentSensor.ColorFrameReady += OnColorFrameReadyHandler;

        // Start the sensor
        _currentSensor.Start();

        // Set basic angle
        SetBasicElevationLevel();
    }
}

Conclusion

In this tutorial we learned how to connect our sensor, enable & consume our color stream. Next to that we enabled to tilt the motor and select the output format for our sensor at runtime.
Last but not least we stabilized our application by supporting changing states of our current sensor and adding & removing devices.

I’m always open for feedback or suggestions so mail me at hello@kinectingforwindows.com

You can find my code on GitHub.

Tom.