Kinect for Windows SDK 2.0 day – Public Preview availability & looking back at MVA

UPDATE 23/07/2014 – The MVA sessions are available on-demand here!

Note- Some of the information provided was already covered in previous articles, I recommend reading them to get deeper into that topic.

Yesterday was a big day for Kinect for Windows – Ben Lower & Rob Relyea hosted a Microsoft Virtual Academy on the day that the public preview of the Kinect for Windows SDK 2.0 has been released & is available here!

Public Preview SDK

The free SDK allows you to get started with the new sensor that leverages improved skeletal tracking, higher depth fidelity, 1080p HD video, new active infrared capabilities, extended field of view, and so on.

This allows the SDK to be more powerful as wel f.e. improved skeletal to track hand states and more joints or even use the highly requested face expression tracking!

Source – Kinect for Windows blog

The SDK now ships with a new version of Kinect Studio that allows you to record clips and play them without being connected to a sensor, this is a really big win! Next to that you can use one sensor in multiple apps at the same time!

Source – Kinect for Windows blog

Last but not least you can now build Windows Store or Unity apps for Kinect as well!

Some of my latests posts already covered the alpha version of the SDK that give you a kickstart, feel free to read them! The SDK also ships with a lot of samples for WPF, Store apps & C++!

If you have any questions regarding the public preview, feel free to post them on the forum or ask me!

Looking back at the Microsoft Virtual Academy

The academy covered seven modules going from an introduction to data sources, face tracking, advanced topics like custom sources and so on. Pretty nice online event that covered most of the features that are available!

The modules will be available for on-demand later on, I recommend watching them!

Source – Kinect for Windows blog

Updated blog samples

I’m glad to announce that my samples have been update in GitHub to run with the preview SDK, if you notice bugs – feel free to let me know!

You can read the official statement here or order a sensor here!

Thanks for reading,

Tom Kerkhove

Posted in News | Leave a comment

First look at Expressions – Displaying expressions for a tracked person

UPDATE (15/07/2014) – The sample is updated based on the public preview SDK.

One of the biggest feature requests was the ability to track the expressions of the users. Today I’m happy to tell you that this is now available in the alpha SDK thanks to the facetracking!

In this post I will walk you through the steps to display the expressions for one user but this is possible for all the tracked persons!

Template

I developed a small template that displays the camera so you can follow along & is available here.

Tutorial

Setting up expression tracking is pretty easy – We just need to set up body tracking, assign a FaceFrameSource to it and start processing the results. This requires us to add two references – Microsoft.Kinect for body tracking & Microsoft.Kinect.Face for the face tracking.

As I mentioned in my basic overview we need to create a BodyFrameReader to start receiving BodyFrameReferences in the FrameArrived event.

/// <summary>
/// Body reader
/// </summary>
public BodyFrameReader _bodyReader;

/// <summary>
/// Collection of all tracked bodies
/// </summary>
public Body[] _bodies;
		
/// <summary>
/// Initialize body tracking
/// </summary>
private void InitializeBodyTracking()
{
    // Body Reader
    _bodyReader = _kinect.BodyFrameSource.OpenReader();

    // Wire event
    _bodyReader.FrameArrived += OnBodyFrameReceived;
}

Next we need to determine what FaceFrameFeatures we will use and create a FaceFrameSource & FaceFrameReader as a global variable.

/// <summary>
/// Requested face features
/// </summary>
private const FaceFrameFeatures _faceFrameFeatures = FaceFrameFeatures.BoundingBoxInInfraredSpace
														| FaceFrameFeatures.PointsInInfraredSpace
														| FaceFrameFeatures.MouthMoved
														| FaceFrameFeatures.MouthOpen
														| FaceFrameFeatures.LeftEyeClosed
														| FaceFrameFeatures.RightEyeClosed
														| FaceFrameFeatures.LookingAway
														| FaceFrameFeatures.Happy
														| FaceFrameFeatures.FaceEngagement
														| FaceFrameFeatures.Glasses;

/// <summary>
/// Face Source
/// </summary>
private FaceFrameSource _faceSource;

/// <summary>
/// Face Reader
/// </summary>
private FaceFrameReader _faceReader;

Once the BodyFrameReferences arrive we need to create a new FaceFrameSource based on our _kinect-instance. We’ll assign the requiested face features & the TrackingId of the first tracked body to the source, this is how a face is being linked to a certain body.

Next we will create a new FaceFrameReader-instance and start listening to the FrameArrived & TrackingIdLost-events. Note – This only supports one user

/// <summary>
/// Process body frames
/// </summary>
private void OnBodyFrameReceived(object sender, BodyFrameArrivedEventArgs e)
{
    // Get Frame ref
    BodyFrameReference bodyRef = e.FrameReference;

    if (bodyRef == null) return;

    // Get body frame
    using (BodyFrame frame = bodyRef.AcquireFrame())
    {
        if (frame == null) return;

        // Allocate array when required
        if (_bodies == null)
            _bodies = new Body[frame.BodyCount];

        // Refresh bodies
        frame.GetAndRefreshBodyData(_bodies);

        foreach (Body body in _bodies)
        {
            if (body.IsTracked && _faceSource == null)
            {
                // Create new sources with body TrackingId
                _faceSource = new FaceFrameSource(_kinect)
                                  {
                                       FaceFrameFeatures = _faceFrameFeatures,
                                       TrackingId = body.TrackingId
                                  };

                // Create new reader
                _faceReader = _faceSource.OpenReader();

                // Wire events
                _faceReader.FrameArrived += OnFaceFrameArrived;
                _faceSource.TrackingIdLost += OnTrackingIdLost;
            }
        }
    }
}

Once everything is set up and the sensor detects the requested face, based on the tracking ID, the methodology is pretty much the same – Retrieve the reference, acquire a frame & process the data.

Here I’m reading all the face properties and displaying them in the UI.

private void OnFaceFrameArrived(object sender, FaceFrameArrivedEventArgs e)
{
    // Retrieve the face reference
    FaceFrameReference faceRef = e.FrameReference;

    if (faceRef == null) return;

    // Acquire the face frame
    using (FaceFrame faceFrame = faceRef.AcquireFrame())
    {
        if (faceFrame == null) return;

        // Retrieve the face frame result
        FaceFrameResult frameResult = faceFrame.FaceFrameResult;

        // Display the values
        HappyResult.Text = frameResult.FaceProperties[FaceProperty.Happy].ToString();
        EngagedResult.Text = frameResult.FaceProperties[FaceProperty.Engaged].ToString();
        GlassesResult.Text = frameResult.FaceProperties[FaceProperty.WearingGlasses].ToString();
        LeftEyeResult.Text = frameResult.FaceProperties[FaceProperty.LeftEyeClosed].ToString();
        RightEyeResult.Text = frameResult.FaceProperties[FaceProperty.RightEyeClosed].ToString();
        MouthOpenResult.Text = frameResult.FaceProperties[FaceProperty.MouthOpen].ToString();
        MouthMovedResult.Text = frameResult.FaceProperties[FaceProperty.MouthMoved].ToString();
        LookingAwayResult.Text = frameResult.FaceProperties[FaceProperty.LookingAway].ToString();
    }
}

Copying the Nui database

Last step to get this working is to copy the NuiDatabase to the output folder, without it the values will always be “No”.

We will use a simple post-build event in our project settings that will copy it for us -

xcopy "C:\Program Files (x86)\Microsoft SDKs\Windows\v8.0\ExtensionSDKs\Microsoft.Kinect.Face\2.0\Redist\CommonConfiguration\x64\NuiDatabase" "NuiDatabase" /e /y /i /r

The result should look like the following -
post-build event

My guess is that this database contains all the values that the SDK will use to detect happiness, wearing glasses, etc. but I haven’t found documentation on this.

Lost track of body

People come and go, this also means that the sensor will lose track of a body, everytime this occures the event ‘TrackingIdLost’ is thrown where we will blank out the values and reset our variables.

private void OnTrackingIdLost(object sender, TrackingIdLostEventArgs e)
{
    // Update UI
    HappyResult.Text = "No face tracked";
    EngagedResult.Text = "No face tracked";
    GlassesResult.Text = "No face tracked";
    LeftEyeResult.Text = "No face tracked";
    RightEyeResult.Text = "No face tracked";
    MouthOpenResult.Text = "No face tracked";
    MouthMovedResult.Text = "No face tracked";
    LookingAwayResult.Text = "No face tracked";

    // Reset values for next body
    _faceReader = null;
    _faceSource = null;
}

Testing the application

When you give the application a spin this is how it should look like –
result

Conclusion

In this post I illustrated how easy it is to set up expression tracking for one person and what it allows you to do f.e. user feedback when they see a new product at a conference.

Keep in mind that the sensor is able to track up to six persons and your algorithm should support this as well.

Download my full code sample here.

Thanks for reading,

Tom.

Posted in Kinect for Windows Developer Program, Second Generation Kinect for Windows, Tutorial | Tagged | 2 Comments

dotnetConf – Kinect for Windows introduction by Ben Lower

dotnetconf_logo
Last week Ben Lower gave a wonderful introduction session on Kinect for Windows Gen. II for dotnetConf!

He talked about the differences with the first sensor, gave an introduction to Kinect Studio, talked about some scenarios, how you can use interactions in Windows Store apps and the sample that are part of the SDK

You can watch the session here or attend the Kinect for Windows jump start on 15th of July!

Posted in Presentation, Second Generation Kinect for Windows | Leave a comment

Microsoft Virtual Academy – ‘Programming the Kinect for Windows Jump Start’ announced

UPDATE 23/07/2014 – The MVA sessions are available on-demand here!

With the pre-ordering of Kinect for Windows Gen. II announced comes a new beta SDK that will be available for everyone!

A new SDK means a new way of talking to the Kinect, there for the Kinect for Windows team is hosting a Microsoft Virtual Academy session called “Programming the Kinect for Windows Jump Start” on 15th of July 2014 from 9am – 5pm PDT.

Devs, are you looking forward to building apps with Kinect for Windows v2 this summer? In this Jump Start, explore the brand new beta Software Development Kit with experts from the Kinect engineering team, and see how Kinect v2 enables speech, gesture, and human understanding in applications and experiences.

​Learn about the new APIs and app model, and see fascinating demos and samples (plus source code) for both desktop and Windows Store apps. Get the details on Kinect Fusion (real-time 3D modeling), Face Tracking, and Visual Gesture Builder. Discover the new sensor technology, natural user interface (NUI), accessibility potential, and practical applications.

Even if you don’t have a Kinect device, you won’t want to miss this entertaining event. The instructors even show you how you can start building an app without a sensor. Be sure to bring your questions!

Course outline

Here are the topics that will be covered in this course -

  • ​Introducing Kinect Development
  • Kinect Data Sources and Programming Model
  • Kinect Interactions and Speech
  • Using Kinect with Other Frameworks or Libraries
  • Face Tracking, HD Face, and Kinect Fusion
  • Kinect Studio and Visual Gesture Builder
  • Advanced Topics: Custom Sources, Filtering, and More

Presenters, Registration & More information

This jumpstart will be hosted by Ben Lower, yes the Kinect Ninja that has been working closely with the community to embrace all the developers & creative minds to take Kinect a step further!

He brings Rob Relyea that has been working on the runtime API & Kinect Studio but mainly focusses on Kinect Interaction!

More information about the session, speakers or anything else can be found here.

Want to join? Don’t hesitate and sign up on the link above!

Join prepared!

Since the release of the developer kit I occasionally blogged about my adventures with it – What has changed from a hardware & SDK perspective, how you can get started, etc.

Here is a list of all my posts on Kinect for Windows Gen. II -

  • Second Gen. Kinect for Windows – What’s new? (Read more)
  • [Tutorial] Gen. II Kinect for Windows – Basics Overview (Read more)
  • Kinect for Windows Gen. II hardware is revealed! (Read more)
  • //BUILD/ 2014 – Introduction to Kinect, releasing this summer and support for Unity & Windows Store apps (Read more)
  • Gen. II Kinect for Windows – Comparing MultiSourceFrameReader and XSourceFrameReader (Read more)
  • Gen. II Kinect for Windows – Introducing ‘Frames Monitor’ (Read more)

Good luck,

Tom.

Posted in News, Second Generation Kinect for Windows | 1 Comment

Pre-order your Kinect for Windows Gen. II sensor now!

Today marks a new milestone towards the release of the Kinect for Windows Gen. II – The Kinect for Windows team is proud release the pre-order of the sensor in 23 different countries!

The sensors will be shipped in July to everyone from the preview program & who pre-ordered one in the store. The first beta version of the SDK will be released as well!

Wonderful news to hear since they are a few months ahead of the general availability of the sensor & SDK. Congratulations to the Kinect for Windows team!

Read the offical blog post here and don’t hesitate to order your sensor here!

Posted in News, Second Generation Kinect for Windows | 1 Comment

Gen. II Kinect for Windows – Introducing ‘Frames Monitor’

UPDATE (15/07/2014) – The sample is updated based on the public preview SDK.

In my previous post I introduced you to the MultiSourceFrameReader and how it syncs the frames between different data sources.

What is Frames Monitor?

I created a simple tool that visualizes the amount of frames/second that are received when using the MultiSourceFrameReader in a certain environment.

It enables you to run the test in a specific environment and take in account how much of the frames you are going to receive. This result will depend on the whether you selected the color data streams & what the amount of light is in the scene.

This tool is only supported with the Gen. II Kinect for Windows and is created with C# & WPF and uses the MVVM-pattern.

You can download this tool here.

Screenshots

This monitor has been run in an environment where there was a lot of light in the room and the result is a total of 30 FPS.
This monitor has been run in an environment where there was a not much light in the room and the result is a total of 15 FPS.

Happy to help,

Tom.

Posted in Kinect for Windows Developer Program, Second Generation Kinect for Windows, Tools | Leave a comment

Gen. II Kinect for Windows – Comparing MultiSourceFrameReader and XSourceFrameReader

In a previous post I talked about how you can implement the basic data streams into your application. Next to that I also explained the new streaming model they introduced with the frame readers for each type of stream, f.e. ColorSourceFrameReader.
New stream model
With the new model came an equivalent to the AllFramesReady event called a MultiSourceFrameReader that is capable of reading on several different data streams at once.

In this post I will go deeper in the differences between the MultiSourceFrameReader and the single SourceFrameReaders.

Disclaimer

I am bound to the Kinect for Windows Developer Program and can not share the new SDK/DLL.

“This is preliminary software and/or hardware and APIs are preliminary and subject to change”.

MultiSourceFrameReader vs XSourceFrameReader

In my basics overview I visualized the camera with a Body overlay that showed a couple of joints by using code that is similar to the one below -

private KinectSensor _kinect = KinectSensor.Default;

private ColorFrameReader _colorReader;
private BodyFrameReader _bodyReader;

private void StartKinect()
{
    // Open Kinect-stream
    _kinect.Open();

    // Open readers
    _colorReader = _kinect.ColorFrameSource.OpenReader();
    _bodyReader = _kinect.BodyFrameSource.OpenReader();

    // Hook-up frame arrived events
    _colorReader.FrameArrived += OnColorFrameArrivedHandler;
    _bodyReader.FrameArrived += OnBodyFrameArrivedHandler;
}

private void OnColorFrameArrivedHandler(object sender, ColorFrameArrivedEventArgs e)
{
    // Retrieve frame reference
    ColorFrameReference colorRef = e.FrameReference;

    // Acquire the color frame
    using (ColorFrame colorFrame = colorRef.AcquireFrame())
    {
        // Check if frame is still valid
        if (colorFrame == null) return;

        // Process Color-frame
    }
}

private void OnBodyFrameArrivedHandler(object sender, BodyFrameArrivedEventArgs e)
{
    // Retrieve frame reference
    BodyFrameReference bodyRef = e.FrameReference;

    // Acquire the color frame
    using (BodyFrame bodyFrame = bodyRef.AcquireFrame())
    {
        // Check if frame is still valid
        if (bodyFrame == null) return;
        
        // Process Body-frame
    }
}

What we did was use one reader dedicated to one type of data and process it there, leaving us with two readers. This means that our readers can be out of sync, f.e. one reader skipped a frame. In our case this isn’t an issue but it can be very important in some scenarios.

In the first generation each frame had a property FrameNumber that you could use to determine if both data frames are in sync. In the second generation this has been replaced with a RelativeTime that indicates the correlation between frames.

Kinect v1 offered a more efficient way that does this for use by using the AllFramesReady-event that includes all the enabled stream data, I’m glad to tell you that this concept is also available in the new source/reader concept by using a MultiSourceFrameReader.

We can easily replace our two XSourceFrameReader with one MultiSourceFrameReader where we specify the data streams by using the FrameSourceTypes-enumeration. The rest of the processing is pretty much the same – You listen to the MultiSourceFrameArrived-event, you retrieve the MultiSourceFrameReference and the MultiSourceFrame. Now with the MultiSourceFrame you are able to retrieve the corresponding frames for your selected FrameSourceTypes.

Here is an example on how you can use a MultiSourceFrameReader –

private KinectSensor _kinect = KinectSensor.Default;

private MultiSourceFrameReader _multiReader;

private void StartKinect()
{
    _kinect = KinectSensor.Default;

    // Open Kinect-stream
    _kinect.Open();

    // Open multi source frame reader
    _multiReader = _kinect.OpenMultiSourceFrameReader(FrameSourceTypes.Color | FrameSourceTypes.Body);

    // Hook-up the frames arrived event
    _multiReader.MultiSourceFrameArrived += OnMultipleFramesArrivedHandler;
}

/// <summary>
/// Process the frames from all the sources
/// </summary>
private void OnMultipleFramesArrivedHandler(object sender, MultiSourceFrameArrivedEventArgs e)
{
    // Retrieve multisource frame reference
    MultiSourceFrameReference multiRef = e.FrameReference;

    using (MultiSourceFrame multiFrame = multiRef.AcquireFrame())
    {
        if (multiFrame == null) return;
                     
        // Retrieve data stream frame references
        ColorFrameReference colorRef = multiFrame.ColorFrameReference;
        BodyFrameReference bodyRef = multiFrame.BodyFrameReference;

        using (ColorFrame colorFrame = colorRef.AcquireFrame())
        {
            using (BodyFrame bodyFrame = bodyRef.AcquireFrame())
            {
                if (colorFrame == null || bodyFrame == null) return;

                // Process the frames
            }
        }
    }
}

Conclusion

You should try to use the MultipleSourceFrameReader when your data should be synced or use the RelativeTime-property to check the correlation.
If this is not required you can still use the XSourceFrameReader for that specific data streams.

You can also use a combination of several MultiSourceFrameReader & XSourceFrameReaders depending on your requirements.

Thanks for reading,

Tom.

Posted in Code, Kinect for Windows Developer Program, SDK, Second Generation Kinect | 1 Comment