UPDATE (15/07/2014) – The sample is updated based on the public preview SDK.
One of the biggest feature requests was the ability to track the expressions of the users. Today I’m happy to tell you that this is now available in the alpha SDK thanks to the facetracking!
In this post I will walk you through the steps to display the expressions for one user but this is possible for all the tracked persons!
I developed a small template that displays the camera so you can follow along & is available here.
Setting up expression tracking is pretty easy – We just need to set up body tracking, assign a FaceFrameSource to it and start processing the results. This requires us to add two references – Microsoft.Kinect for body tracking & Microsoft.Kinect.Face for the face tracking.
As I mentioned in my basic overview we need to create a BodyFrameReader to start receiving BodyFrameReferences in the FrameArrived event.
/// Body reader
public BodyFrameReader _bodyReader;
/// Collection of all tracked bodies
public Body _bodies;
/// Initialize body tracking
private void InitializeBodyTracking()
// Body Reader
_bodyReader = _kinect.BodyFrameSource.OpenReader();
// Wire event
_bodyReader.FrameArrived += OnBodyFrameReceived;
Next we need to determine what FaceFrameFeatures we will use and create a FaceFrameSource & FaceFrameReader as a global variable.
/// Requested face features
private const FaceFrameFeatures _faceFrameFeatures = FaceFrameFeatures.BoundingBoxInInfraredSpace
/// Face Source
private FaceFrameSource _faceSource;
/// Face Reader
private FaceFrameReader _faceReader;
Once the BodyFrameReferences arrive we need to create a new FaceFrameSource based on our _kinect-instance. We’ll assign the requiested face features & the TrackingId of the first tracked body to the source, this is how a face is being linked to a certain body.
Next we will create a new FaceFrameReader-instance and start listening to the FrameArrived & TrackingIdLost-events. Note – This only supports one user
/// Process body frames
private void OnBodyFrameReceived(object sender, BodyFrameArrivedEventArgs e)
// Get Frame ref
BodyFrameReference bodyRef = e.FrameReference;
if (bodyRef == null) return;
// Get body frame
using (BodyFrame frame = bodyRef.AcquireFrame())
if (frame == null) return;
// Allocate array when required
if (_bodies == null)
_bodies = new Body[frame.BodyCount];
// Refresh bodies
foreach (Body body in _bodies)
if (body.IsTracked && _faceSource == null)
// Create new sources with body TrackingId
_faceSource = new FaceFrameSource(_kinect)
FaceFrameFeatures = _faceFrameFeatures,
TrackingId = body.TrackingId
// Create new reader
_faceReader = _faceSource.OpenReader();
// Wire events
_faceReader.FrameArrived += OnFaceFrameArrived;
_faceSource.TrackingIdLost += OnTrackingIdLost;
Once everything is set up and the sensor detects the requested face, based on the tracking ID, the methodology is pretty much the same – Retrieve the reference, acquire a frame & process the data.
Here I’m reading all the face properties and displaying them in the UI.
private void OnFaceFrameArrived(object sender, FaceFrameArrivedEventArgs e)
// Retrieve the face reference
FaceFrameReference faceRef = e.FrameReference;
if (faceRef == null) return;
// Acquire the face frame
using (FaceFrame faceFrame = faceRef.AcquireFrame())
if (faceFrame == null) return;
// Retrieve the face frame result
FaceFrameResult frameResult = faceFrame.FaceFrameResult;
// Display the values
HappyResult.Text = frameResult.FaceProperties[FaceProperty.Happy].ToString();
EngagedResult.Text = frameResult.FaceProperties[FaceProperty.Engaged].ToString();
GlassesResult.Text = frameResult.FaceProperties[FaceProperty.WearingGlasses].ToString();
LeftEyeResult.Text = frameResult.FaceProperties[FaceProperty.LeftEyeClosed].ToString();
RightEyeResult.Text = frameResult.FaceProperties[FaceProperty.RightEyeClosed].ToString();
MouthOpenResult.Text = frameResult.FaceProperties[FaceProperty.MouthOpen].ToString();
MouthMovedResult.Text = frameResult.FaceProperties[FaceProperty.MouthMoved].ToString();
LookingAwayResult.Text = frameResult.FaceProperties[FaceProperty.LookingAway].ToString();
Copying the Nui database
Last step to get this working is to copy the NuiDatabase to the output folder, without it the values will always be “No”.
We will use a simple post-build event in our project settings that will copy it for us -
xcopy "C:\Program Files (x86)\Microsoft SDKs\Windows\v8.0\ExtensionSDKs\Microsoft.Kinect.Face\2.0\Redist\CommonConfiguration\x64\NuiDatabase" "NuiDatabase" /e /y /i /r
The result should look like the following -
My guess is that this database contains all the values that the SDK will use to detect happiness, wearing glasses, etc. but I haven’t found documentation on this.
Lost track of body
People come and go, this also means that the sensor will lose track of a body, everytime this occures the event ‘TrackingIdLost’ is thrown where we will blank out the values and reset our variables.
private void OnTrackingIdLost(object sender, TrackingIdLostEventArgs e)
// Update UI
HappyResult.Text = "No face tracked";
EngagedResult.Text = "No face tracked";
GlassesResult.Text = "No face tracked";
LeftEyeResult.Text = "No face tracked";
RightEyeResult.Text = "No face tracked";
MouthOpenResult.Text = "No face tracked";
MouthMovedResult.Text = "No face tracked";
LookingAwayResult.Text = "No face tracked";
// Reset values for next body
_faceReader = null;
_faceSource = null;
Testing the application
When you give the application a spin this is how it should look like –
In this post I illustrated how easy it is to set up expression tracking for one person and what it allows you to do f.e. user feedback when they see a new product at a conference.
Keep in mind that the sensor is able to track up to six persons and your algorithm should support this as well.
Download my full code sample here.
Thanks for reading,