2020.09 - 2020.11

This is a research project advised by Professor Brian A. Barsky at University of California, Berkeley

Tools: Blender, Python, OpenCV

Table of Contents

• Project Overview
• Technical Overview
  • Haar Cascade
  • Facial Landmark API
  • Perspective-n-Point
• What Could Go from Here

Project Overview

Blendemoji, used on Vincent, an Opensource Blender Model available on Blender Cloud

Blendemoji is a Python scripted plugin for Blender which reads the user’s facial expression through Webcam and simulates it on a 3d rigged model.

Technical Overview

The idea is simple:

Extracting data from the user’s face input -> interpret the data and apply it onto the model.

Haar Cascade

I start by recognizing rectangular faces in our webcam picture.

Haar Cascade is a machine learning object detection algorithm used to identify objects in an image or video and based on the concept of features proposed by Paul Viola and Michael Jones in their paper “Rapid Object Detection using a Boosted Cascade of Simple Features” in 2001.

The picture on the right demonstrate the idea. Top row shows two good features. The first feature selected seems to focus on the property that the region of the eyes is often darker than the region of the nose and cheeks. The second feature selected relies on the property that the eyes are darker than the bridge of the nose.

In OpenCV library the algorithm applied by calling function void detectMultiScale()

void cv::CascadeClassifier::detectMultiScale
(	InputArray 				image,
	std::vector< Rect > &	objects,
	double 					scaleFactor = 1.1,
	int 					minNeighbors = 3,
	int 					flags = 0,
	Size 					minSize = Size(),
	Size 					maxSize = Size() 
)

Facial Landmark API

with my detected landmark feature. I used Facial Landmark API to further recognize different facial components.

Facial Landmark API uses “Active Appearance Models” algorithm to match statistical models of appearance to images.

Active Appearance Models Algorithm is a method of matching statistical models of appearance to images, developed by Timothy F. Cootes, Gareth J. Edwards, and Christopher J. Taylor. (Their Paper)[https://people.eecs.berkeley.edu/~efros/courses/AP06/Papers/cootes-pami-01.pdf]

Documentation for the API

To detect landmarks we use function virtual bool cv::face::Facemark::fit()

virtual bool cv::face::Facemark::fit	(
	InputArray			image,
	InputArray			faces,
	OutputArrayOfArrays	landmarks 
)

Now we have our data extracted, let’s try to interpret data and apply it on our models.

Perspective-n-Point

My data extraction happens on a 2D plane image, the next step is to convert it into 3D space.

Perspective-n-Point is the problem of estimating the pose of a calibrated camera given a set of n 3D points in the world and their corresponding 2D projections in the image. The camera pose consists of 6 degrees-of-freedom (DOF) which are made up of the rotation (roll, pitch, and yaw) and 3D translation.

Let’s break down the equation:

S is the scale factor for the image point

Homogeneous Image Point: the point on the 2D image (In this case it’s the coordinate of user’s facial component on the image)

Intrinsic Camera Matrix: the intrinsic parameters of our camera (In this case it’s the webcam, the physical pinhole camera). f here is the focal length, u v is the coordinate of principle points which are commonly zero. Gamma is the skew coefficient which is also commonly zero.

Homogeneous World Point: the point on the 3D model (In this case it’s the coordinate of the specific bone corresponding to the facial component)

And that leaves us with the matrix on the middle, the [R T] matrix. Which can be broke down as 3D Rotation and Translation which can be applied onto the model.

bool cv::solvePnP	(	
	InputArray 	objectPoints,
	InputArray 	imagePoints,
	InputArray 	cameraMatrix,
	InputArray 	distCoeffs,
	OutputArray	rvec,
	OutputArray	tvec,
	bool 		useExtrinsicGuess = false,
	int 		flags = SOLVEPNP_ITERATIVE 
)

And… Done!

What Could Go from Here

I personally enjoyed the whole process developing this project. And as a game designer I would really love to see what would happen if we can gamify this. Because it’s such a unique way of human computer interaction. And it reminds me of the game Ring Fit Adventure which a strap on players’ thig to have body movement as input. The game was created to encourage people working out their bodies.

If the same design philosophy can be applied here, we might create a game to excourage people working out their facial muscles as a way to prevent facial paralysis, etc.