Moving face 3D reconstruction

The researchers from the University of Washington made an interesting presentation during the European Conference on Computer Vision (ECCV-2014). They described a technology of video frame 3D reconstruction.

Using YouTube video, this program automatically builds highly detailed 3D-models for each video frame.

Given the complexity of the problem, this is quite an impressive result, since facial mimicry is very difficult to reproduce. It is important to catch the exact position of the eyes, bending eyebrows, wrinkles, etc. The slightest error in such 3D-model becomes visible in no time.

The vast majority of other software for 3D-tracking and face digitization is based on the method of blend shapes, when object’s shape changes and «flows» from one station to another. The method of smooth modifications has a big disadvantage – the loss of small details that are so important for face perception. The authors of new algorithm abandoned this approach.

On the other hand, although during the video frame reconstruction every single model is «separate» from each other, when playing it 30/60 frames per second the result should be more realistic than in the case of smooth modifications.

And even more: unlike other similar technologies, this one doesn’t require human involvement in the test shooting. Instead, with the help of 3D optical flow method, the program uses a large archive of his or her photographs with different lighting and poses. The authors of the research say that nowadays each person has a large archive of photographs.

The program has proven successful under various lighting and with different facial expressions.

The work was presented at the European Conference on Computer Vision, held on September 6–12, in Zurich.

Scientific article about the new technology
Supplement materials

This technology can be used in cinematography, 3D-printing, computer games and virtual reality; it may be used also to recreate the exact appearance and facial expressions of the deceased by their photos and videos.

For now, the creation of more realistic 3D-animation movies and characters’ facial expressions in computer games seems the most direct application of this new technology.