Artificial intelligence is proving to be a very capable tool when it comes to manipulating videos of people. Face-swapping deepfakes have been the most visible example, but new applications are being found every day. The latest? Call it deepfakes for dancing. It uses AI to read someone’s dance moves and copy them on to a target body.
The actual science here was done by a quartet of researchers from UC Berkley. As they describe in a paper posted on arXiV, their system is comprised of a number of discrete steps. First, a video of the target is recorded, and a sub-program turns their movements into a stick figure. (Quite of a lot of video is needed to get a good-quality transfer: around 20 minutes of footage at 120 frames per second.) Then, a source video is found and a stick figure made of their movements. Then, the swap happens, with a neural network synthesizing video of the target individual based on the stick figure movements of source.
It sounds simple like this, of course, but there’s lots of clever engineering at work here. For example, there’s a subroutine that smooths the movement of the stick figures so the dancers don’t jerk about too much, and there’s a completely separate neural network dedicated to retracing the target’s face to ensure realism.
There are also limitations to the program. The network can’t accurately model loose fabrics, for example, so the target individual has to wear tight-fitting clothes. In the video above, you can also see quite a few visual anomalies, like moments where the joints of the target and source dancer don’t quite line up or where the AI software couldn’t reproduce complex movements, like a hand flipping from back to front.
Still, it’s impressive work. It’s the sort of video manipulation that would probably take a whole team days to produce. Now, all it takes is some source video and the right AI software. Expect it to be turned into an app before too long, and just don’t think about the possible implications about what technology like this will do to our trust in video evidence.