The horror/wonder of AI video translation

How AI killed the uncanny valley

Thanks to AI, I can now speak Spanish! And Chinese, and Tagalog, and even Welsh.

I only recently discovered the world of AI dubbing - the use of artificial intelligence to translate audio from video into another language, complete with changes to the lips/mouth movements that make it appear as if the person is a fluent speaker of a foreign language. No user input required; just upload a video to an app or website, and after a minute or so of processing, download a new video of yourself speaking a different language. Here’s an example of AI dubbing I made - pay attention to how perfectly my AI-generated mouth movements match the new languages:

While the translation is not perfect, and the voices in the target languages are not that similar to mine, this is still a mind-blowing situation to a Gen-Xer like me. And something I didn’t even know existed until yesterday.

But the truly mind-blowing part of this is how not freaked out I am by that AI manipulated version of me. It doesn’t feel creepy to see/hear myself speaking AI Spanish, or AI Dutch, or AI English. Why? Because there is no indication that something is “wrong” - nothing about my facial anatomy says unnatural/CGI. The me in that video still looks and moves like the real me. There is nothing to trigger the uncanny valley effect.

Likely you’ve heard of the uncanny valley, but if not, the basic idea is this: when something looks nearly identical to a real human but contains a feature that betrays their status as a non-human, it makes us feel uncomfortable. It’s most noticeable when it comes to faces.

The Polar Express is often trotted out as the worst/best example of the uncanny valley phenomenon. The intention of the animators was to create realistic-looking CGI animated humans, but there is something off about the way these characters’ eyes and mouths both look and move. It’s that almost but not quite human feeling that generates a yuck repones. Check out this clip from the Polar Express and see if that yuck response happens for you:

Not everyone finds these nightmare-children as creepy as I do. That’s because the uncanny valley response is not universal in how it manifests, and some people rarely or never experience it. Case in point: Reborn Dolls.

Reborn Dolls are extremely lifelike silicon dolls that look and feel just like a human baby. Many people recoil in horror at these dolls because of the uncanny valley - they are so lifelike that when it’s revealed that they are fake, the mind experiences an explosion of cognitive dissonance. But other people find them adorable and charming, experiencing no uncanny valley effect whatsoever. Here is an example of a Reborn Doll - what emotions do you experience when watching this video?

It’s the head-squishing thing that gets me. Truly disturbing!

Now, had AI done a poor job creating my language dub video—maybe a situation where my mouth moved in a weird, inhuman way (like the Polar Express kids)—I might have recoiled, overcome by the uncanny valley effect. The reason that didn’t happen was because there was no head-squishing moment; there was nothing to tip me off that the me in the video was fake or manipulated in some way. AI has reached a stage in video manipulation that it can bypass the uncanny valley. And that’s both amazing and terrifying.

The uncanny valley was one of the last lines of defenses we had against the spread of deepfakes on the internet. It had been easy to spot CGI generated art online—even photorealistic art—because of consistent weirdness that betrays the artificiality of it all. AI usually sucks at generating human hands, for example, which is why you get the uncanny creepiness of the hands in this otherwise realistic looking AI generated photo:

Courtesy of Midjourney

But lately, AIs are increasingly able to solve the creepiness problem. As I demonstrated with my video, we now live in a world where AI can generate realistic videos of you speaking a language you don’t speak. And that video will be devoid of uncanny valley triggers.

There are a million ways in which this new normal will be a boon to society. Sometimes these synthetic humans manifest as fun social media influencers, like the virtual human Lil Miquela, with 2.5 million followers on Instagram:

lilmiquela

A post shared by @lilmiquela

Or the Ukraine's Foreign Ministry AI spokesperson, Victoria Shi, who can read the news in any language and is available for newsreading duties 24/7. Or AI/CGI characters used in art and cinema, like a de-aged Harrison Ford in Indiana Jones and the Dial of Destiny, or the use of digital stunt doubles (virtual CGI clones of actors) in movie scenes that are just to dangerous to film with real humans.

But I probably don’t even need to explain the ways in which the breakdown of the uncanny valley in online media is a problem. If you keen to generate disinformation or propaganda as part of your job, say, working for a dictatorial regime that wants to destroy the global foundations of democracy by undermining the general public’s faith in electoral systems (or maybe you just can’t think of anything else to do on a Wednesday night), then this is a great time to be alive generating artificial media. You can pump the internet full of fake audio/videos of world leaders doing and saying things they never said or did (like this fake robocall from “Joe Biden” urging people to not vote in the New Hampshire primary). Or manipulate photos and video of active war zones to make your side seem like the good guys and not the baddies. Or maybe just post fake nudes of your enemies.

All if this is extra disturbing because even a luddite-adjacent Gen-Xer like me can do it with a few clicks of a mouse button.

RIP uncanny valley. I’m sorry to see you go. Truly.

*****

Life update! The uncanny valley and AI stuff are subjects I will be exploring in my upcoming book: Humanish. I’ve just completed my first draft of the book, which is a BFD. I hope to write more regularly about these subjects in this newsletter over the coming months.

Also: I had a very kind message from a reader saying that they’d be keen to support this newsletter financially. I am quite happy to have this newsletter freely available for everyone and am not keen to monetize it, but I am also aware of the value of providing opportunities for people to support artists/writers/creators whose work they enjoy. It feels good to toss money at art one likes, so if you are enjoying this newsletter and want to toss money at me, I have decided to activate the paid subscription function to facilitate that. But I won’t put any content behind a paywall; this is solely for folks to send extra cash as a token of appreciation.

If you are a kindly benefactor keen to keep wordsmiths afloat, I also urge you to check out other writers on Substack. I have been so comically derelict in my use of Substack that I am only following one other writer on here: Tove Danovich, who wrote the book Under the Henfluence (and is a fellow Ologist) and now writes the A Little Detour newsletter for Substack. I promise to find more more writer-pals to shout out as I up my Substack game. In the meantime, you should check out and support Tove: