They stole my voice with AI

Everyone has their own fears about AI, but my fears are especially chilling; what if AI was used to imitate a person saying something blasphemeous?

My country is already has blasphemy lynching mobs based on the slightest perceived insult, real or imagined. They will mob you, lynch you, burn your corpse, then distribute sweets while you family hide and issue video messages denouncing you and forgiving the mob.

And this was before AI was easy to access. You can say a lot of things about 'oh backward countries' but this will not stay there, this will spread. You can't just give a toddler a knife and then blame them for stabbing someone.

Has nothing to do with fame, with security, with copyright. This will get people killed. And we have no tools to control this.

https://x.com/search?q=blasphemy

I fear the future.

I don't see why using AI would get around Midler vs. Ford. If anything, there is even less of an argument to be made in your defense when you use AI to replicate a voice, instead of using another voice actor to replicate the voice.

I appreciate his pointer to precedent, but the truth is that while precedent is a start, we're going to need to do work with principles beyond precedent. When tech introduces unprecedented capabilities, we will either figure out how to draw boundaries within which it (among other features of society) works for people, not against them, or we'll let it lead us closer to a world in which the strong do what they will and the weak (or those just trying to keep a camry running) suffer what they must.

If LLMs are the ultimate remix machine, then is anyone with a RAG a digital DJ?

One can't help but wonder what theft even means any more, when it comes to digital information. With the (lack of) legal precedent, it feels like the wild wild west of intellectual property and copyright law.

Like, if even a superstar like Scarlett Johansson can only write a pained letter about OpenAI's hustle to mimic her "Her" persona, what can the comparatively garden-variety niche nerd do?

Like Geerling, feel equally sad / angry / frustrated, but merely say "Please for the love of all that is good, be nice and follow an honour code.".

(Obviously not a lawyer) Overlooking the AI part, isn't it a gross misrepresentation of Jeff's opinion or an unauthorized use of his image? By using his voice, it creates an implicit (fabricated) endorsement for their product and that feels very wrong. I'm sure laws exists to deal with these cases, since way before AI existed.

Maybe I am crazy but I don't really think it sounds that much like him. It's a little similar but different. It's slightly higher pitch, more nasal, and the intonation is a little different.

I’m a small tech YouTuber and I’ve also had contact with Elecrow. As far as I know, employees (not just at Elecrow) receive rewards, promotions, or commissions when they secure long term partnerships and video collaborations with YouTubers. Perhaps someone thought it would be clever clone Jeffs voice since his channel is quite popular in this field. This certainly isn't great PR for Elecrow right now. I would also wonder if they will confess to that this was intentional...

The idea that stolen voice tones are going to matter at all is one of the shortest sighted bits of AI investment - powered by Hollywood "never make anything new" thinking.

In about 5 years AI voices will be bespoke and more pleasant to listen to then any real human: they're not limited by vocal cord stress, can be altered at will, and can easily be calibrated by surveying user engagement.

Subtly tweaking voice output and monitoring engagement is going to be the way forward.

It's all fun and games until someone produces a recording of somebody else saying something incriminating and it will be used in court. This is the part of AI I hate.

We have 100s of tools that are about voice cloning - of course we’ll get content with cloned voices.

Same as it happens with unauthorized use of someone’s images. And platforms and their moderation teams have processes in place to report and remove that. Looks like we need something similar for voice.

When you say that lawyers always cost a lot of money: I’d absolutely do this pro bono but more than likely you’re not in a state where I’m licensed.

You can absolutely positively find a free lawyer if your issue is interesting enough.

This is the most interesting issue of our day.

California banned several forms of deepfake and digital replicas without consent just days ago.

https://techcrunch.com/2024/09/19/here-is-whats-illegal-unde...

Not sure if those laws apply to Jeff tho, as they concern porn, politics and employer contracts.

Elecrow seems a Chinese company, right? In that case, I don't expect any reply.

>I haven't decided what to do.

Make a video, say what you think, get views, and probably put more pressure on Elecrow to respond.

It’s the Wild West and will be for some time but I agree, they should have the decency to use only voices of the dear departed. The library should be open source and hosted on GitHub. the talking dead seems like a good name for it. Obviously we will have to put it to a vote among the living.

Ex-Google CEO says successful AI startups can steal IP and hire lawyers to ‘clean up the mess’ / “But if nobody uses your product, it doesn’t matter that you stole all the content,” Eric Schmidt said during a recent talk at Stanford that has been taken offline.

Since that guy was CEO of Google it’s all good right???

https://www.theverge.com/2024/8/14/24220658/google-eric-schm...

More and more I am starting to wish I had gone ahead with the novel I had sketched out in the 1990s. The backdrop was a kind of post-imitative-AI collapse of trust in society, because it had become effortless to fake up, say, your least favorite political candidate talking about the merits of eating babies, so the various echo chambers bore a kind of ghastly fruit, each stance finding its own "evidence" for its beliefs, right down to the flat earth types. Paranoia runs rampant, and so on.

It looks like we're heading in that direction.

For what it's worth this seems to be a thing, for example here's a video promoting some batteries-thingie with the help of AI-voice based on a (famous, in some circles) podcast girl.

[1] https://old.reddit.com/r/redscarepod/comments/1fmiiwt/which_...

from the discourse here, the main pain point really is the accessibility to do something like this thanks to the new models.

IANAL and not sure about regional precedence on these topics, but there are plenty of ads where lookalikes or voice actors are used to use someone's likeness. they are mostly in satire, but there is yet to be a case where there was a litigation over this or prior approval needed.

we have ai-based voice abuse in the political sphere, and where there was only one legislation for banning the use in voice calls for one country (https://news.ycombinator.com/item?id=39304736), another country actively used the same underlying tech to aid their own rallies (https://news.ycombinator.com/item?id=40532157).

the tools are here to stay, but what is fair use needs to be defined more than ever.

This is exactly the reason why I'm not open sourcing a tool I developed where you can take an audio book together with an epub to build an ljspeech dataset and train a voice model.

Although it was not too hard to create I believe making it easier is something i don't like to achieve...

I hate to say this but ruining a narrators existence with AI seems to get easier every day.

What if they had someone that sounded just like you and had the person do a voice over? What if they had someone that sounded like you, had the person give them sample and used AI to generate voice? What if the hired someone that could imitate your voice, then had the person give them sample and used AI to generate voice?

I don't know where you live, but where I live I believe this sort of thing meets all the required elements for fraud.

So, do we "solve" this by impersonating a voice from a very litigious company? Let's say we clone Micky Mouse's voice to say some blasphemous things. Does that bring the full weight of Disney down on this? Will that force a change? Would it be a good change?

My aunt is (supposedly) in her 90s, having left the US for Ecuador 10 or so years ago. We've remained in regular contact via phone over the years. Recently we went more than a few months without speaking.

She has/had two numbers; magic jack and google. When I tried to call her, the magic jack was no longer in service and google said something about "unavailable".

I reached out to my cousin (my aunt's daughter) to inquire. I was told her number (and perhaps other things) had been "hacked", whatever that means. She had recently broken her hip and was in a hospital recovering.

With this on my mind, I received a call (from the google number), strangely, while processing files with GPT. My skepticism was primed and ready, possibly making me paranoid. However, I did my due diligence and asked dozens of questions, mostly boring things that she typically wouldn't have patience for. Sometimes she'd reply with a reasonable answer and sometimes not, which made it difficult to evaluate. Toward the end, I asked where she was. She said, with an awkward tempo "I'm at home, in Cuenca", which I found odd because she'd normally just say she was at home, period. I then pressed her to tell me where she was before she returned home. She said she didn't understand. I rephrased the question, stating that it was a simple inquiry, eg "where were you before going home?" She said "this is getting too strange and confusing " and killed the call.

I notified my cousin, telling her I thought something was suspicious, still cognizant of all the characteristics one would expect from a 90 year old recovering from a serious injury. My cousin might, technology wise, be in AOL territory.

About 5 days later, I received a call from my aunt, on the google line. This time,I was more passive and cautious, but again, asked dozens of boring questions to probe the situation. I was surprised by both her ability to answer certain questions and also her inability to answer some questions. I tried to ask questions on topics we'd never discussed, in case the line had been tapped for a long time and referencing was established by an imposter. I had begun to suspect I had been paranoid. But several aspects were burning me: 1) typing noises in the background 2) Shatneresque pauses for nearly every reply 3) refusal to answer some specific questions.

At the end of our apparent conversation, I asked her to do a very serious favor for me: send me a selfie, with one hand making the thumbs up gesture. She replied "I'll send you a photo of my passport ". I replied "that's stupid, ridiculous and serves no purpose. Don't do that. Understand? Do NOT send me a passport photo. I'm asking you something very important. Do exactly what I asked. Will you do this?" Her reply: "yes. What is your email address?" This was odd. I told her she already knew and it's the same one she'd had for years. She asked that I tell her anyway. Ok, 90 years old, traumatic injury, possible prescription drugs... "It's my full name @ xyzmail com". We killed the call.

I immediately called my cousin and told her of my suspicions, including some my aunt's babbling about all her finances and accounts being inaccessible. She said that was strange because she just deposited 8k into her account. Meanwhile, a notification appears in the phone, an email from my aunt. It's a photo of her passport.

Having no authority in this situation, but plenty well annoyed, I immediately jumped on a real computer and ran the photo through exiftool. The photograph was taken in 2023 and it was August of 2024. I then grabbed the geo coordinates (cryptically presented in exiftool) and with some effort, geolocated the image to right on top of her former residence, in Cuenca.

I still don't know WTF is going on and my cousin thinks I'm a dingbat. But what I know for sure, is this is an age where such things are plausible enough and will soon be inevitable. The way I think may be deranged, but I truly don't even know if my aunt still exists. But I can have a pretty compelling conversation, either with her, or something strongly resembling her, minus the Shatneresque pauses, typing noises and selective amnesia.

We should adopt and use to it. There will be more and more fake AI created content every day so we should be confrontend with it to learn how to react to it.

Regulating prolong adoption and take resources.

> I remember when OpenAI practically cloned Scarlett Johanssen's voice ...

I don't have a dog in this fight but just to be clear, OpenAI has stated that they paid a voice actor to create the voice ("Sky") that sounds like Scarlett Johanssen. There was no "cloning" or "stealing" (that they say).

https://openai.com/index/how-the-voices-for-chatgpt-were-cho...

Let's just assume you can't trust any ugc on the internet from now. It's all done, but fun whilst it lasted.

Investment in tools that can verify the authenticity of audio and video content is crucial

Slightly off topic, but what’s that logo at the bottom of his website?

Is that some sort of a coat of arms?

I want to ask and answer two of my own questions here:

1. Why clone Jeff's voice?

When I was messing with stable diffusion using Automatic1111's interface, I noticed it came with a big list of artists to add to the prompt to stylize the image in some way. There was a big row in the media about ai art reproducing artists work and many artists came forward feeling it was a personal attack. But... I mean the truth is more general than that. When I pressed a button to insert a random name into a prompt, my goal was not "yes give me this person's art for free", it was "style this somehow".

I wasn't personally interested in any particular artist, I honestly would have preferred a bunch of sliders.

Jeff here is clearly a good speaker. That's a practiced talent and voice actors exist because it's hard. Elecrow wanted a voice over and they wanted it to be as good as they could make it. Jeff is very good. So did they want Jeff?

I think what they really wanted was a good and cogent narration with the tenor of a person. Not a machine making noises that sound like english. If they had an easy way to get that, we wouldn't be talking about it here.

2. What function does copyright serve?

Well. I think a reasonable argument would be that if people were able to reproduce your work for free, you would quickly find yourself without a monetary incentive to make more of it.

So. What happens if you combine answer 1 with answer 2?

I think it leads to: "We should consider making it illegal to automatically reproduce the work of an artisan.", you know, the luddic argument. An argument that has been perceived to be, more or less, settled.

So it seems to me: That for individuals, harms matter, and for society, it doesn't.

Yeah it's pretty much an inevitability that anyone who's ever posted more than 10 seconds of their voice will have it stolen eventually if anyone has a motive to do it. That's already all it takes to few shot a decent TTS _today_.

Most likely all existing youtubers will have complete voice and video digital clones made out of them. Then you can also tune an LLM on their scripts and it'll respond in the same character as well.

In theory you could also bring back ones who are dead, which would be very interesting in a historical sense. Like if we had hundreds of hours of Napoleon talking in front of a camera, it would be trivial to recreate a digital version of him for anthropologic study, maybe even having various figures debate things with each other. That's what historians a century later after we all die will be able to do with impunity.

how do people know if its a very similar sounding voice, or identical?

We’ve been in a post-truth world for T least ten years anyway.

We already had fake news and organizations willingly spread fake news.

We had clearly fake pictures and people believing that.

Flat-earthers, no-vax and whatever.

This is just another brick in the wall.

It's not even close to his voice lmao, just has similar cadence.

>I remember when OpenAI practically cloned Scarlett Johanssen's voice...

There is absolutely zero evidence for this. I find it infuriating that this keeps being stated as a fact. So they go and hire a voice actor and clearly use her voice to train, but then they also scrape Scarlett Johansson from youtube and splice it into the training data to make the voice a bit more like hers? Really does that sound realistic?

> I remember when OpenAI practically cloned Scarlett Johanssen's voice

Except that never happened and the voice belonged to a completely different voice actress and Scarlett Johanssen had exactly zero right to prevent this person from making money as a voice actress lending it to AI.

These complaints remind me a little bit of the story that a man complained that his photo was used to illustrate the article about how all hipsters look the same and it eventually turned out it wasn't his photo.

Initially I'd say well if you're a public figure and upload your own voice online, of course this will happen. So its something to expect, however, this shouldn't be a problem for Jeff to solve... instead it should be YouTube's problem as they profit from the video monetization. Eventually they'll have to have some kind of detection for all uploaded content.

This whole argument rests on the absurd assumption that you can "own" a voice as if it's property. Does this mean people can own the patterns of vibration in air? It's completely nonsensical.

It is like becoming meme famous: is up to you how to monetize it, nobody owes you nothing.