Disrupting a covert Iranian influence operation

CNN just did a piece on private Israeli groups doing the exact same thing. It is pretty scary the shear scale of this. Literally any entity or group can spin up a ton of bots and use any AI service locally or otherwise, to attempt to sway public opinion.

These jokers seem like the AI version of "script kiddie" hackers, and OpenAI may be engaging in a bit of humble bragging. It doesn't take considerable investments in time or money to run local LLMs, INCLUDING ChatGPT, where your questions, prompts, and results are not sent home to the mothership, so it's a BS article as to (the real) actors who may or may not be doing this. NOW, if OpenAI or Gemini or LLama, etc, showed how they analyzed social media posts and flagged the ones that were AI generated and the analysis as to WHY the article is flagged, then that would be much more useful, actionable by at least some of the readers and would put the accounts spreading the content (particularly the rebroadcast fluffers) in the spotlight.

From a Google search it looks like this is one of the articles in question: https://teorator.com/index.php/2024/08/12/x-censors-trumps-t...

I requested the Internet Archive grab this copy: https://web.archive.org/web/20240816210620/https://teorator....

The headline should be "OpenAI publicly admits it supported Iran influence operation despite the sanctions"

I wonder if it would be possible to get a list of countries that can have influence operations using ChatGPT and countries that can't.

One area that OpenAI did not comment on, was the likelihood that the AI-generated content here was itself used in the training data for a later model.

Let's say the OpenAI engineers are working on ChatGPT 5 and spent last month scraping Teorater and X/Twitter, where this material ended up. How does OpenAI know that the new model is not poisoned?

This isn't just OpenAI's problem of course. Anyone training on the open Internet now has this problem.

Its great that openAI is using this infraction as an opportunity to posture about how open they are and how they are a company that can tame evil applications of AI, while totally missing and not addressing the broader concern, what if this was run on a local instance?? How are we stopping / spotting and squatting that?!

Side note. This is some pretty terrible propaganda. The post about Kamala, immigrants and climate change barely makes any sense.

X just hosted trump for a live stream, who is being affected by a headline that reads ‘X censors trump tweet”.

I'd be more interested in an analysis of the likely intention of the campaign. Is it just an attempt to reduce voter turnout? If so, that doesn't seem all that useful by itself.

This is as much an indictment of ChatGPT as it is of the Iranians. According to OpenAI, their product produces output that no one in their right mind would want to read for any purpose.

The linked PDF (Storm-2035 [1]) from Microsoft is more detailed and interesting than the blog post. However, what's missing from the reports is how they detected those operations and how they tied them to different groups. There's a lot of claims being made without showing all of the supporting evidence.

To give them the benefit of the doubt, they likely want to keep their detection methods secret to make circumvention more difficult. And it all sounds totally plausible of course. But at the same time, a degree of skepticism is warranted because Microsoft has a huge incentive to fearmonger about AI so they can lock it down and capture the market. And what better way is there than to use the usual bogeymen to do so.

[1] https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcor...

The only thing noteworthy about this is how small-scale this is, and that the perpetrators don't even bother/have the means to set up their own infrastructure.

"...We ran these images through our DALL·E 3 classifier, which identified them as not being generated by our services..."

I'd be shocked if Twitter weren't stripping the metadata they are checking.

It is apparently C2PA per [1]

[1] https://openai.com/index/understanding-the-source-of-what-we...

Main takeaway is that OpenAI is reading all of your prompts :)

speaking of influence operations, the strawberry dude and 'lily' just did one of those twitter voice group chat things where everyone tried to guess if lily was an ai or not. there just happened to be a worldcoin rep in the room...

Why wouldn't Iran just use Llama or something?

Wow. What a high value target. [0]

I mean. I get that low rent actors will use low rent services to try to generate political garbage. Is there any evidence that this is actually having a measurable or meaningful impact?

Who exactly is fooled by these sites? And is it the sites that are the problem or the relative lack of sophistication in American education when it comes to political corruption?

[0]: https://niothinker.com/

Yeah. Knowing my tools are judging my political positions and could self destruct if the authors disagree just makes me love using my computer.

This is why I only use local models these days.

EDIT: Out of posts for today but I've been pretty happy with Gemma2. The context is short but the performance is very good and it's easy to disable refusal.

LLM powered disinformation machines is terrifying. The barrier to entry and sustained cost is so low.

But the social impact is significant. I'm reminded of a fake story about kid kidnapping in India, that caused a mob to burn the two people that were targeted alive... They were completely innocent, the mob attacked them based on fake news. Now, that can happen enmasse.

It's hard to take an article like this at face value when they provide zero evidence for any of their claims.

This is coming soon after Trump decided to accuse Iran of being behind his assassination attempt (done by a white 20 year old) and Israel literally assassinated Hamas's chief negotiator while he was visiting Iran.

It seems like the powers that be are desperate for a war with Iran and will continue beating the drum to build consent.

Reminds me of the build up to the 2003 Iraq invasion (you know, because "they have WMDs")

I don't think people should jump to any "this is the level Iran is at?" conclusions.

Many nations employ "patriotic citizens" in an informal and semi-formal fashion along with trained propaganda and "infowar" experts. I know of China and Israel doing this but I'd assume it's everywhere.

influencing the politics of the United States? Is there an alternative choice I have not heard of? I thought it was samesame in blue or red as it always has been.

Nothing like 900 million articles about secret Iranian plots from government entangled dystopian megacorps and the pundits who love them, soon after the US starts moving troops into the Middle East to defend the progress of an ongoing genocide.

I suppose this is a great opportunity for the people whose entire income comes from the fact that the US overpays insiders to supply its own military; if Trump wins, he gets to play tough and have the papers start his term praising him to the skies for starting a war with Iran, he'll also be able to blame everything on the last administration and work closely with the people who replace Netanyahu. If Harris wins, she's made no promises and has no beliefs, and will be aided by the press in blaming Gaza on Iran.

OpenAI itself is surely running larger covert influence operations in order to affect US legislation and elections.

> Similar to the covert influence operations we reported in May, this operation does not appear to have achieved meaningful audience engagement. The majority of social media posts that we identified received few or no likes, shares, or comments. We similarly did not find indications of the web articles being shared across social media.

Sounds like Russian Facebook ads.

What they don't say on their post but that we can guess and is from interest is that they probably had to spy their user messages to determinate that they used the account for generate content for the influence operation.

For sure the purpose is noble, but it is good to remind everyone that everything you type, submit or generate there is not private but could be randomly snooped by strangers!

Given that any self hosted open source model would have worked just as well. I can’t see this good faith post as anything more than forwarding open ais long campaign for regulatory capture.