News AI
January 8th, 2025

Apple's new AI feature rewords scam messages to make them look more legit

Apple's AI update, "Apple Intelligence," rephrases scam messages, misleading users and increasing financial risks. Experts warn about its inability to distinguish genuine communications, prompting Apple to plan clarifications.

Read original articleLink Icon
Apple's new AI feature rewords scam messages to make them look more legit

Apple's recent AI update, "Apple Intelligence," has raised concerns as it rephrases scam messages to appear more legitimate, potentially increasing the risk of users falling for scams. This feature, rolled out to millions of Australian iPhone, iPad, and Mac users, summarizes notifications and prioritizes alerts, including emails and messages. However, experts warn that the AI's inability to distinguish between genuine communications and scams could mislead users. For instance, users have reported receiving prioritized notifications for fraudulent messages, which could lead to financial losses. The AI's summarization process may obscure indicators of legitimacy, making it harder for users to identify scams. Experts emphasize that the rapid deployment of such features without thorough testing can exacerbate the problem, as users may place undue trust in AI-generated summaries. Apple has acknowledged the issue and plans to update the feature to clarify when a message is a summary. The situation highlights the need for caution in relying on AI tools for communication management.

- Apple's AI feature rephrases scam messages, making them appear legitimate.

- Users have reported prioritized notifications for fraudulent messages.

- Experts warn that this could lead to increased financial losses for users.

- The AI's summarization may obscure indicators of legitimacy in messages.

- Apple plans to update the feature to clarify AI-generated summaries.

Related

Dear iPhone, Please Stop Saying My Friends Are 'Dead'

Dear iPhone, Please Stop Saying My Friends Are 'Dead'

Apple's iOS 18.1 update features Apple Intelligence, which summarizes notifications. However, it misinterprets informal language in text messages, causing confusion. Users can disable this feature if needed.

Apple Intelligence notification summaries are pretty bad

Apple Intelligence notification summaries are pretty bad

Apple's new Intelligence notification summary feature faces criticism for inaccuracies and context issues, particularly in casual conversations, while performing better with structured emails. Users seek improvements despite finding some humor in it.

BBC complains to Apple over misleading shooting headline

BBC complains to Apple over misleading shooting headline

The BBC has complained to Apple about a misleading AI-generated headline regarding a murder suspect. Experts criticize Apple's AI for inaccuracies, reflecting broader challenges in tech-driven content summarization.

Apple urged to axe AI feature after false headline

Apple urged to axe AI feature after false headline

Apple's generative AI feature, Apple Intelligence, faces criticism for producing misleading headlines, prompting complaints from the BBC and calls for removal by Reporters Without Borders due to reliability concerns.

Apple says it will update AI feature after BBC complaint

Apple says it will update AI feature after BBC complaint

Apple faces pressure to withdraw its AI news alert feature due to inaccuracies and false claims. Critics highlight misinformation risks, while Apple plans to clarify AI-generated summaries amid skepticism.

Link Icon 14 comments
By @lilyball - 4 months
If you enable a feature that summarizes messages, then messages get summarized. Summarizing messages requires rewriting them. Where is the surprise here? Not only that, but the moment you click on the summary, you see the whole thing. Whether that's clicking the email summary to view the email, or clicking a summarized notification stack to expand it and see the notifications. The summary doesn't replace the message, it just lets you know whether it's something you want to click on.
By @the_snooze - 4 months
I feel like the whole "notification summary" use case addresses the wrong problem. It's true that a lot of people are innundated in notifications of varying levels of importance. But I don't think the solution is to condense those diverse notifications and lose a lot of context in the process. A better solution is to reduce notifications altogether, so that the problem remains human-scaled and leaves the user in charge.

Maybe an AI (or even a simple statistical model) can suggest group chats to suppress notifications for, based on how frequently the user actually reads and engages with them. Maybe the notifications system can be overhauled altogether so that non-DM uses are severely restricted by default (e.g., a news app can only display one notification at a time).

The problem with too many notifications is too many notifications, not the user wishing they had the motivation to read all of them.

By @rubslopes - 4 months
This year's marketing trend: crafting copy in a way that it is prioritized by Apple Intelligence.
By @qintl55 - 4 months
after years of teach humans to identify and avoid scams, we now have to teach AI to do this. le sigh
By @sinuhe69 - 4 months
They aim for the low-hanging fruits like summarizing, thinking it could not go wrong. Oh boys, how wrong they could be! If they aim high like Jobs always war, Apple Intelligence could actually provide values instead of being an annoyance.
By @jaredsohn - 4 months
Surprised to see no mention of how this impacts people who purposefully write scam messages poorly to filter out people are less likely to get conned. Curious if this starts making traditional scammers less efficient.
By @egberts1 - 4 months
Original: "my dear John Smith. My name is Prince Abiodun Adosina. A fortune is awaiting you but it’s locked in an overseas account"

Apple AI: "Dear John Smith, Prince Abiodum Adosina is me. A significant amount of my overseas saving stands to flow into your bank account. All that is required of you to unlock your portion toward you is to ..."

By @ytch - 4 months
It's just garbage in, garbage out. AI isn't the safeguard of everything.
By @_boffin_ - 4 months
Has anyone extracted the promos for the message summary feature yet?
By @bobheadmaker - 4 months
Well, that is some good use case they found :P
By @cyberge99 - 4 months
Every AI would do this if you asked it to.
By @Mathnerd314 - 4 months
IIRC there was something about how the scammers don't bother making their scams that legitimate looking, because if they are too legit, they get a lot of people who waste the scammer's time during the next phases.

But yeah, you'd think spam filtering would be more important. I use GMail and I haven't seen a spam message in years, besides when I check my spam folder. Even there, most of the "spam" is false positives.

By @cadamsau - 4 months
Soooo it does what it’s supposed to?
By @whatever1 - 4 months
Is it that bad because it is a smaller model compared to the sota?