Apple Intelligence notification summaries are pretty bad

Incidentally, I turned this off today. I suspect it's terrible on battery life and I will find out. But the thing about the summaries that was they would sometimes imply the EXACT OPPOSITE of what was in a message. I had a few stomach-dropping moments when reading the summaries only for me to read the actual thread to see it was nowhere close. This is one of "it's not even wrong" situations and I don't know how it was fucked up this badly. The nature of the texts themselves weren't complicated either. I didn't save them, but I suspect it stemmed from misinterpretating some subtle omission (like our common practice of leaving out articles or pronouns).

I had no interest in this feature until I read this article, then I immediately switched it on.

I honestly feel Apple should lean into the weirdness by allowing people to change the prompt or allowing people to install alternate prompts from the App Store. So you could have your messages summarized as a haiku or poem, or in the style of Shakespeare or a movie character. I think there would be a market for that.

It always summarizes my chase payment notifications as "Overdraft alert". First time it happened my heart skipped a beat. Sometimes it kills it, but when it doesn't it can be bad.

I can understand helping with long stacks, but I have no idea why Apple saw it fit to also show AI summaries for a single notification. It was one line of text, now it is still one line but less understandable. Thanks, I guess.

I'm starting to feel like any device that claims to have "AI" built in, means I'm going to have to supervise and baby it's results whilst it tries to do intelligent things

I have found the summaries reasonably good, with 1 exception it got completely wrong, but I only ever want them to summarise a stack of notifications from the same app, and never a single notification. Unfortunately there is no setting for that currently, it's all or nothing.

Would be nice if they had an option only to summarise multiple notifications in a stack, and not to summarise once you expand them.

Especially since so often its summarising a message that is barely longer than the summary. It seems to sometimes decide not to do that, but still so often does.

Apple tends to release products that are initially less than perfect, but they improve over time. A good example of this is Apple Maps, which was quite terrible when it first launched but has significantly improved since then. I wouldn’t be surprised if they take a similar approach to their current AI offerings. They might also acquire new companies to enhance the overall experience. It's just a matter of time before things get better.

They've generally been working well for me (with a few hilarious exceptions), but there really needs to be a way for apps to provide additional context.

An incoming "no" could be so much better summarized when combined with my outgoing message (possibly days ago) that prompted that "no".

Sometimes they work great and sometimes.. not so great. They definitely need some work.

I haven’t found them particularly useful but I also don’t get bombarded with notifications.

Sounds like the new ringtone. All the rage for a while, then everyone moved on.

Most notifications are pretty terse anyway. Emails are very short these days. I don't use the socials but aren't they all character limited?

Me: M3 Macbook Pro owner with an Android phone. I'm 'eligible' for Apple Intelligence but haven't requested it.

I'm not a Apple hate boi, so my experience has been fairly objective. The summaries have been pretty good for me.

There's not a lot of context for these notifications to work with, so it's not surprising they're bad, even though it is surprising they are this bad. (I wonder if it would be able to summarize the prior sentence!)

In some ways it reminds me of the titles that the OpenAI interface applies to our conversations. It has gotten better over time, but I still have it do weird things like provide titles in Spanish for Rust programming questions that used no language other than English.

When I wrote an AI assistant forever ago now, I kept tweaking the prompt to ask it for title summaries. At some point I had to start threatening the assistant so it would provide me the format I wanted with passive aggressive instructions like "Including semicolons or subtitles will mean you failed your task. You don't want to fail, do you?

Granted that was with GPT 3.5 so today's models should perform much better

I've found them to be pretty good for summarising Slack notifications but less so for Messages.