Goldman on Generative AI: doesn't justify costs or solve complex problems [pdf]

Except for a short window around the release of GPT-4 (especially the inflated claims around beating expert trained humans at legal and math tests, as well as "replacing google"), I think people have more or less right-sized their expectations for large language models and generative AI. Clearly it can do interesting and impressive things but it's not superintelligence, and the folks predicting we're just around the corner have been recognized once again as shysters, hucksters, and charlatans. It doesn't help that state of the art ML researches have gotten so good at over-hyping the actual abilities of their technology.

However, I do think we'll continue to see impressive advances in the areas of media consumption and production, with complex reasoning on hard problems being a likely area of improvement in the near (1 decade) future. While I once never expected to see something like HAL in my lifetime, I feel that many aspects of HAL (voice recognition, ship automation, and chess-playing) have been achieved, if not fully integrated into a single agent. We can expect most applications to be banal- the giants who have the largest data piles will train models that continue to optimize the addictivity of social media, and click-thru rates of ads.

I am also quite impressed at the recall of information by language models for highly factual and well-supported things (computer reviews in particular).

Generative AI appears fantastic aid for many smaller tasks where there's enough training data, and correctness of the answer is subjective (like art), or easily verifiable by a human in the loop (small snippets of code, checking that summary of an article matches the contents of the original). Generally it helps with the tedious parts, but not with the hard parts of my job.

I don't have much belief in fully autonomous generative AI agents performing more complex tasks any time soon. It's a significant productivity boost for some jobs, but not a total replacement for humans who do more than read from a script, or write clickbait articles for media.

Heartbreaking: The Worst People You Know Just Made A Great Point

This interview by Adam Conover, really is a wonderful discussion on the topic https://www.youtube.com/watch?v=T8ByoAt5gCA I was pretty amazed with GPT when it came up, but increasingly find it makes to many mistakes. I full use it as a tool in writing code, but it needs to be treated as Intellisense plus, or something to that affect, not something that will handle complex tasks. GPT and Claude make many mistakes and unless they can solve it from completely making up stuff (which I don't think they can) will not advance much more beyond waht they currently are.

There is a paradox. To build the future requires irrational belief. And to sell that vision.

Perhaps the difference between insanity and visionary, "scam" and genius is simply the outcome.

When someone like Sam Altman declares optimistically that we will get AGI and talks about what kind of society we will need to build... It's kind of hard to tell what mix of those 4 is at work. But certainly it will be perceived differently based upon the outcome not the sincerity of the effort.

[dupe]

Please don't post wayback links unnecessarily. Content still fresh and available.

Discussion here: https://news.ycombinator.com/item?id=40856329

The only question I have is whether Goldman is shorting NVIDIA..

Original: https://www.goldmansachs.com/intelligence/pages/gs-research/...

Is there more than the cover to this ? On mobile I only see one page of the PDF.

Pretty sure I read that Goldman itself is currently creating its own internal models using its proprietary data to help its analysts, IT and investors.

Ironically, AI still sucks at accurately parsing PDFs.

Great report.