Delving into ChatGPT usage in academic writing through excess vocabulary

> We show that the appearance of LLM-based writing assistants has had an unprecedented impact in the scientific literature, surpassing the effect of major world events such as the Covid pandemic.

I'm not quite sure this follows. At the very least, I think they should also consider the possibility of social contagion: if some of your colleagues start using a new word in work-related writing, you usually pick that up. The spread of "delve" was certainly bootstrapped by ChatGPT, but I'm not sure that the use of LLMs is the only possible explanation for its growing popularity.

Even in the pre-ChatGPT days, it was common for a new term to come out of nowhere and then spread like a wildfire in formal writing. "Utilize" for "use", etc.

Their list:

    delves
    crucial
    potential
    these
    significant
    important

They're not the first to make this observation. Others have picked up that LLM's like the word "delves".

LLMs are trained on texts which contain much marketing material. So they tend to use some marketing words when generating pseudo-academic content. No surprise there. I'm surprised it's not worse.

What happens if you use a prompt containing "Write in a style that maximizes marketing impact"?"

('You can't always use "Free", but you can always use "New"' - from a book on copywriting.)

One confounding factor here is the proliferation of autocorrect and grammar “advisors” in popular apps like gmail etc. One algorithm tweak could change a lot of writing at that scale.

While the word frequency stats are damning, there doesn’t seem to be any evidence presented that directly ties the changes to LLMs specifically.

> We study vocabulary changes in 14 million PubMed abstracts from 2010-2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words.

"Delving".. Sounds like the authors might have used an LLM while writing this paper as well.

It's also possible that humans are starting to sound more like LLMs too, due to reading (wittingly or otherwise) more LLM output.

This just makes me think how now, more than ever, it really pays to develop your own distinctive writing voice. It’s not just a way to make yourself stand out, but given how much the way you write can influence your thought process, I worry that all the students ChatGPTing their way through language arts classes will be ultimately more susceptible to groupthink and online influence campaigns.

I don't see why many people complaining on this issue. Not everyone mastered English unfortunately. I am especially very weak at writing a paper, and to be honest, find it taxing. I love research but after having results, turning it into a paper is not fun. I edit almost everything important I write like emails and papers with LLMs because even though the content is nice my writing feels very bland and lacks lots of transition. I believe many people do this and actually, this helps you learn over time. However, what you learn is to write like LLMs since basically we are supervised by the LLM.

It’s possible this is caused by the editors rather than the authors.

An old partner edited papers for a large publisher - largely written by non-native speakers and already heavily machine translated - and would almost always use ChatGPT for the first pass when extensive changes were needed.

She was paid by the word and also had a pretty intense daily minimum quota so it was practically required to get enough done to earn a liveable wage and avoid being replaced by another remote “contractor”.

Great article. One of the papers it cites is https://arxiv.org/abs/2403.07183, which is also great and looks at the issue of LLM usage to write peer reviews.

It’s an issue I’ve noticed personally, as I’m seeing an increasing number of reviews that lack substance and are almost entirely made of filler content. Here’s an excerpt from a particularly egregious recent example I ran into, which had this to say on the subject of meaningful comparison to recent work:

> Additionally, while the bibliography appears to be comprehensive, there could be some minor improvements, such as including more recent or relevant references if applicable.

The whole review was written like this, with no specific suggestions for improvement, just vague “if applicable” filler. Infuriating.

Can even See this in Google trends:

https://trends.google.com/trends/explore?date=today%205-y&q=...

Nothing wrong with it. Citing each other introduces more bias than chatGPT anyway. do you expect me to write the same introduction to the same subject for the 17th time? Large parts of any paper are redundant

Even pre-LLMs, I’d basically rewrite papers using simpler language when I was a co-author. The baseline was never very good, I’m afraid. For some reason researchers love using ten-dollar words even though their overuse makes the paper read pretentious and obfuscates the meaning. Folks, please read the following essay by Orwell, and then read it again and again until it sinks: https://www.orwellfoundation.com/the-orwell-foundation/orwel...

This apparent language shift is mainly due to Lon’s being trained on all language and not just modern American writing tied to post Hemingway ideals of brevity. Much of the language considered odd is standard British English, like the singular they. Also the use of wider vocabulary isn’t considered bad writing outside the Us where specificity is valued over simplicity.

I can’t trust any paper with y-axis labels that are all over the place paired next to each other. Figure 1 is a hot mess.

It won't take long before it's possible to mask LLM use by making the text sound more human like, without the excess vocab.

The leap from how the LLMs write now to how a professional sounding scientist might write their paper is probably not that big.

Its possible now to train them on your own writing style.

Haha great wink wink title.

I wish we would distinguish editing, from writing.

good. embarrassing.

human and machine both should aim for brevity and clarity, and feel shame otherwise.

then we can read more and better in our lives.

We live in an era where academics use gpt to spew out papers, and have an audience (not all) who use gpt to summarize and extract the meaningful content from it.

We truly live in the informations age.

On a side note, do people still use ChatGPT to fill out their papers? I found Claude to be way better at spitting out more content. In my experience, ChatGPT has been in the middle while Gemini is the worst, it even cuts-off in the middle of the sentence.

Ha ha at “delving” common gpt language

Is the Oxford comma also on purpose?

Gemini: "subtle nuances"

LLMs used in writing of 1/3 of scientific papers. On the one hand scary, on the other it seems people find them useful. Maybe this AI thing is not like crypto after all...

This is so disappointing.

"delving"