Can ChatGPT do data science?
A study led by Bhavya Chopra at Microsoft, with contributions from Ananya Singha and Sumit Gulwani, explored ChatGPT's challenges in data science tasks. Strategies included prompting techniques and leveraging domain expertise for better interactions.
Read original article
ChatGPT's ability to assist in data science tasks was explored in a study led by Bhavya Chopra at Microsoft, with contributions from collaborators like Ananya Singha and Sumit Gulwani. The study highlighted challenges faced by data scientists when using ChatGPT, such as difficulties in providing context, false assumptions made by the AI, and misaligned expectations in responses. Data scientists struggled with adapting ChatGPT's responses to their tasks, dealing with repeated code generation, and validating the generated code. Strategies to overcome these challenges included various prompting techniques, leveraging domain expertise, and considering alternative resources. Recommendations for designing AI-powered data science tools included providing preemptive context selection interfaces, inquisitive feedback loops, and transparent mechanisms for sharing context and domain expertise. The study emphasized the need for improved tools to facilitate efficient interactions between data scientists and AI assistants in the data science domain.
Related
ChatGPT is biased against resumes with credentials that imply a disability
Researchers at the University of Washington found bias in ChatGPT, an AI tool for resume ranking, against disability-related credentials. Customizing the tool reduced bias, emphasizing the importance of addressing biases in AI systems for fair outcomes.
The Death of the Junior Developer – Steve Yegge
The blog discusses AI models like ChatGPT impacting junior developers in law, writing, editing, and programming. Senior professionals benefit from AI assistants like GPT-4o, Gemini, and Claude 3 Opus, enhancing efficiency and productivity in Chat Oriented Programming (CHOP).
ChatGPT just (accidentally) shared all of its secret rules
ChatGPT's internal guidelines were accidentally exposed on Reddit, revealing operational boundaries and AI limitations. Discussions ensued on AI vulnerabilities, personality variations, and security measures, prompting OpenAI to address the issue.
How Good Is ChatGPT at Coding, Really?
A study in IEEE evaluated ChatGPT's coding performance, showing success rates from 0.66% to 89%. ChatGPT excelled in older tasks but struggled with newer challenges, highlighting strengths and vulnerabilities.
GenAI does not Think nor Understand
GenAI excels in language processing but struggles with logic-based tasks. An example reveals inconsistencies, prompting caution in relying on it. PartyRock is recommended for testing language models effectively.
4 commentsI finally found this tucked away towards the end of the paper:
"Browser-based version of ChatGPT version was updated as we conducted our task-based studies. P1-P9 used the March 14, 2023 version of ChatGPT, and P10-P14 used the March 23, 2023 version."
GPT-4's release date was March 14th 2023 - but only for paid users. Was this study using GPT-4 or GPT-3.5? That's still not clear to me.
Even more importantly: was it using Code Interpreter / data analysis mode? https://help.openai.com/en/articles/8437071-data-analysis-wi...
I'm almost certain it was not - none of the screenshots showed evidence of that mode, and that feature was made available (as part of the plugins beta) to beta testers on March 23rd - it's weird that the dates mentioned in the paper happen to coincide exactly with the GPT-4 and the Code Interpreter preview dates!
Without including data analysis mode I don't think this paper addresses the question "can ChatGPT do data science?". That's what that 15 month old feature is for, and any study that doesn't include it is missing the most important factor in answering that question.
Related
ChatGPT is biased against resumes with credentials that imply a disability
Researchers at the University of Washington found bias in ChatGPT, an AI tool for resume ranking, against disability-related credentials. Customizing the tool reduced bias, emphasizing the importance of addressing biases in AI systems for fair outcomes.
The Death of the Junior Developer – Steve Yegge
The blog discusses AI models like ChatGPT impacting junior developers in law, writing, editing, and programming. Senior professionals benefit from AI assistants like GPT-4o, Gemini, and Claude 3 Opus, enhancing efficiency and productivity in Chat Oriented Programming (CHOP).
ChatGPT just (accidentally) shared all of its secret rules
ChatGPT's internal guidelines were accidentally exposed on Reddit, revealing operational boundaries and AI limitations. Discussions ensued on AI vulnerabilities, personality variations, and security measures, prompting OpenAI to address the issue.
How Good Is ChatGPT at Coding, Really?
A study in IEEE evaluated ChatGPT's coding performance, showing success rates from 0.66% to 89%. ChatGPT excelled in older tasks but struggled with newer challenges, highlighting strengths and vulnerabilities.
GenAI does not Think nor Understand
GenAI excels in language processing but struggles with logic-based tasks. An example reveals inconsistencies, prompting caution in relying on it. PartyRock is recommended for testing language models effectively.