Exploring ChatGPT's accuracy and confidence in high-resource languages

Pelucchi, Martino (2023) Exploring ChatGPT's accuracy and confidence in high-resource languages. Bachelor's Thesis, Artificial Intelligence.

Preview	Text Thesis_MartinoPelucchi.pdf Download (618kB) | Preview
	Text toestemming.pdf Restricted to Registered users only Download (127kB)

Abstract

As soon as it was published, ChatGPT took the world by storm for its impressive abilities. Due to its release without documentation, scientists immediately began to attempt to formally identify its limits, mainly through its performance in natural language processing (NLP) tasks. This paper aims to join the growing literature regarding ChatGPT’s abilities by focusing on its performance in high-resource languages as well as on its capacity to predict its answers’ accuracy by giving a confidence level. The analysis of high-resource languages is of interest as studies have shown that low-resource languages perform worse than English in NLP tasks, but no study so far has analysed whether high-resource languages perform as well as English. The analysis of ChatGPT’s confidence calibration has not been carried out before either and is critical to learn about ChatGPT’s trustworthiness. In order to study these two aspects, five high-resource languages and two NLP tasks were chosen. ChatGPT was asked to perform both tasks in the five languages and to give a numerical confidence value for each answer. The results show that all the selected high-resource languages perform similarly and that ChatGPT does not have a good confidence calibration, often being overconfident and never giving low confidence values

Item Type:	Thesis (Bachelor's Thesis)
Supervisor name:	Valdenegro Toro, M.A.
Degree programme:	Artificial Intelligence
Thesis type:	Bachelor's Thesis
Language:	English
Date Deposited:	26 Jul 2023 06:46
Last Modified:	26 Jul 2023 06:46
URI:	https://fse.studenttheses.ub.rug.nl/id/eprint/30857

Actions (login required)

View Item