UPSC-CSE: Nothing artificial about it!

The internet is flushed with articles proclaiming the news “ChatGPT fails to clear the UPSC preliminary test”. This news has started a conversation about the infallibility of Artificial Intelligence and in the process exposed its frailties. We at Forum IAS wanted to dig deep and wanted to derive insights which would help us connect the dots. For this, we have not limited our research to the current year’s prelims question paper, but we have widened the scope of our research to include CSE Prelims 2021 paper (where we have official answer keys given by UPSC).

“ChatGPT is scary good. We are not far from dangerously strong AI”. This ominous tweet by Elon Musk sent shockwaves across the world. ChatGPT was here and the world as we know was about to change.
For the uninitiated, ChatGPT is an artificial intelligence chatbot developed by Open AI and was launched in November 2022. It claims to perform feats which invariably has created a feeling of unease as everyone is realising how ChatGPT is actually “scary good”.

However, in this part of the world, something interesting has happened. In 323 BC the world conquest of Alexander was halted on Indian soil, in 2023 another juggernaut has been humbled and this time the challenger is not the “perceived” might of a standing army but is an exam which produces future bureaucrats-The hallowed Civil Service Examination conducted by UPSC.

ChatGPT has registered its exploits by cracking the prestigious entrance exam of the Master of Business Administration (MBA) programme at the prestigious University of Pennsylvania’s Wharton School. Besides this ChatGPT has also managed to crack US Medical licensing Exam and the US Bar exam. However just as Alexander in 323 A.D realised his limitations when it entered Indian soil, similarly the limitations of world-conquering ChatGPT have been exposed by Civil Services Exam. According to a report by Analytics India Magazine ChatGPT could only answer 54 out of 100 questions from Question Paper 1, Set A, of the UPSC Prelims 2022. Thus, it could not qualify preliminary stage of UPSC exam!

We wanted to dig a little deeper to analyse how and why ChatGPT “failed” to qualify for UPSC. We tested the report and came out with some insights. The result reinforces the “Artificial” in the AI of this Chatbot.

(The data sheet analysing the responses of ChatGPT vs UPSC Prelims Answer key 2022 and 2021 by ForumIAS is attached with the document)

Here are our top 5 key insights from our results:

#1. Accuracy is inversely proportional to the complexity/nuances of the subject

Fig 1: Subject-wise Accuracy: CSE Prelims 2022

Fig 2: Subject-wise Accuracy: CSE Prelims 2021

You will find a pattern in both of these graphs. ChatGPT was able to answer factual questions which were easily available on the internet. This is the reason it clocked very high accuracy in questions from Economy, Science and Technology and Environment. However, things get harder for the chatbot as soon as the nuances, subjective interpretation and complexity of the subject increase. This is the reason you will find subjects like Polity and History, not fairing very well as far as accuracy is considered.

You will also find for both years ChatGPT performed miserably in questions related to Current Affairs.
That can be attributed to the fact that ChatGPT’s knowledge is limited to September 2021.

#2. ChatGPT often picks the wrong data from the internet and presents it confidently as an answer (without any disclaimer)

According to the Bot, Constitutional Amendment Bill requires the prior recommendation of the
President of India and it quotes Article 368 to justify its answer. It takes confidence to another level.

In the AI verse Azerbaijan shares its border with Afghanistan.

#3. ChatGPT creates new options when it does not find an answer. Talk about confidence!

#4. Some of the replies are sheer hallucinatory in nature

#5 ChatGPT learns very fast and reiterates its answers

Some of the answers which ChatGPT produced the next day of our exercise were different (and usually correct) from the day before. At least 3-4 answers that it had messed up were magically transformed into the correct answers the very next day. So, a small disclaimer that the results of the same exercise when carried out at a future date may present a different picture.

You will also notice that the overall accuracy of ChatGPT in CSE 2021 was much higher across all the subjects compared to accuracy in CSE 2022. This gives us an idea that ChatGPT’s knowledge is time specific. The longer the data is available on the Internet, the better its chances to be answer the question.

Is all the jazz around ChatGPT relevant for UPSC?

In long run, nobody knows but as things stand today, ChatGPT and other similar AI bots lack nuance and depth in understanding issues which are relevant to clear UPSC. Over-reliance on a chatbot inhibits critical thinking which is a skill not only to clear the exam but to flourish in this hypercompetitive age.

Moreover, it is not only the Intelligence Quotient (IQ) which is tested in this examination but Emotional Intelligence (EQ) is also tested at every stage of the examination. Easily dished-out information by AI can help one gain facts and information at the click of a mouse, but a well-rounded personality which
will eventually hold the Steel frame of India is made from the age-old formula of Blood, Sweat and Tears!

When asked whether it can pass the prelims exam for UPSC, ChatGPT replied acknowledging its
limitations:

“As an AI language model, I do possess a vast amount of knowledge and information, including about the UPSC examination and related topics. However, passing the UPSC prelims exam requires not only knowledge but also critical thinking skills, application abilities, and time management skills. Therefore, I cannot give a definite answer on whether I can pass the prelims exam for UPSC or not.

This post has been authored by Abhishek Kulranjan.