Researchers note that GPT-4 seems to be getting dumber and dumber. They come to this conclusion after several tests.

After comparing studies in two different months, researchers from the universities of Berkeley and Stanford have found that GPT-4 seems to be getting dumber and dumber. They published their findings in a recently published report.

comparison tests

Three researchers from the American universities UC Berkeley and Stanford examined the development of the LLM chatbots GPT-3.5 and GPT-4. They gave the AI ​​models four different tasks in March and later in June this year and then compared the results. The tasks consisted of:

• math problems
• Sensitive or “dangerous” questions
• generate code
• visual thinking

Some results were quite surprising. For example, in March, GPT-4 was able to detect prime numbers with a very high level of accuracy, but two months later the accuracy had dropped by no less than 95 percent. For 3.5, these results were better in June. In addition, in June, GPT-4 was significantly less willing to answer difficult and sensitive questions.

Both versions also made more formatting errors during code generation in June. There was only a slight improvement of two percent in visual reasoning for both models.

Results

One of the researchers’ conclusions is that it is striking how much the behavior of a large language model can fluctuate in a relatively short period of time. They therefore point out that constant monitoring of the technology is essential.

The three researchers see the opacity of how and when both AI models get an update as the reason for the moodiness. As a result, they are reluctant to incorporate LLMs into larger workflows due to the lack of consistency.

GPT-4 has been publicly available since earlier this month. Earlier this week we reported on certain visual features that OpenAI has deferred for the chatbot.