Researchers find that Google’s Gemini model consistently performs almost, but not quite, at the level of the old GPT-3.5. The paid version of ChatGPT with GPT-4 is superior.

Gemini, Google’s new LLM, does not reach the same level of performance as OpenAI’s latest models. Researchers noted this in an article published on Arxiv.org. The research in question was conducted by the renowned Carnegie Mellon University and a startup called BerriAI, which primarily provides access to multiple AI models with prompts. The research in question appears to be thorough and reliable, but has not yet passed peer review.

Thorough testing

The researchers pitted Gemini Pro against GPT-3.5 Turbo in a variety of tests in various domains, including knowledge, reasoning, mathematics and translation. In all cases, Gemini performed slightly worse than the older GPT 3.5 Turbo model. GPT 4 Turbo performed significantly better than the others.

The researchers explain their testing method in an understandable paper. For example, the knowledge of both models was tested using 57 multiple-choice questions, with Gemini’s answers being the least accurate. For general justification, the models were presented with 27 tasks from a previous study. Here, too, the twins didn’t do so well. Especially in a matter where an item is exchanged between different entities (a story where different friends buy different books and then pass them on), Gemini cannot follow this example.

Do Gemini have a knack for math? Neither, according to the researchers. The LLMs had to solve problems of varying levels and once again Gemini was the star of the course.

linguistic proficiency

Gemini’s strength lies in languages. The models were given twenty translation tasks and all in all, Google’s model lost the competition here too, but the result was close. In eight out of twenty cases, Gemini performed better than GPT 3.5 and even GPT 4.

The result of the investigation is clear: Gemini does not reach the level of OpenAI’s latest model. The difference is significant. Google hasn’t caught up yet and OpenAI remains champion. We don’t think the results come as a surprise to Google. When the model was presented, the demo seemed to have been staged. That doesn’t immediately show trust.

GPT-3.5 is available for free via ChatGPT and remains the best open access model currently available. GPT-4 is much better and is currently unparalleled. To get started, you’ll need a paid subscription to ChatGPT.