Open source models like Llama 2 perform worse in a zero-shot study than closed source models like GPT-4 and Claude 2. The study was conducted in the medical sector.

Generative AI models like Meta’s Llama 2 make their source code available to users, unlike closed-source models like OpenAI’s GPT-4. A study shows that open source generative LLMs do not perform better than closed source LLMs. The research was based on a specific medical field, namely the science of the kidneys. Therefore, these results cannot be generalized based on this research.

One of the most famous open source LLMs is Meta’s Llama 2. This open source model is based on user contributions and the source code is visible to everyone. This is in contrast to models such as GPT-4 and Claude 2, where the source code is not available to the public. Which model type performs better?

Zero-shot tasks

To test which type of model performed better, they were compared using questions within a specific domain, in this study, medical knowledge. This showed that Llama 2 was poor at answering nephrology (science of the kidneys) questions and the quality and accuracy of the answers were substandard. In contrast, the closed-source model GPT-4 performed exceptionally well and even came close to human performance for most subjects. The model achieved a score of 73.3 percent, which is just below the 75 percent that is sufficient for a human to answer questions correctly.

This study involved a test known as the zero-shot tasks. This uses a language model without adjustments or examples of right and wrong answers. This tests the ability of a language model to acquire new skills that were not yet included in the training data. Open-source LLMs therefore appear to have low zero-shot capability with respect to nephrology, which will be further explored in medical education in the future.