With Orca 2, Microsoft is bringing an AI model onto the market that is as good or even better than the larger and more complex competing alternatives.

In general, the larger and more complex the generative AI model, the better its performance. However, training larger models is more expensive and working with them is more inconvenient. That is why the focus of AI research is increasingly on optimization and efficiency. It is in this context that we should consider Microsoft’s launch of Orca 2.

Orca 2 contains two variants of this Great language model (LLM): one with seven billion parameters and one with thirteen billion parameters. The model proves to be as good or better than competing alternatives that are five to 10 times larger, such as Meta’s Llama-2-70B, which has 70 billion parameters.

Think step by step

The intelligence comes from a different approach to the model, where queries are split into different tasks. A large model like GPT-4 can answer one question at once, but Orca 2 breaks questions into multiple blocks. By breaking a query into steps, Orca 2 can still arrive at complex answers.

To achieve the desired behavior, Microsoft trained Orca 2 using a synthetic data set. Through this strategy, Orca 2 learned to properly divide tasks and justify queries. Microsoft shows an example in which different models are confronted with a simple problem.

Note that Microsoft still used ChatGPT 3.5 in the example above. When we pass the issue to ChatGPT 4, the model correctly responds that both believe the ball is where they left it.