Apple has quietly introduced a new open source Large Language Model (LLM) that uses images in search.

In October, the first signals from Apple’s open source LLM called Ferret were visible on GitHub. This caught the attention of many AI researchers and got the ball rolling. Ferret is an LLM that uses images for search. This isn’t the first new move in AI for Apple, but the company remains under the radar when it comes to various AI developments.

Images as search

Apple’s LLM Ferret is a multimodal model that reportedly uses parts of images as queries. The model can examine specific areas of an image, determine the elements within them that are relevant to forming an answer, and then draw a frame around them.

When a user sends an image to the model and asks what type of animal it is, the LLM can identify that animal based on the image. Additional questions can also be answered based on elements that the model recognizes in the image.

Open source model

The introduction of the open source model could have a positive impact on Apple. Competing with leading AI providers such as Microsoft and Google is not easy for Apple because the company does not have the necessary computing resources like its competitors. In order not to lag behind, Apple relies on open source models.

This isn’t Apple’s first, low-profile attempt to embrace AI. Recently, Apple developed a new framework that allows developers to build AI models on Apple Silicon and developed Apple GPT. Whether this model will accelerate Apple in the AI ​​race remains to be seen.