With PaLM-E, the robotic assistant comes a little closer: an AI model that interprets speech and converts it into actions performed in the physical world.

No apocalyptic robot revolt without robot slavery, and Google is taking a major step in that direction with PaLM-E. Thanks to this AI model, the smart robotic household help is getting closer. PaLM-E is a complex model combining AI computer vision and language understanding. In this case, control is added via a robotic arm.

From command to action

The model is able to translate a command into concrete actions. When you ask, “Remove the batteries from the drawer,” PaLM-E combines that instruction with what it sees around it to come up with an action plan. A robot then performs this action autonomously.

The robot in question does not have to be trained specifically for the situation. PaLM-E looks at its surroundings through the camera and tries to deduce what needs to be done. So no prior human intervention is required to annotate the environment with cheat sheets, for example.

Google states that PaLM-E can also handle changes. If someone takes the batteries out of the drawer and they land on the kitchen table, the controlled robotic platform can get them there.

argumentation

The model is capable of modest reasoning. As an example, Google cites someone saying, “I spilled my drink, can you get me something to clean up?” PaLM-E translates this into a sequence of actions: find sponge, pick up sponge, bring sponge to user, put sponge down. The model would also be about positive transmission learn from previous actions.

Google now wants to further develop the model and is looking at applications within automation in the home, but also in industry.