A new biological sensor can recognize moving objects in a single video frame and successfully predict where they will move. This smart sensor described in the document Nature Communicationdynamic vision will be a valuable tool in many fields, including automated control, industrial process control, robotic control and autonomous driving technology.

Modern motion detection systems require many components and complex algorithms for frame-by-frame analysis, making them inefficient and energy-intensive. Inspired by the human visual system, researchers at Aalto University have developed a new neuromorphic vision technology that combines sensation, memory and processing in a single device that can detect motion and predict trajectories.

At the heart of their technology is a series of photomemistors, which are electrical devices that generate an electric current in response to light. The current does not stop immediately after the light is turned off. Instead, it degrades gradually, meaning that photomemistors can effectively “remember” whether they have been exposed to light recently. As a result, the sensor, consisting of an array of photomemistors, not only records instantaneous information about the scene, as a camera does, but also contains a dynamic memory of previous moments.

“The unique feature of our technology is that it can integrate a series of optical images into a single frame,” explains research scientist Hongwei Tan. “The information of each image is embedded as hidden information in subsequent images. That is, the last frame in the video also contains information about all previous frames. This allows us to detect motion in the video early by analyzing only the last frame. With a simple neural network, the result is a compact and efficient sensor.”

To demonstrate the technology, the researchers used videos that alternately show the letters of a word. The last frame of all the videos looked similar, as all the words ended with the letter “E”. Conventional vision sensors could not tell if the letters “APPLE” or “GRAPE” were followed by the letter “E” on the screen. However, using the hidden information in the last frame, the photomemistor matrix can identify which letters come before it and guess what the word is with almost 100% accuracy.

In another test, the team showed videos of the sensors showing a simulated human moving at three different speeds. The system not only recognized motion by analyzing one frame, but also accurately predicted subsequent frames.

Accurate motion detection and object position estimation are vital for autonomous driving technology and intelligent transportation. Autonomous vehicles need accurate predictions of how cars, bicycles, pedestrians and other objects will move to guide their decisions. By adding a machine learning system to the photomemistor array, the researchers demonstrated that their integrated system could predict future motion based entirely on internal processing of the information frame.

“With our compact onboard memory and computing solution, motion recognition and prediction opens up new possibilities in autonomous robotics and human-machine interaction,” says Professor Sebastian van Deyken. “The in-frame information we capture in our system with photomemristors prevents unnecessary data flows, enabling power-efficient real-time decision making.”