An international research team has developed a deep learning system that can detect natural disasters using images posted on social networks. The researchers used computer vision tools that were able to analyze, filter and detect real disasters after learning from 1.7 million photos. The article was published in the journal IEEE Processes on Model Analysis and Machine Intelligence.

One of the researchers in the MIT-led project was Agatha Lapedriza, head of the AIWELL research group for human well-being, affiliated with the Center for eHealth, and member of the Faculty of Computer Science, Multimedia and Telecommunications. Auberta de Catalunya University (UOC).

As global warming progresses, natural disasters such as floods, hurricanes and forest fires become more frequent and destructive. With tools still lacking to predict where and when such incidents will occur, it is vital that emergency services and international cooperation agencies can respond quickly and effectively to save lives. “Fortunately, technology can play an important role in these situations. Social media posts can be used as a low-latency data source to understand the progress and impact of a disaster,” Lapedriza explained.

Previous research has focused on analyzing text messages, but this study went even further. While working at MIT’s Computer Science and Artificial Intelligence Lab, Lapedriza contributed to the development of event taxonomy and the database used to train deep learning models, and conducted experiments to validate the technology.

The researchers created a list of 43 categories of events, including natural disasters (avalanches, sandstorms, earthquakes, volcanic eruptions, droughts, etc.) and accidents (plane crashes, construction accidents, etc.) that involve some elements of human response. This list, along with 49 place categories, allowed the researchers to tag the images used to train the system.

The authors created a database called Incidents1M with 1,787,154 images that they then used to train an event detection model. Of these images, 977,088 had at least one positive label that associated them with one of the event classifications, while 810,066 had negative class labels. Meanwhile, 764,124 images for place categories had class-positive tags and 1,023,030 images were found to be class-negative.

Avoiding false positives

These negative signs meant that the system could be trained to eliminate false positives; For example, a photograph of a fireplace, even if it has some visual resemblance, does not necessarily mean that the house is on fire. After building the database, the team trained the model to detect events “based on a multitasking learning paradigm and using a convolutional neural network (CNN).

After the deep learning model was trained to detect events in images, the team ran a series of experiments to test it, this time using multiple images downloaded from social networks, including Flickr and Twitter. “Our model was able to use these images to detect events, and we checked whether they corresponded to specific, recorded events, such as the 2015 earthquakes in Nepal and Chile,” Lapedriza said.

Using real-world data, the authors demonstrate the potential of a deep learning-based tool to extract information about natural disasters and human events from social networks. “This will help aid organizations better understand what happens during disasters and improve the way humanitarian aid is managed when needed.”

After this success, the next challenge may be to use the same flood, fire or other event footage, for example, to automatically determine the severity of events or even monitor them more effectively over time. The authors also suggested that the scientific community could do more research by combining image analysis with accompanying text to provide more accurate classification. Source