At the car manufacturer Toyota, production in Japan had to be shut down for around 36 hours due to IT problems. This time there was no cyber attack, but there was not enough storage space on some servers.

Fourteen factories in Japan were unable to produce Toyota cars for 36 hours. This was due to an IT glitch that shows that overprovisioning is not always a bad thing. The exact course of events is unclear, but apparently the server and database maintenance at the car giant had gone drastically wrong.

During the maintenance job, the databases were reorganized until this process stopped abruptly due to insufficient free disk space. Toyota works with a high-availability cluster, which is identical to the primary cluster. After the failover, the backup servers also crashed for the same reason.

Higher capacity

The servers took care of orders for spare parts for the factories. Toyota uses one just in timeChanged strategy for production, leading to a complete standstill in fourteen factories. It took technicians 36 hours to fix the problem. The solution: move the data to a server with more storage capacity.

It’s not the first time that digital issues have brought Toyota production to a halt. If the cyber incident this time was the result of a mistake and not an attack, it was different in March last year. Then production stopped at every third Toyota plant after a cyber attack on a supplier.