Summary: A hardware failure on one of the database servers caused a critical service outage affecting part of our customers. The faulty hardware prevented the server from being restored to operational status, necessitating a full database migration to a new server. The migration was delayed due to compatibility issues between the older and the newer hardware.
Resolution: The affected databases were migrated to a new server, and the compatibility issues were addressed to restore service.
Next Steps: We have already improved our server hardware rotation to minimize hardware failures. We also identified the possible compatibility issues which enable faster service restoration in case of future incidents.