Service Outage Report - 20181128

Summary

This outage is caused by the wrong input validation for Custom Item which leads the crash of internal RTT daemons. Due to these crashs, the load average of the mudfish master server was increased abnormally then it made this service outage.

Outage Time

• 2018-11-28 10 AM ~ 2018-11-28 2 PM (KST)
  • For 3 ~ 4 hours, the login / RTT calculation process doesn't work well.

Current Status

• All services of Mudfish should work fine.
• Enhanced the input validation for Custom Item.
• Enhanced the checking of Input Validation for server daemons too and those are currently deployed into the production server.