• Diff for "Service Outage Report - 20190306"
Last updated at 2019-03-06 15:25:25
Differences between revisions 1 and 2
Revision 1 as of 2019-03-06 15:13:20
Size: 805
Editor: loxch
Comment:
Revision 2 as of 2019-03-06 15:22:40
Size: 830
Editor: loxch
Comment:
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
This outage is caused by DoS attack to mudfish nodes. At the previous outage, its target was to Mudfish Web Servers. However at this time, its target was Mudfish Nodes. This issue was started at the over loads of DB servers but we found that it's caused by internal log collector daemons because due to DoS attack too many logs were generated. This outage is caused by DoS attack to mudfish nodes. At the previous outage, its target was to Mudfish Web Servers. However at this time, its target was Mudfish Nodes. This issue was started at DB servers overloaded but we found that it's caused by internal log collector daemons because due to DoS attack too many logs were generated and inserted into the DB server.

Service Outage Report - 20190306

Summary

This outage is caused by DoS attack to mudfish nodes. At the previous outage, its target was to Mudfish Web Servers. However at this time, its target was Mudfish Nodes. This issue was started at DB servers overloaded but we found that it's caused by internal log collector daemons because due to DoS attack too many logs were generated and inserted into the DB server.

Outage Time

  • 2019-3-6 8:30 PM ~ 2019-3-6 11 PM (KST)
    • For 2 ~ 3 hours, accessing the services of Mudfish was hard and very laggy including Web Server / Authentication services.

Current Status

  • As temporary workarounds, I'd disabled to trace the user / nodes events which used internally to reduce the overloads of DB.
  • All services of Mudfish should work fine now.

Service Outage Report - 20190306 (last edited 2019-03-06 15:25:25 by loxch)