Khanatics,

At exactly 12:35 PM (Manila Time) everyone suddenly got disconnected. But because the servers are being monitored by a service (when a server crashes) it automatically got restarted which is why everyone was able to reconnect immediately.

After a few minutes the server crashed once again. This was the the moment I was informed by GM-Bobher and GM-Trinity of the issue.

I immediately started monitoring the server logs and while sifting through it, the server crashed for the 3rd time, and this time I caught the problem:

27|Inspect | panic: runtime error: slice bounds out of range
27|Inspect | routine 819 [running]:
27|Inspect | panic(0x75e1a0, 0xc04200c100)
27|Inspect | /panic:500 +0x1af
27|Inspect | main.(*client).accept(0xc042210500)
27|Inspect | /client:219 +0x1fb8
27|Inspect | created by main.startClient
27|Inspect | /client:87 +0x4e2

This was triggered by an unexpected packet size or a BAD PACKET. Which leads us to believe that this incident was a deliberate attack. Possibly in response (revenge) of the recent mass banning of cheaters.

I have put in place a temporary fix to catch and discard any unexpected packet sizes so it won't affect the rest of the servers. This should stop the server from crashing in reference to the error above. 

The fix is now implemented on the live server. If the server still crashes after this then we will review the fix and implement a better one.

ROLLBACK

There shouldn't be any issue of rollbacks from this incident. The game database server was running at the time the server restarted which means everyone got saved correctly.

Blaze GM Team