RocketNode - System Reboot - Bad CFX Update – Incident details

All systems operational

System Reboot - Bad CFX Update

Resolved
Major outage
Started 18 days agoLasted 2 days
Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    London, Germany, Singapore, and Sydney have had the update take effect. We will continue monitoring all locations closely.

  • Monitoring
    Monitoring

    All Virginia game servers have been rebooted and CPU usage has stabilized. We will begin rebooting game servers in other locations shortly.

  • Update
    Update

    Approximately half of VA game servers have been rebooted. We expect to have the rest fully rebooted within an hour. Thank you for your patience.

  • Identified
    Identified

    Virginia servers are in the process of being rebooted. All servers with artifact version set to 'latest' with auto-update enabled will be rolled back to version '12911' from last week.

  • Monitoring
    Monitoring

    All Dallas game server machines have been rebooted - reverting anyone with artifact version set to 'latest' with auto-update enabled to '12911'

  • Identified
    Identified

    Hi all, our team has been been reviewing system metrics across all our locations and noticed a significant spike in CPU usage in the past week. We've determined that newer CFX builds are taxing the CPUs on our systems much more than previous ones due to a bad update. We are seeing upwards of 90-100% CPU usage compared to 25-30% from before the most recent builds.

    Action being taken:
    We are rolling back all servers which have their artifact version set to 'latest' and auto-update enabled to version '12911' from last week.

    We will begin the rollback process with our Dallas servers shortly, and will continue to deploy this fix across additional locations.

    We appreciate your cooperation and patience as our goal is to provide a smooth gameplay experience for everyone.