System Reboot - Bad CFX Update
- Resolved · UTCThis incident has been resolved.
- Update · UTC
London, Germany, Singapore, and Sydney have had the update take effect. We will continue monitoring all locations closely.
- Monitoring · UTC
All Virginia game servers have been rebooted and CPU usage has stabilized. We will begin rebooting game servers in other locations shortly.
- Update · UTC
Approximately half of VA game servers have been rebooted. We expect to have the rest fully rebooted within an hour. Thank you for your patience.
- Identified · UTC
Virginia servers are in the process of being rebooted. All servers with artifact version set to 'latest' with auto-update enabled will be rolled back to version '12911' from last week.
- Monitoring · UTC
All Dallas game server machines have been rebooted - reverting anyone with artifact version set to 'latest' with auto-update enabled to '12911'
- Identified · UTC
Hi all, our team has been been reviewing system metrics across all our locations and noticed a significant spike in CPU usage in the past week. We've determined that newer CFX builds are taxing the CPUs on our systems much more than previous ones due to a bad update. We are seeing upwards of 90-100% CPU usage compared to 25-30% from before the most recent builds.
Action being taken:
We are rolling back all servers which have their artifact version set to 'latest' and auto-update enabled to version '12911' from last week.
We will begin the rollback process with our Dallas servers shortly, and will continue to deploy this fix across additional locations.
We appreciate your cooperation and patience as our goal is to provide a smooth gameplay experience for everyone.