RocketNode - Notice history

All systems operational

Operational

Keymaster - Operational

Platform Server (FXServer) - Operational

Client authentication (CnL) - Operational

Notice history

View current status

Nov 2023

RN27, Dallas TX Outage
Resolved92 hours 32 minutes
  • ResolvedUTC

    All clients have been transferred to our new RN27 node

  • UpdateUTC

    We are in the process of transferring all clients on RN27 to our new machine in order to prevent further downtime due to hardware issues.

  • IdentifiedUTC

    We have identified a hardware issue with RN27 which is causing the entire node to lock up. We are working with our provider to get a new system provisioned and all game servers moved over.

RN27, Dallas TX Outage
ResolvedPartial outage8 hours 27 minutes
  • ResolvedUTC

    This incident has been resolved.

  • InvestigatingUTC

    We have been made aware via our monitoring stack that RN27 Texas is down. We are investigating the cause and escalating this issue upstream to get the server checked to see why it's down and resolve the issue as soon as possible! We'll keep you updated.

London RN46 & RN47 Outage
ResolvedPartial outage12 hours 13 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result.

  • UpdateUTC

    The machines were found to be faulty and have been replaced, we are working on getting the machines back up ASAP.

  • UpdateUTC

    We are continuing to work on a fix for this incident with our upstream provider.

  • IdentifiedUTC

    We are continuing to work on a fix for this incident.

  • InvestigatingUTC

    We are currently investigating this incident.

Dallas VPS Outage
ResolvedPartial outage25 minutes
  • ResolvedUTC

    This incident has been resolved.

  • IdentifiedUTC

    We have identified the issue and are implementing a fix.

  • InvestigatingUTC

    We are currently investigating this incident.

NOV12
Internet failover in VA
CompletedMaintenance1 hour 7 minutes
  • CompletedNovember 12, 2023 at 11:07 PMUTC

    The internet failover has successfully completed. We are now running on our primary internet connection.

  • In progressNovember 12, 2023 at 10:00 PMUTC

    Maintenance is now in progress

  • PlannedNovember 12, 2023 at 10:00 PMUTC

    We are planning to rollback to our primary internet connection later this afternoon. While we don't expect any downtime, you may experience a tiny bit of packet loss as the connection rolls over.

Oct 2023

OCT3
RN27, Dallas TX - Scheduled Maintenance
CompletedMaintenance1 hour 38 minutes
  • CompletedOctober 03, 2023 at 4:38 PMUTC

    The work has been completed and servers are booting again now with the machine being operational. We will continue to monitor for any issues that could arrise.

  • PlannedOctober 03, 2023 at 3:00 PMUTC

    We are expecting to take RN27 down for roughly 2 hours to replace the CPU in an attempt to resolve the issue with recent crashes on this node. We will update you if the expected timeframe changes and of any information we believe is relevant.

  • In progressOctober 03, 2023 at 2:50 PMUTC

    We are now shutting down RN27 in preparation for the work to be completed. We will provide more updates as they come.

FiveM is down - Keymaster Offline
ResolvedMajor outage2 hours 49 minutes
  • ResolvedUTC

    CFX has said it's resolved and it does appear that is the case. Marking as resolved.

  • MonitoringUTC

    A fix has been implemented (as reported by cfx.re) and it appears it's online. We will consider this resolved when Cfx.re concludes the incident.

  • IdentifiedUTC

    We will monitor their status page and update ours once it's back up.
    You can also check here for the latest results: https://status.cfx.re/

    During this time, servers will not be able to authenticate your server key and your server won't start.

Sep 2023

RN27, Dallas TX - Full Outage
ResolvedPartial outage6 hours 20 minutes
  • ResolvedUTC

    The node appears stable so we're marking this as resolved for now.

  • MonitoringUTC

    We've been able to get the system up due to a hardware reboot but we are still investigating the logs to see what caused the machine to go down. RN27 will resume operation unless specified otherwise. We will monitor the system for the time being.

  • UpdateUTC

    We've been informed techs are being sent to inspect the server and bring it back online. We will keep you informed.

  • InvestigatingUTC

    We have been made aware via our monitoring stack that RN27 Texas is down. We are investigating the cause and escalating this issue upstream to get the server checked to see why it's down and resolve the issue as soon as possible! We'll keep you updated.

Dallas TX RN27 Full Outage
ResolvedPartial outage5 hours 44 minutes
  • ResolvedUTC

    Seems like everything is working as intended and the overall load of RN27 is within acceptable levels.

  • MonitoringUTC

    The machine is back online. We identified there was a faulty fan in the system which caused the machine to overheat. This fan has now been replaced and the system is back online. We will continue to monitor system performance and move some servers to our new RN30 Texas Node to lighten the load on RN27 for improved performance.

  • UpdateUTC

    We have been informed the tech is on-site. We will update you once we have a newer update on the situation.

  • UpdateUTC

    We are still waiting for an on-site tech to attend to our server.

  • IdentifiedUTC

    We are aware that RN27 is down and have identified the cause. We are working with the datacenter to get it back online, but due to delays, we are still waiting for it to be brought back online. We will keep you all up to date.

Actively thwarting DDoS attacks against the VA network
ResolvedDegraded performance183 hours 59 minutes
  • ResolvedUTC

    This incident has been resolved.

  • UpdateUTC

    We are continuing to monitor the attacks against our VA network.

    Our engineering teams have recently pushed out a new feature set to automatically enable our enhanced protection on a per-server basis. We will be backfilling servers tomorrow morning (restarting your server tonight can also adopt the protection if you don't want to wait until tomorrow). The more servers we adopt into our enhanced protection, the less room there is for an attack to hurt. We will ensure everything is locked down ironclad tomorrow.

    To also provide some additional clarity, enhanced protection is not a new subscription plan or something like that, but rather our game specific filtering which will allow our network to thwart attacks in milliseconds rather than seconds (like the 60 second downtime sprints we've been seeing).

    We will keep everyone updated with any further developments. Thank you!

  • MonitoringUTC

    For the past few days or so, our network engineering team has been actively handling DDoS attacks against the VA network.

    We believe the attacks have concluded and have fully identified the perpetrator. We will continue to monitor the network and be available to provide any manual intervention necessary. In the meantime, our team is still working around the clock to improve our automated DDoS mitigation strategies and response time.

    We will keep everyone updated with any further developments.

  • IdentifiedUTC

    For the past day or so, our network engineering team has been actively handling DDoS attacks against the VA network. We are aware of these attacks and have a strong understanding of their scope.

    If you are seeing your server or our panel go down for around 60 seconds and then come back, this would be the cause. Please bare with us during this period - all our teams are working around the clock to improve the latency response of the network.

    We will keep everyone updated with any further developments.

SEP9
Upgrading VA Cluster
CompletedMaintenance4 hours 6 minutes
  • CompletedSeptember 09, 2023 at 10:36 PMUTC

    Maintenance has been completed successfully. We will be writing a post-mortem detailing this incident later.

    We will be monitoring our systems for the next 24 hours to ensure reliability - for now, we are looking good. Thank you so much for dealing with us throughout the entire incident.

  • UpdateSeptember 09, 2023 at 10:27 PMUTC

    A fix has been implemented and we are now recovering the services.

  • UpdateSeptember 09, 2023 at 10:06 PMUTC

    The root cause of the downtime has been identified - we are performing the fixes necessary now

  • In progressSeptember 09, 2023 at 6:30 PMUTC

    Maintenance is now in progress

  • PlannedSeptember 09, 2023 at 6:30 PMUTC

    We are planning to upgrade our Kubernetes VA cluster in the next few minutes. We expect zero downtime during this upgrade - we always like to inform our customers before we make any adjustments.

Previous

Sep 2023 to Nov 2023

Next