Issue connecting to Gravity, cPanel, and Sites hosted on Gravity
We are aware of 504 connecting to gravity
We're now marking this incident as resolved, as we've seen services restored and stabilizing now.
At this time, all core services have been restored and sites are now operating back on Gravity. We apologize again for the extended panel downtime. If there's anything we can do at all, please reach out.
If you make further changes on your site while it's pointed to the failover server, they may not be reflected once the DNS propagates. If this happens, contact us and we are more than happy to sync the changes for you.
We have begun the process of proactively issuing refunds for this month. Please reach out to request one if you do not see this in your account. A post mortem will be published shortly. Major infrastructure and communication changes will be made.
We've begun the process of reprovisioning our host server to restore cPanel access to all our users.
We are still coordinating with our data center technicians in restoring the rest of the data at this time. While we'd like to provide an ETA, we can't promise anything at this point. This is a level of cascading failure that we've never seen before, and we're truly sorry for any disruption, inconvenience, and inaccessibility this has caused. If there's anything we can do to help alleviate this while we continue to resolve this incident, please reach out to us.
At this time, we're beginning the process of merging the current data with the backed up data on the failover server. We'll be doing this manually to ensure minimal issues. We'll be doing an in-depth post-mortem and audits for this incident. We realize that the amount of downtime and instability faced today was completely unacceptable. We will be making major changes to both our server architecture and protocols to ensure such an incident does not happen again.
We're working closely with our data center to bring the main host machine back online. A faulty driver update has resulted in an exponential memory leak, rendering the server unusable. At this time, all accounts restored on our failover server. Sites will be back, pending DNS.
If there's major changes/content you've made to your site in the past 1-2 days, we recommend leaving your site as is, as once the host server is back online, we'll be merging the old copies with the current version. If only minor changes were made, feel free to continue as normal.
Once the host node is back, our team can assist you with conflict checking and verifying that there are no conflicts between the two versions. We sincerely apologize for this - it's absolutely not the norm. We're issuing a 200% monthly credit refund to all our clients.
At this time, our control panel is still not accessible. We're working on syncing conflicts with that. Emails sent at this time will not be received. Incoming emails will be deferred, but not lost, as MTA servers will retry delivery for up to 72 hours in case of failed delivery.
Please update your A record for your domain and subdomains (cpanel, webmail, and disk excluded) to 184.108.40.206 temporarily, to restore connectivity. If you have your DNS hosted with us, it will automatically be updated.
We're beginning a temporary restoration of a backup to a failover server while we investigate issues. Site will come online as DNS propagate but maybe 1-2 days outdated. There is no data loss - we're simply doing this to serve sites temporarily to visitors. The up to date copies of sites will be restored shortly. This is only temporary. We advise users not to make changes to your site, as it will cause conflicts once the up-to-date versions of a site is restored.
We're gradually bringing back services online now.