At approximately 5:15 PST we were notified that our VPS panel and children VPSes on our main node were inaccessible. We looked into this and discovered that the network interface was disconnected at the DC due to human error. While we use redundant gigabit uplinks, the disconnect happened at an ISP service level, therefore both uplinks were rendered inaccessible.
We worked closely with our upstream to bring the network back online and service was fully restored at 8:23 AM PST. During this time both incoming and outgoing connections to the public internet were inaccessible, though all VPSes continued to run. The total period of downtime will be considered 4 hours for compensation purposes.
What happens now?
We're sincerely sorry to our customers for the interruptions caused by this outage. We understand you trust us host and keep your services online and today we failed that. We're providing 8x compensation per our SLA for the downtime. Please contact us to receive the credit or refund.
How will future incidents like this be prevented?
We've worked internally to improve notification and communication with our upstream provider and we're also evaluation further redudancy solutions, including a high availability or backup-failover setup.
We accept fully responsibility for this incident. Please contact us with any questions or concerns.
This is a good time to remind everyone, keep your own backups! Our VPSes are non-managed services, and we are not responsible for maintaining backups of your data, in the (rare) event of data loss.