Portal Home > Knowledgebase > Articles Database > Defender/PowerVPS Outage


Defender/PowerVPS Outage




Posted by TomK, 04-24-2005, 09:46 AM
We are working an issue in our Ashburn Equinix location that is causing an outage/high packet loss across the network, we will update this thread when we find out more. We are sorry about this issue, and we have everyone on this urgent matter at this time.

Posted by JenniH, 04-24-2005, 09:49 AM
Yes... and your ticket system doesn't work either! Every ticket we open gives: Fatal error: Call to a member function on a non-object in /var/www/html/powervps.com/support/cerberus-support-center/includes/widgets/track_tickets/box_track_ticket_view.php on line 60 And as you point out... every one of our sites there is down. Please give an ETA as soon as possible, so we can take an objective view on whether to move our most time sensitive sites.

Posted by layer0, 04-24-2005, 09:52 AM
I am pinging all the way through to powervps.com without any packet loss. Seems like it is fixed now. Cheers

Posted by JenniH, 04-24-2005, 09:56 AM
Definitely still down from here!

Posted by TomK, 04-24-2005, 10:03 AM
BGP is coming back up, and right now we show one /24 offline and some packet loss on the others. JenniH, not sure why the web portal isn't working for you, try again or submit via email to support@defenderhosting.com or support@powervps.com. Will update when we get all BGP sessions back online and when everything is working 100%.

Posted by DensityHost, 04-24-2005, 10:23 AM
Wow Dinix & Now this. Thanks to all those people who referred me to Powervps. Still down here as well.

Posted by CeriPlex, 04-24-2005, 10:28 AM
Dinix & PowerVPS are completely different storys. Dinix planned that downtime, PowerVPS did not.. simple as. However, if you find a network thats completely bullet proof with the service you get through PowerVPS, Feel free to PM me now wont ya

Posted by JenniH, 04-24-2005, 10:29 AM
Sorry Tom... but this is STILL down!!! Alerta: 04/24/2005 10:28:42 Atlanta USA Error N/A 30.00 N/A 04/24/2005 10:28:42 Oklahoma City USA Error N/A 30.00 N/A 04/24/2005 10:28:42 Las Vegas USA Error N/A 30.00 N/A 04/24/2005 10:28:42 Hong Kong CHINA Error N/A 30.01 N/A 04/24/2005 10:28:42 Chicago USA Error N/A 30.00 N/A 04/24/2005 10:28:42 Frankfurt GERMANY Error N/A 30.00 N/A 04/24/2005 10:28:42 Orlando USA Error N/A 30.00 N/A 04/24/2005 10:28:42 London UK Error N/A 30.00 N/A Can we have an ETA please?

Posted by wstco, 04-24-2005, 10:31 AM
I assumed that it was just my VPS that was down and out, since I was able to reach the main powervps.com website. The portal site continues to be accessable. I was able to submit a ticket without any problem. I did not experience the issue JenniH reported. I got the powevps broadcast message a few seconds after submitting my trouble ticket. I did get through to my VPS for a few seconds about 1/2 hour ago, but it was brief and did not last. At least I know it's not just my VPS that's dead.

Posted by JenniH, 04-24-2005, 10:35 AM
I think the ticket error shows the advantage of also offering phone support (as Servint do: which I think swung it for some people who were torn between them and PowerVPS). Regardless, the main thing at this point is getting thes up and running. Can you give us an ETA Tom? I'll have to start shifting our time critical sites out very shortly, as we will have to assume the owrst (what else can we do?). Can you give us an idea?

Posted by TomK, 04-24-2005, 10:39 AM
We are still investigating the root cause, in the mean time we are tyring to bring up BGP off the core router to bring the sessions and connectivity back until we determine what is wrong with the core. As for an ETA, I will know more shortly on the progress and how long it will take.

Posted by RadioRob, 04-24-2005, 10:49 AM
In cases of massive network issues, all lines would be busy anyway. They're answering e-mail sent to support@powervps.com and support@defenderhosting.com pretty quick letting folks know, and they also sent out a mass e-mail to all the customers letting them know.

Posted by JenniH, 04-24-2005, 10:52 AM
>> They're answering e-mail sent to support@powervps.com and support@defenderhosting.com << Really? We've not had a response yet..... hence the need to communicate here. Tom: Any closer to that ETA yet?

Posted by RadioRob, 04-24-2005, 10:54 AM
A mass e-mail was sent to all e-mail accounts on file explaining they were having network issues. They also have been replying to support e-mails that have been sent. Below is the message I got: Hello, This is just a quick update to acknowledge the unavailability of your server. This is due to a network issue. The hardware nodes themselves and your servers are online. We're currently investigating the problem. We're trying to determine if it's a DDoS attack or something else. We sincerely apologize for the issue and are working to restore service as quickly as possible. I will update this ticket as soon as possible once we know something further.

Posted by JenniH, 04-24-2005, 10:56 AM
Obviously a mass email is NOT the same as "They're answering e-mail sent to support@powervps.com and support@defenderhosting.com". Regardless, at this juncture all I want is the sites back up, or failing that, an ETA so that we have some data to work with and base decisions on.

Posted by TomK, 04-24-2005, 10:58 AM
What is the ticket number you submitted? As far as I am aware all tickets have seen a response, and all customers have been sent an announcement email. We are moving all of the affected subnets off of the core router and onto other routers as I type this until we can fix the main problem.

Posted by CeriPlex, 04-24-2005, 10:59 AM
Jenni, I think its a case of either/or here. Either we personally respond to every downtime ticket and tripple the time it takes to resolve the issue, or we bulk reply to every downtime request and focus on getting the router back up.

Posted by JenniH, 04-24-2005, 11:01 AM
>> We are moving all of the affected subnets off of the core router and onto other routers as I type this until we can fix the main problem.<< How long will that take? I'll looking for a time based ETA... For ticket numbers try CFG-10196-357 or maybe HVW-64403-518. Don't forget though that your ticket system gives us: "Fatal error: Call to a member function on a non-object in /var/www/html/powervps.com/support/cerberus-support-center/includes/widgets/track_tickets/box_track_ticket_view.php on line 60" when we try to open the ticket itself. My prime concern though is the ETA. Can you offer an estimate?

Posted by TomK, 04-24-2005, 11:04 AM
JenniH- It is impossible to give an ETA in these circumstances. As I type this it looks like everything is back up except for one subnet, which we are working on. I also looked into those two tickets and they DID get responses from our staff.

Posted by JenniH, 04-24-2005, 11:08 AM
>> I also looked into those two tickets and they DID get responses from our staff. << We havn't got an email response on them... and as I mentioned, we can't get into the tickets themselves. Maybe the former is caused by the latter. Either way, WHT is currently the only way we can contact you. >> As I type this it looks like everything is back up except for one subnet << Our VPS is still down, so I assume it is on that subnet.

Posted by JenniH, 04-24-2005, 11:12 AM
The sites seem to be back up Tom. Will this be stable (they went down again earlier)? Also, could you get someone to look at out ticket errors? We feel exposed without the ability to use the official support system. Thanks.

Posted by RadioRob, 04-24-2005, 11:13 AM
My VPS just came back up. If you give me your domain name, I'll try to check to see if I can access it.

Posted by TomK, 04-24-2005, 11:17 AM
JenniH- All subnets and VPS servers should now be back online. I assume you sent that email from an address NOT on your VPS right?

Posted by rokamortis, 04-24-2005, 11:20 AM
I didn't get an email from the response to my first ticket either. It was responded to, I just never got an email. I also got the error viewing online like JenniH did - so I never knew it was responded to. So it just appears to be a problem with the ticketing system.

Posted by JenniH, 04-24-2005, 11:27 AM
Yes, it is the ticketing system. Definitely not our emails. And Tom... we did open the tickets from within our account. We simply get the error message when we try to see your responses. Hopefully this can be sorted reasonably quickly. Thanks.... Jenni Last edited by JenniH; 04-24-2005 at 11:30 AM.

Posted by AllenB, 04-24-2005, 11:51 AM
Hello, Thanks Tom and crew!!! We have multiple servers w/ Defender and couldn't be happier. The service and response has been excellent - Including today's. I believe we may have been one of the first to notify that we were affected by this. I believe it was done w/ a simple email to support@. We did not ask for an ETA as we know what it is like when engineers are searching and sorting. Thank you for A) the response we received on our email, B) the email you sent to those affected and C) the post here at WHT. Please keep up the good work. Best wishes,

Posted by DensityHost, 04-24-2005, 12:01 PM
Im still down. I did go back up, but now im down again.

Posted by dean-ircomdesign, 04-24-2005, 12:01 PM
My vps and powervps.com is still unaccessible. I only can reach defenderhosting at the moment.

Posted by DensityHost, 04-24-2005, 12:04 PM
I can reach powervps.com but not my VPS.

Posted by JenniH, 04-24-2005, 12:07 PM
I don't believe it... we are down again too! 04/24/2005 12:06:54 Atlanta USA Error N/A 20.04 N/A 04/24/2005 12:06:54 London UK Error N/A 20.03 N/A 04/24/2005 12:06:54 Chicago USA Error N/A 20.04 N/A 04/24/2005 12:06:53 Oklahoma City USA Error N/A 20.04 N/A 04/24/2005 12:06:54 Orlando USA Error N/A 20.04 N/A 04/24/2005 12:06:54 Frankfurt GERMANY Error N/A 20.04 N/A 04/24/2005 12:06:54 Hong Kong CHINA Error N/A 20.04 N/A 04/24/2005 12:06:54 Las Vegas USA Error N/A 20.04 N/A

Posted by rokamortis, 04-24-2005, 12:07 PM
yep - looks like it is down for us again as well.

Posted by rokamortis, 04-24-2005, 12:09 PM
Looks like it came back up as I was posting

Posted by Stan Marsh, 04-24-2005, 12:11 PM
PowerVPS is up and all my sites are down. And, I'm experiencing the same problem like JennyH with one of my tickets (I've submitted a second one, when realised that I can't check the status of the first one by any means (email is also down...)).

Posted by TomK, 04-24-2005, 12:12 PM
We are still working on it. Seems to be a Juniper router bug and we are routing around the affected core router until we can fix it.

Posted by TomK, 04-24-2005, 12:19 PM
Okay everything should now be up and 100% stable. If you are still down, please reply to your opened ticket or send a new one into support@powervps.com or support@defenderhosting.com

Posted by TomK, 04-24-2005, 12:29 PM
This is preliminary, we are still doing a postmortem to find out if this was indeed the root cause. We had an issue with one of the trunks off the Juniper to the core switches. We are not sure if it is/was a JUNOS issue or something on the core switches, all subnets are now back up and operational. I want to thank Richard from nLayer who came on-site with us and helped troubleshoot these issues. I'll post later once we have more information, if you are still down and having problems, please submit an email to support@powervps.com or support@defenderhosting.com if the helpdesk isn't working for you. We are truly sorry for this major outage.

Posted by MatthewN, 04-24-2005, 12:32 PM
Thanks Tom and team and also Richard! We are all back up now.

Posted by askthexperts, 04-24-2005, 01:38 PM
Thanks for the email heads up in regards to the outage and the hard work and dedication that got this issue resolved.

Posted by Stan Marsh, 04-24-2005, 01:42 PM
The dedication behind this was VERY big. I wonder how dare somebody even try to compare PowerVPS and late-Dynix. Thank you, PVPS for your hard work, even in a stress like this.

Posted by blueface, 04-24-2005, 02:29 PM
I think DefenderHosting/PowerVPS has dealt with this downtime very professionally. They sent in an email blast immediately after the problem was detected and keep us all posted. Thanks and keep up the good work!

Posted by TomK, 04-24-2005, 02:43 PM
JenniH, Did you register a new account in the new HD, the old ones from before we switched do not work? If you did, please try again, and let me know if you can submit and review tickets now, but remember you *must* register a new account in Cerberus, you cannot lookup your history unless you do. Let me know if you are still having problems, also, email works, support@ either domain, trust me, we got about 1100 tickets via email to the HD. If your email tickets are not getting through, what does your mta say in your logs?

Posted by JenniH, 04-25-2005, 02:56 AM
Tom: I registered when the problem emerged. Maybe it was that. It does seem from above though that quite a few others were having the same problem as me. Perhaps they regsitered at a similar time? Who knows. Either way, I'll send emails from several sources next time (if there is a next time) just be sure. Thanks for following up by the way... appreciated. Jenni

Posted by Stan Marsh, 04-25-2005, 03:13 AM
Yes, I can confirm the problem with the helpdesk, but it isn't PVPS problem. It must be a bug within Cerberus.



Was this answer helpful?

Add to Favourites Add to Favourites    Print this Article Print this Article

Also Read
Is eBay down? (Views: 677)

Language: