The Hostworks Network Outage

Portal Home > Knowledgebase > Articles Database > The Hostworks Network Outage

Posted by THW-Dave, 12-07-2003, 12:58 PM
We are currently experiencing the aftermath of a network hardware failure, all services will be unavailable until the faulty hardware is replaced. The hardware is a upstream provider issue, and we are currently working with Sprint and Verizon (local telco) on resolving this issue. We will try to keep everyone up to date on our progress in resolving this issue. Thank you David @ The Hostworks www.thehostworks.com
Posted by bobbyt, 12-07-2003, 05:47 PM
its been over a day and a half, what's an estimated repair time on this? My email has been bouncing all weekend, all my ebay auctions have broken images.
Posted by bobbyt, 12-08-2003, 09:42 AM
Over 2.5 days downtime now for your entire datacenter and 2 days of flakey connectivity leading up to this. Where are these updates you speak of? When will this be fixed? What is being done to ensure this is fixed by that time? Last edited by bobbyt; 12-08-2003 at 05:02 PM.
Posted by bigjig, 12-08-2003, 11:18 AM
Yeah, i'd like to have some info to pass on to others. I'm a part of a nonprofit organization that is using a page there to run a large rundraising campaign at the moment... so this is bad timing to say the least. This is the first downtime i've seen since i've been with them, however... and i've been with them maybe 3 or 4 months?
Posted by Papa Smurff, 12-08-2003, 01:42 PM
Wow, this is a big outage. Glad I don't colocate with Hostworks. What is the EFT (Estimated Fix Time)
Posted by bigjig, 12-08-2003, 05:46 PM
It's EOD and no ETA on EFT. Hopefully he can RE to our posts ASAP.
Posted by ryce, 12-08-2003, 05:53 PM
resolved ns2.thehostworks.com to 24.97.13.162 resolved 24.97.13.162 to rrcs-nys-24-97-13-162.biz.rr.com Maybe they forgot to pay the cable bill
Posted by THW-Dave, 12-08-2003, 10:36 PM
Thanks for the investigative research, Ryce, We have had MAJOR problems, well, not just us, but 1200 other local circuts in our neibhorhood. Due to water damaged, Verizon has 1200 pairs damaged, and have been working around the clock to repair these. At this current time, Verizon has officially posted a ETR or 10:00am Tuesday Morning. We are doing our best to keep everyone up to date, unforturnatly, many clients primary email on records point to servers we have in our DC, which renders them useless in this matter. We have spoken to most of our clients, as we have been explaining the progress to those who have been in contact every step of the way. We are gifted with the patience of our clients at this point and are thankful to have such a understanding and patient client base. Thank you for your concerns, we are pushing the LEC as hard as possable to futher quicken this repair. Regards Dave @ The Hostworks
Posted by THW-Dave, 12-08-2003, 10:38 PM
I will also like to add that our backup DNS is indeed on a dedicated cable link, which has proven to be beneficial to this, for we are still able to keep TTLs alive on all the domains we authortively host. And as you may have guessed, NS2 is in totally different physical location. Thanks Dave
Posted by THW-Dave, 12-09-2003, 07:39 AM
We are back up, 13 hours earlier than Verizon estimated, 10:34PM Monday Night EST. I thank everyone for sticking with us through this extended outage, and for current clientele, please keep an eye out for a email outlining the event, and what we are prepared to do in return for your outstanding patience and understanding. Sincerely David @ The Hostworks
Posted by bigjig, 12-09-2003, 12:16 PM
I think it's down again
Posted by bigjig, 12-09-2003, 12:29 PM
It's back up again!
Posted by THW-Dave, 12-09-2003, 12:47 PM
Hah, yes Verizon has yet to issue a "all done". They repaired the initial issue, from what we are seeing, but are performing preventive maintence on their network where the damaged occured. Please check back for updates. David
Posted by bigjig, 12-09-2003, 07:49 PM
Been down again for quite a few hours. Any new updates?
Posted by THW-Dave, 12-09-2003, 07:49 PM
We have been down for the past 3 hours to do the cable damage which was temporarily fixed last evening by Verizon, we are waiting to hear another ETR on this outage, and I will keep everyone up to date on the progress. Thank you for you patience. David
Posted by THW-Dave, 12-09-2003, 10:05 PM
Verizon just called me stating the circut is 100% back up and running error free. They replaced a number of damaged cables, and doubler cards which were damaged from the damaged cable. We have no reason to expect any further downtime.
Posted by bobbyt, 12-11-2003, 01:42 PM
Thank you for all your help in getting us back on track and your generosity for making up for recent troubles.