Burst.net [merged]

Portal Home > Knowledgebase > Articles Database > Burst.net [merged]

Posted by John[H4Y], 12-05-2006, 11:39 AM
Anyone else noticing burst.net down?? What a bad time for this for us..
Posted by Odd Fact, 12-05-2006, 11:41 AM
Yeah I am down here. Let's give it a few and see what's happening. I was just logged into a server.
Posted by sysc, 12-05-2006, 11:43 AM
Yea, its been down for about 10 mins now.
Posted by sysc, 12-05-2006, 11:46 AM
Just called, they dont know why its down or have an eta for it to be back up.
Posted by WhiteBear, 12-05-2006, 11:48 AM
Yeah, in Brazil burst.net and nocster.com are down.
Posted by HostJedi, 12-05-2006, 11:50 AM
We are aware of the issue, and are working to resolve it as quickly as we can. . .
Posted by Arsalan, 12-05-2006, 11:53 AM
yep. lets hope it dosent stay offline for too long.
Posted by sysc, 12-05-2006, 11:54 AM
bleh, I just signed up with burst too. I was a customer there two years ago. The 12 hr outage they had when the fiber was cut made me move on. Hopefully this is only a few more mins.
Posted by Mat-d-rat, 12-05-2006, 12:00 PM
Have also been down now for 30mins+
Posted by Kran, 12-05-2006, 12:00 PM
2 servers Down from Colombia too
Posted by Rich, 12-05-2006, 12:07 PM
We're also down.
Posted by linux2man, 12-05-2006, 12:11 PM
Me too down form UAE I've over 6 servers there
Posted by a7tech, 12-05-2006, 12:25 PM
Our servers are also down
Posted by AlexAT, 12-05-2006, 12:26 PM
My servers also down - for more than 40 minutes now.
Posted by John[H4Y], 12-05-2006, 12:27 PM
Please, burst.net, let us know when you know what the problem is and an ETA as soon as possible.
Posted by gilman01, 12-05-2006, 12:37 PM
An ETA would be nice - or at least some idea of the problem - it's been an hour since my first notice and the telephone calls are coming in and I've got nothing more than "we know about the problem" to pass along to my customers.
Posted by rogerdavis, 12-05-2006, 12:39 PM
Im glad I looked on here as my ones are down too, been about 40mins now !!
Posted by hostpc.com, 12-05-2006, 12:44 PM
Our Burst/Nocster servers are down > 1hr now. Anyone heard an update from Burst?
Posted by glace, 12-05-2006, 12:45 PM
I assume it's going to be back in around 30 minutes. Usually outages that take longer than 5 minutes take around 1 to 1.5 hours.
Posted by djblamire, 12-05-2006, 12:46 PM
Anyone heard anything from Burst.net ?? Thanks Daniel
Posted by BobbyBlackwolf, 12-05-2006, 12:46 PM
Ours have been down as well. This is not good timing as we have just released some new content that we have advertised as being up...
Posted by HostJedi, 12-05-2006, 12:48 PM
Update: Issue appears to be on the Commonwealth Telephone (CTSI) network. Network engineers are currently working on the issue. As soon as exact details are known, we will provide you with further information. . .
Posted by hostpc.com, 12-05-2006, 12:50 PM
Isn't there a failover or redundancy to get past that?
Posted by gilman01, 12-05-2006, 12:53 PM
Thanks for the update - now that's something I can give to my customers.
Posted by John[H4Y], 12-05-2006, 12:55 PM
I would think the same thing..?
Posted by WhiteBear, 12-05-2006, 12:55 PM
Our servers are downtime to more than the 1:30 hs.
Posted by HostJedi, 12-05-2006, 12:58 PM
<< Isn't there a failover or redundancy to get past that? >> Yes, we purchase fully redundant capacity from them---two completely different fiber paths and entrances from our facility in Scranton all the way to our POP in Philadelphia. Our equipment is showing healthy at both locations/ends. This issue has us quite concerned as well, as an outage should not have occured on their network due to this redundancy. We have seen our service in the past switch over to the redundant path during a previous issue, but why that has not occured here we do not know yet. Our immediate concern is restoring service, and afterwards rest assured we will get to the bottom of that. . .
Posted by HostJedi, 12-05-2006, 01:11 PM
UPDATE: We have just received confirmation from CTSI that the issue is on their end. They are reporting multiple clients are affected with outages. They are currently working to correct the issue. We will update you as further information becomes available. . .
Posted by hostpc.com, 12-05-2006, 01:17 PM
Thanks for the update
Posted by Arsalan, 12-05-2006, 01:37 PM
Thanks for the update.. lets hope this is fixed soon.
Posted by glace, 12-05-2006, 01:41 PM
Do the guys from CTSI have an ETA ? It's been lasting for around 2 hours I think.
Posted by hostpc.com, 12-05-2006, 01:49 PM
We just passed the 2 hr mark for downtime... do we have an ETA yet? This "redundancy" issue isn't giving me warm-fuzzy feelings.
Posted by sysc, 12-05-2006, 01:56 PM
This happened a couple of years ago with nocster, was >10hrs of downtime.
Posted by glace, 12-05-2006, 02:09 PM
Should be 2.5 hours so far... Maybe Burst should use another backup ISP for redundancy purposes. If you are taking both lines from the same ISP you will run into issues like this.
Posted by HostJedi, 12-05-2006, 02:11 PM
<< This happened a couple of years ago with nocster, was >10hrs of downtime. >> We have since overhauled the network and added redundancy to prevent against such an occurence. That redundancy has protected our network from outages multiple times since it was installed, outages which have occured without loss of service to our client base. Obviously this is a failure of our carrier to deliver the redundancy properly that we have been paying for in this instance---but the reason why is not known yet. We will be getting to the bottom of that once service is restored, and take neccessary steps to prevent such an occurence again with this carrier, or add an additional carrier if need be.
Posted by glace, 12-05-2006, 02:16 PM
Does your ISP have an ETA yet (or any comments that allow to make assumptions at least)
Posted by HostJedi, 12-05-2006, 02:18 PM
<< Maybe Burst should use another backup ISP for redundancy purposes. If you are taking both lines from the same ISP you will run into issues like this. >> There should not be if they are completely redundant fiber paths, equipment, entrances, etc...as we are paying for there to be. It is the same as having two completely different carriers, but we just pay the bill to the same company and have the same support/tech contacts. This level of redundancy has protected us successfully during the past 2+ years, but has failed us for yet unknown reasons during this issue today. We will be looking into implementing further/different redundancy due to this outage, with an alternate carrier. . .
Posted by sysc, 12-05-2006, 02:18 PM
Is this going / looking to b e another 5-12 hr outage?
Posted by HostJedi, 12-05-2006, 02:19 PM
UPDATE: CTSI has confirmed a fiber cut in the Philadelphia region. Engineers are currently working to repair the issue. Why our service has not switched to the redundant path/route as it has done so in the past is being investigated. . .
Posted by sysc, 12-05-2006, 02:21 PM
Ahh, so same exact issue of 2 years ago I must be bad luck, I canceled then because the outage was >10hrs. I signed back up last week and its happening again.
Posted by HostJedi, 12-05-2006, 02:25 PM
<< Is this going / looking to b e another 5-12 hr outage? >> We certainly hope not. Worst case scenario we have to wait for the fiber splicing to be completed. Best case scenario, CTSI corrects the redundant path info, and we switch over as we should have. CTSI is working on both currently. I do know we are a priority however, as we are their largest client in Northeast PA, and they know the redundancy failed on their end as well. . .
Posted by surebro, 12-05-2006, 02:27 PM
Hi, We cannot access our servers for last 2 hours in burst, even burst.net website is not opening. Any one is also facing the same issue?
Posted by nkawit, 12-05-2006, 02:28 PM
Good to know BurstNet is taking CTSI head on and updating us ... for me personally keeping my clients informed is a MUST. Prevents hostility. As for you sysc ... stay away from our hosting providers!!! or we beat you with spikey 2x4's! j/k
Posted by glace, 12-05-2006, 02:28 PM
>>Worst case scenario we have to wait for the fiber splicing to be completed. That would take long, right ? Like 12 to 24 hours...
Posted by crunch42, 12-05-2006, 02:29 PM
Yeah, my two nocster servers are down right now for a few hours.
Posted by HostJedi, 12-05-2006, 02:31 PM
<< Ahh, so same exact issue of 2 years ago I must be bad luck, I canceled then because the outage was >10hrs. I signed back up last week and its happening again. >> There is a major difference there... At that time we we did not have proper/sufficient redundancy, and we can only blame ourselves for that outage. However, at this time we do have the proper redundancy in place, and due to yet unknown reason(s) that redundancy has failed. The blame clearly lies on our carrier at this point, although that is little consolation to our client's downtime experience. We felt that we had resolved such issues from ever occuring again. All we can do is implement a different redundancy method for the future, which I assure you we will be doing after this experience. . .
Posted by TCP/IP Warrior, 12-05-2006, 02:32 PM
Confirmed from this side of the world surebro. Both burst.net and nocster.com are inaccessible.
Posted by Odd Fact, 12-05-2006, 02:35 PM
LIke most I do not enjoy downtime, however compared to two years ago Burst has made great improvements in hardware and service. We can maon about the outage but Burst is a good value for the money.
Posted by bteeter, 12-05-2006, 02:36 PM
Thanks for the update. Lets hope they are fast splicers...
Posted by cybaix, 12-05-2006, 02:38 PM
Ditto.. I can't reach our server on BurstNET/Nocster or the burstnet/nocster website.
Posted by HostJedi, 12-05-2006, 02:38 PM
<< That would take long, right ? Like 12 to 24 hours... >> Doubtful that long, unless a severe obstacle is run into, such as not being able to get right of way to access the fiber for repair. That is what happened 2-3 years ago to delay the repair, but it was an unusual circumstance. I would expect service to be restored today, one way or another---it better be. BTW, we have two GIG-E circuits with CTSI to Philly, not just one, and both of them are "fully redundant", and both of them are down. CTSI has no explanation as of yet as to why the redundancy failed. . .
Posted by TheDPQ, 12-05-2006, 02:39 PM
West Coast and i can't get to burst.net
Posted by HostJedi, 12-05-2006, 02:41 PM
<< Lets hope they are fast splicers... >> CTSI uses Henkyls & McCoy or all their fiber install/repair work---one of the oldest and best in the business, and based very close to the outage (Blue Bell, PA). . .
Posted by rileez, 12-05-2006, 02:42 PM
Splice would be an understatement when dealing with fiber.
Posted by TheDPQ, 12-05-2006, 02:43 PM
For more information you can go here: http://www.webhostingtalk.com/showthread.php?t=567387
Posted by Forwi, 12-05-2006, 02:43 PM
cant get to any of their sites or our server here in UK
Posted by hostlab, 12-05-2006, 02:44 PM
Wow this downtime isn't good - I hope we are all compensated for this...
Posted by surebro, 12-05-2006, 02:44 PM
Just got the information from other peoples.. servers beginning with 66.197.* all are down
Posted by cbwass, 12-05-2006, 02:45 PM
Looks like it could be quite a few hours then, my dog will be very pleased, a nice long walk.
Posted by HostJedi, 12-05-2006, 02:45 PM
BTW, anyone who is not aware of whom CTSI is, or never even heard of them, they are the 3RD largest CLEC in the entire US, and based right in our backyard, They were recently acquired by Citizens/Frontier in a billion dollar plus deal in the past 2-3 months. Just in case you were wondering, and thinking we were trusting our entire operation to some no-name backwoods phone company...that is absolutely not the case. They own most of the fiber in our region, and are not a typical CLEC reselling Verizon's fiber. . . Last edited by HostJedi; 12-05-2006 at 02:52 PM.
Posted by HostJedi, 12-05-2006, 02:47 PM
Please see here for details: http://www.webhostingtalk.com/showthread.php?t=567387 . .
Posted by djblamire, 12-05-2006, 02:50 PM
How does it work - Would we be entitled for compensation ?? Thanks Daniel
Posted by RobM, 12-05-2006, 02:51 PM
Yes I can confirm CTSI is a big company as we own a large security firm and do work with telecom companies in NY, NJ, PA. Here a good question for burst, as I know its available and has been available for many years. Why just go south to philly and not have the backup going north to NY or to NJ Also just because you have redundancy with them, have you ever asked to make sure its not in the same line / feed and have it in writting? ( I seen this before, cut 1 and both go down ) You can see CSIT map -> http://www.ctsi1.com/carrier/bigmap.html It does go other ways then just to philly, which could have prevented this outage....
Posted by sysc, 12-05-2006, 02:52 PM
I dont remember being compensated the last time it happened. I could be wrong though.
Posted by fleed, 12-05-2006, 02:57 PM
Just a big thanks to Burst for keeping us informed. This is a great improvement on the last outage which affected me, with almost no info and a very hostile attitude. It really makes it better to be fed information on what's going on, even if it's just a ping every 1/2 hour to say nothing's changed, you still looking into it, etc. THANKS!!!
Posted by glace, 12-05-2006, 02:59 PM
I don't think there will be any compensation.
Posted by HostJedi, 12-05-2006, 03:00 PM
<< Also just because you have redundancy with them, have you ever asked to make sure it’s not in the same line / feed and have it in writting? ( I seen this before, cut 1 and both go down ) >> Yes, of course--and a fiber cut should NOT take us out, and has not when it occured in the recent past either on this same redundancy setup. It has happened in the past year, and our client have not known about it, since the redundancy took over. Why it did not this time remains to be answered yet. Based on that CTSI fiber map which was posted, I'll supply you a little details. We have one route that goes straight south thru Allentown, and another tha goes Southwest thru Harrisburg, As you can see on the map, these are two completely different paths. A fiber cut should not have taken these circuits out. . .
Posted by glace, 12-05-2006, 03:01 PM
Yes, I feel the same way. This time they deal better with it. However, I fear this is going to be a long, serious outage. I wouldn't be surprised about many more hours of downtime.
Posted by Cirtex, 12-05-2006, 03:02 PM
Hi, Is there any ETA from them at least so we can update our customers on burstnet servers? This isnt good as we've had a new server setup with new customers signing up and this is completely ruining it for us today I really hope things can return to normal within next hour, but I'm probably hoping for too much :\ Please try your best to get things resolved asap. Cheers
Posted by Nick H, 12-05-2006, 03:04 PM
I believe you're hitting around the big question that people want to know: Will they be compensated for the downtime?
Posted by HostJedi, 12-05-2006, 03:07 PM
<< Why just go south to philly and not have the backup going north to NY or to NJ >> We know all the fiber available in the area, and the vast majority goes south to Philly. There is some outside of the city that goes north to Binghampton, and East to NJ---but getting to it is proving complicated. A few other options exist, but they are priced out of even being an option or they do not have capacity to service our facility. The state of PA is currently working on a huge fiber project which will bring fiber from NJ/NY direct to NE PA...but that is awhile off yet, and surprisingly does not yet exist. For this reason there is no affordable transit in NE PA directly yet---we actually probably have more capacity in Scranton ourselves than most of the carriers here. We have been looking into another route, which we planned on adding in 2007 even before this issue occured. . .
Posted by hostpc.com, 12-05-2006, 03:07 PM
Fortunately, I've only got one server for our customers in Burst, the others are resold to another company... I really hope this gets resolved without an entire day of downtime. Thanks for the updates (Benji?) - looking forward to good news soon.
Posted by HostJedi, 12-05-2006, 03:11 PM
<< I believe you're hitting around the big question that people want to know: Will they be compensated for the downtime? >> We have no answer on this yet, and is not our immediate concern---service restoration is. Our management will most likely have a meeting after this is done with to discuss the matter. Our SLA does include clauses for fiber cuts, "acts of god" etc..,in thepast we have given some compensation for such extreme issues, but in the past we also felt like we were to blame for lack of proper redundancy, unlike this outage. I would recommend contacting Customer Service once service is restored, and inquire---but give them a little time to get back to you because they will obviously be swamped with inquiries. Thank you all for your continued patience and understanding... . .
Posted by glace, 12-05-2006, 03:12 PM
I think if redundancy could be restored it would have been done already. Since CTSI did cut a line (quite an important line as it seems) I wouldn't be surprised about 5 to 12 more hours of downtime. They really need to repair an entire line...physically. That's going to take long.
Posted by hostpc.com, 12-05-2006, 03:15 PM
What if I'm an aethist and dont believe in "god"? LOL- sorry, couldn't resist
Posted by HostJedi, 12-05-2006, 03:15 PM
<< Thanks for the updates (Benji?) - looking forward to good news soon. >> Shawn actually....PRES/CEO...Benji is one of many manning the phones right now, which are riging of the hook as you can imagine. . .
Posted by HostJedi, 12-05-2006, 03:18 PM
<< What if I'm an aethist and dont believe in "god"? >> "act of nature" or "act of weird science" then ;-) Thanx for the moment of humor, as you can imagine...I could use it right now in this stressful day!...and I'm already on only 4 hours of sleep last night due to working on a large project :-( . .
Posted by server4sale, 12-05-2006, 03:23 PM
Guys give them some break. Stuff happen.... They are doing there best. Hey Shawn should i start singning akoona matata
Posted by RobM, 12-05-2006, 03:24 PM
I know how the 4 hours feels like, did that all week helping the other company we own out... Had a good nite of sleep last nite, so it helps little today
Posted by Arsalan, 12-05-2006, 03:25 PM
lol, lets just hope this gets fixed soon..
Posted by WhiteBear, 12-05-2006, 03:26 PM
BurtstNET... Already had been 4 hours of downtime. All we are losing customers. How much time more for the decided problem?
Posted by Mat-d-rat, 12-05-2006, 03:29 PM
These kind of comments are not helpful... the situation has been told, it's not Bursts fault and they are waiting for CTSI to repair the fibre link and find out why the backup line has not kicked in. So what do you want Burst to do?
Posted by sysc, 12-05-2006, 03:31 PM
I dont see it getting fixed soon, this is exactly the same thing that happened a couple years ago. Same kind of replies from burst as well.
Posted by GordonH, 12-05-2006, 03:33 PM
It is most likely a cut in the city somewhere on the cable company's own network and not on either of the two seperate lines. I have seen this sort of thing before. The important thing is that this is going to take a long time to fix and there is a risk that the routes will have gone stale or there could be routing table problems once it comes back up. This is another problem i have seen before and not just at Burst. A lot of ISP's drop the routes if there is a long outage. In my opinion this is a more serious risk than the mechanical problem with the cable, which will be fixed when it is fixed.
Posted by JenniH, 12-05-2006, 03:33 PM
How about contacting CTSI, and demanding a progress report: an ETA, details of what has been done already and what is left to do, etc? Equally, I would argue that ultimately responsibility does lie with BURST as they selected the supplier and created their own infrastructure. I'm not wishing to sound confrontational, but more detail on the specifics of what is happening and the progress would help everyone. The lack of this tends to make be suspect we still have a long way to go.
Posted by hostpc.com, 12-05-2006, 03:34 PM
Indeed. s*** happens - it's unfortunate, but I'm sure Shawn, Benji and the gang are working as hard as they can to regain stability - and will address the redundancy issues later. Harassing them aint gonna make it go faster. They've been very good about providing updates here as the time goes on, on a regular basis ... lets cut them some slack, sit back, have a cappuchino and tell our customers we're (and the datacenter) is doing the best we can with the situation. John Doe can't go fix a fiber cut - it takes engineers. It WILL happen, we just need to wait it out. Just my 2c
Posted by BuycPanel-Jeff, 12-05-2006, 03:38 PM
Hi sysc, Our hosting company knows Shawn and his team personally. I can guarantee you they are doing everything in their power to restore service. Remember, they are loosing time and money each minute too.
Posted by sysc, 12-05-2006, 03:41 PM
Oh, I have no doubt. I was just saying, same replies as a couple of years ago when this happened. I have no doubt its going to be a while before its up.
Posted by HostJedi, 12-05-2006, 03:46 PM
Believe me, we have been on the phone constantly with CTSI thru this whole ordeal, from sales reps, to customer service, to tech support, to engineering, every dept we have been in contact with. As we have been getting information, I have been relaying that information to you. As soon as we have an actual ETA we will pass that along to you. UPDATE: The fiber cut is directly in Philadelphia on the street. Apparently the cut was done by a contractor doing work for a local fiber company (Sunesys)---this part is unofficial though and just what we have "heard" from unofficial sources. This is somewhat good news, as it tells us this is in a public area, and not in a problematic "right-of-way" location which may have caused access issues for the repair crew. A repair crew is currently on-site working on the issue we have been told. No word on the redundant path yet, but we are continuing to pursue that of course still. . .
Posted by JenniH, 12-05-2006, 03:47 PM
What makes me suspect the worst is the lack of specific detail on the outage (progress of recovery). The top guy at CTSI is bound to have an idea now of how long.... or at least what is done and what is still to do. Why can't we be told that? Perhaps the answer is that it is bad news? I really hope I am wrong, but that is how it looks to me. UPDATE: Fair enough BURST... but are CTSI not venturing an estimate?
Posted by RobM, 12-05-2006, 03:49 PM
I wonder if they are Union (Sunesys)-> Union fight coming, love those...
Posted by GordonH, 12-05-2006, 03:51 PM
Here is what we are telling customers: Maybe that sounds a bit trite, but there is nothing anyone can actually do.
Posted by RobM, 12-05-2006, 03:56 PM
It not just look in the hole and say give me 3 feet new cable... First you have to see if more damage was done like after they cut the fiber did it get stuck to the backhoe and get pulled apart elsewhere which will causing a much harder break to repair...
Posted by Eric HRF, 12-05-2006, 03:59 PM
That link is a joke burst net. "We are aware of the problem, um, we'll fix it by reloading a backup" Last edited by Eric HRF; 12-05-2006 at 04:14 PM.
Posted by server4sale, 12-05-2006, 04:00 PM
We have been with BUrst for i guess 5 years now i think.... They had ups and downs no doubts. But they always come back. Why not just move etc Cause they care about there client. Atleast the CEO is here sitting and updatingunlike some other dc i have seen who keeps saying we are working on it and no updates.
Posted by bhaputi, 12-05-2006, 04:06 PM
JenniH, I am the person who is personally handling all of our contact with CTSI, and I can tell you that what Shawn has told you so far is everything we know. No ETR at this time, but crews are on site. We will update as we get more information. Keep in mind that this was not an act of CTSI themself, but rather another contractor doing work for another company. We are just "collateral damage" from a backhoe's mistake....
Posted by ClayM, 12-05-2006, 04:10 PM
Yet more proof that the backhoe's natural prey is the underground cable, including those used for telecommunications and power distribution. Seriously, I don't know why they insist on editing that out of the wikipedia entry
Posted by donehosting, 12-05-2006, 04:16 PM
BurstDan, Do you have any update on what the problem is with the redundant switchover and what is being done with that issue. Is it some sort of hardware failure and are they changing the component? Thanks for all of the communication you guys are giving us. It has been very helpful. Our customers are frustrated but by being able to give them updates they have been very understanding. Thanks again and good luck. Best Regards, Rodney
Posted by HW-David, 12-05-2006, 04:18 PM
I'm just gessing I'm on a burstnet server... been down for... 6 hours now.
Posted by linux2man, 12-05-2006, 04:24 PM
How much time to repair this damage ?
Posted by bhaputi, 12-05-2006, 04:24 PM
Unfortunately, no. I have pestered them several times about this and, so far, all I know if that their engineering people are looking into it.
Posted by GordonH, 12-05-2006, 04:26 PM
Cable problems are more common in countries where they put the fibre on poles. In the UK its 99% underground and people are not allowed to dig without a map and sometimes a survey from BT so these sort of things are less common even in cities.
Posted by HW-David, 12-05-2006, 04:26 PM
that means we're screwed We're looking into it = We dunno wtf we're doing. Sorry its been a bad day.
Posted by BobbyBlackwolf, 12-05-2006, 04:28 PM
Apparently, from an aquatence that is in that area, a lot of other individuals are also without Internet due to this outage - it's not just Burst that's hit by this. That doesn't make my boss's temper go down any, though.
Posted by GordonH, 12-05-2006, 04:28 PM
and thank you Dan and Shawn for your updates. I have emailed all our affected customers.
Posted by HostJedi, 12-05-2006, 04:40 PM
<< and thank you Dan and Shawn for your updates. I have emailed all our affected customers. >> You are quite welcome Gordon. We are working on doing the same. . .
Posted by si2040, 12-05-2006, 05:06 PM
This is just really sad. A datacenter like this has been riddled with problems. I do sympathise but it goes to show that there is not enough done in DR planning
Posted by sysc, 12-05-2006, 05:09 PM
I just got off the phone, with someone from burst they are saying by 8pm tonight. Pretty unacceptable. Im starting to wonder if there really is any redundancy? This is EXACTLY to the T what happened two years ago, same replies same amount of time down same situation.
Posted by server4sale, 12-05-2006, 05:15 PM
sysc , Two years ago was a downtime close to much more then this. Atleast we have an ETA now
Posted by JenniH, 12-05-2006, 05:17 PM
Can you enlighten me: how long is that from now (in your time zone)? It sounded like a long one from the lack of any sort of ETR above, but given that it is 9pm here (UK), I'm not sure what your 8pm translate to. Thanks.
Posted by skater boy, 12-05-2006, 05:17 PM
hey burstnet, have you tried viagra, you'll get better 'uptime' than you're currently getting.
Posted by sysc, 12-05-2006, 05:19 PM
4 hrs 30 mins from now, total of 11 hrs of downtime if it comes up at 8pm
Posted by bhaputi, 12-05-2006, 05:25 PM
We have been told that this was a problem with a contractor knocking down a power line (telephone pole), taking power and a large fiber bundle with it. We have been told that power should be restored by 8 PM Eastern Time, at which time CTSI can get access to splice the fiber. No ETA other than that at this point.
Posted by JenniH, 12-05-2006, 05:26 PM
Thanks sysc. Appreciated. The lack of even a broad heavily qualified ETR usually indicates a lengthy outage, which is why I pressed it above. Looks like the early 'doom-mongers' were correct. "at which time CTSI can get access to splice the fiber" Dan: so are you saying that they can only begin work at 8pm? In which case the outage could me much longer?
Posted by twistedgamer, 12-05-2006, 05:29 PM
Depends on the size of the fiber cut and whether its a cut or a nick. Nicks tend to take longer to fix because they actually have to cut the knick out and splice in a new section. Living in Omaha couple years back the dedicated fiber line that severed my 1200+ unit Apt Complex was cut by a road crew and Cox managed to have it repaired in 45 mins. But the fiber wasn't that big. Probably had 50 strands in it. Thats why the US has whats called a diggers hotline. Your not allowed to dig with out them marking the area ahead of time. Its the only way to keep from being sued after the fact. When you call the operator takes your location, your name and your call back number (you'll get a all clear to dig call). Now the companies that have services in there area are responsible for getting one of there locate techs out to mark it with in 24 hours. They don't make the 24 hour window and you hit something your not liable. They mis mark something and your not liable. Once burst is back up, first thing they need to figure out is why there redundancy didn't work, then find out who did the locating and give them a nice fat bill for the down time. I would almost bet it was an on call locating service that doesn't know the area and is going to get a nice big bill from CTSI and hopefully the effected customers for the screw up. If CTSI's own tech did the marking, that tech a) probably going to get fired. and b) compensating the effected customers. edit: darn dan posted while I was typing.
Posted by sysc, 12-05-2006, 05:31 PM
I am just amazed that two years later this kind of thing is still taking burstnet down for a substantial amount of time. No redundancy at all, does'nt matter what you say you have we are seeing it here.
Posted by twistedgamer, 12-05-2006, 05:33 PM
I think a secondary stand alone fiber qualifies as a DRP. CTSI evidently needs theres looked at as the fiber map posted shows that one cut shouldn't have taken burst's second line down.
Posted by Arsalan, 12-05-2006, 05:37 PM
Thank god its the middle of the night here. I hope this is not delayed beyond that, if it is, we would have 1000% more screaming customers.
Posted by matrafox, 12-05-2006, 05:42 PM
The middle of the night is in my country to .
Posted by sysc, 12-05-2006, 05:43 PM
Phone wise, I was told they will have an exact ETA somewhere around 10-11pm EST which puts the downtime well over 13 hrs which is worse than two years ago. Redundancy? I don't think so....
Posted by twistedgamer, 12-05-2006, 05:43 PM
early afternoon for Burst and mid afternoon for me....
Posted by Mekhu, 12-05-2006, 05:47 PM
10-11pm EST!? LMAO... are you serious? Can someone say Cancel!? Thank God we only have 1 machine with them. Never did order more then 1. Jeff
Posted by bhaputi, 12-05-2006, 05:48 PM
Yes, that is exactly what I am saying unfortunately. I wish I had better news, and I hope you understand this is extremely frustrting for us also.
Posted by HostJedi, 12-05-2006, 05:49 PM
<< Redundancy? I don't think so.... >> Are you accusing us of lying about our redundancy? I would not advise of that...as you will end up looking like a fool. We have no problem providing proof of the redundancy once we have a spare minute after service is restored. I assure you this is 100% due to CTSI's failure on our redundant service. Unfortunately we have no idea why as of yet---and I am impatiently waiting to hear their explanation. There is really zero excuse why a fiber cut should take out redundant paths---none at all. I am beyond furious with them, and will be considering legal action--as the extent of the damage is growing by the minute for both our firm and our clientele. . . Last edited by HostJedi; 12-05-2006 at 05:55 PM.
Posted by sysc, 12-05-2006, 05:51 PM
I don't call this redundancy, this is exactly what happened to me the last time I had server(s) with you two years ago. I talked to benji in length last week and he assured me something liek that could never happen again. Going on 7 hours down, does'nt look like very good redundancy to me? You can show us in graphs/network maps whatever floats your boat but bottom line is, 7 hours down going on 12 minimum shows NO redundancy to me.
Posted by Mekhu, 12-05-2006, 06:01 PM
lol, Benji did a good job sweet talking me for two boxes a couple months back. As mentioned I still only have 1 box as we cancelled the other new one within days due to their shotty network... couldn't handle our streaming.
Posted by HostJedi, 12-05-2006, 06:02 PM
Technically it couldn't happen again based on the service we have installed...unless our vendor (CTSI): a) Lied to us about the redundancy, or extent of such b) Botched the redundancy configuration c) Didn't have sufficient capacity to handle the redundancy for so many clientele at once d) Physically and purposely stopped/blocked our service from switching to the redundant route e) Had multiple fiber cuts or equipment failues on both fiber paths simultaneously f) Had an issue on one route pre-existing that was not yet repaired by the time this route damage occured f) or is lying to us now about what actually occured ...whichever it is, it does not look good for them, and my day tomorrow will be spent screaming at them most likely to get to the bottom of this. . .
Posted by jimbo1969, 12-05-2006, 06:04 PM
Are you accusing us of lying about our redundancy? I would not advise of that...as you will end up looking like a fool. Comments like that and the service you provide already proves who is the fool, all of us that pay for your "Budget rate" service... And yes, I call you a liar, you do not know what TRUE redundancy is, If you did, I wouldn't have just lost 2 Clients. Hopefully you'll feel my pain as your clients start leaving you out of fustration of the same issues over and over. The downtime is crazy and is proof your redundancy is non exisitant or conceived by apes. BTW learn how to professionally respond to people, the replies from burstnet on these forums show the childish nature at burstnet... Thanks, PISSED!
Posted by jimbo1969, 12-05-2006, 06:06 PM
one ISP is not redundancy and RULE #1 in PA DO NOT LISTEN TO COMMONWEALTH ABOUT ANYTHING!!!
Posted by Mekhu, 12-05-2006, 06:08 PM
So what provider are you moving to? I would assume you're leaving Burst after this? Any other PA providers?
Posted by RobM, 12-05-2006, 06:09 PM
can I get access to work on a server, I mean there down so now would be a good time, I did try to send a pm but it's not allowed and there is no other way to contact you's...
Posted by hostlab, 12-05-2006, 06:14 PM
I'm glad we also only have 1 server with BURST, thankfully all of my clients' services are on other networks - the server with burst simply holds our development stuff and control panel. All of the other providers I use have maybe 10 min downtime, maybe an hour or two at the max - but none of them have hours on end downtime. This isn't the first time the downtime has been very heavy with BURST this year either: Here are a few other threads to display BURST's poor uptime capabilities over the past 6 months: http://www.webhostingtalk.com/showthread.php?t=525467 http://www.webhostingtalk.com/showthread.php?t=509946 http://www.webhostingtalk.com/showthread.php?t=514963 http://www.webhostingtalk.com/showthread.php?t=510223 (Oh boy, I remember that one!) Last edited by hostlab; 12-05-2006 at 06:19 PM.
Posted by cianuro, 12-05-2006, 06:20 PM
We have two servers with Burst. In fairness, one day in two years is a small enough price to pay for the price we're paying. (little under 1k for both). Of course, I, like everyone else, am not happy. But I appreciate the updates here.
Posted by GordonH, 12-05-2006, 06:24 PM
To be honest this one os not Burst's fault directly. I don't think there is anything they could have done to avoid this one, and I say that after having spent a few days in robust discussion with Shawn about a number of issues we have been unhappy about. If you leave this incident aside as its not their fault, I feel that that the root of the other problems are down to the volume of work they are doing for not enough profit and therefore not enough staff. Its a vicious circle and it means they don't get stuff done that they should e.g. a streamlined and organised upgrade procedure (chargeable obviously and another way for them to make additional money without endless hassle), letting larger customers manage their own IP's on a class C basis across all their boxes so we don't have to request IP's. There are a whole list of these small issues that need addressed and I don't think they have the physical time or energy to get to grips with them. They need a business consultant to help them get the business running more effectively. Thats just my opinion as a customer but there does seem to be a lot of missed opportunities and a feeling of a rabbit caught in the headlights of a car (rightly or wrongly it comes across like that sometimes).
Posted by server4sale, 12-05-2006, 06:26 PM
For client who are angry and pissed, Do you really think burstnet is not doing there best to resolve this? I think they are doing it. After all they dont want to issue credit. Stuff like this happen. Its not burst fault. During the past 2 years they have recovered exceptionally very very very well. Shouting wont help resolve your issue. We are all in same boat. SMA can also get pissed and give no further updates ( well thats a possibility if people start bashing). Here is for the people who are pissed. Hold your anger till network comes back up. Let the situation get normal. After that backup your data. Move somewhere else and then BASH ALL YOU WANT. Jeeez start behaving like adults not whining kids. Whining wont help you.
Posted by gschmidtccs, 12-05-2006, 06:28 PM
It is not the downtime that is most concerning to me, and it should not be anyone's primary concern because there are failures like this that happen behind the scenes all the time. What I really need to understand going forward to have an acceptable level of comfort is the failure in the redundancy and what is done to address that. I have to say that we also co-locate with a much larger provider and are considering consolidating our business with them and I personally visited the burst.net facility last week and am more than comfortable with them as a company. It is important for me to see how they recover from this and address the telco provider failure. Last edited by gschmidtccs; 12-05-2006 at 06:33 PM.
Posted by Mekhu, 12-05-2006, 06:31 PM
Wow, what's with all the Burst love all of a sudden? More mean comments! More mean comments!
Posted by RobM, 12-05-2006, 06:33 PM
I will agree with this, they have invested a lot of money in there network and even with that most still cry, even when they have a few second sometime of downtime. Remember things can and will happen, but you all should be happy to see burst here saying whats going on as they get it.
Posted by viGeek, 12-05-2006, 06:33 PM
looks like things at Burstnet havent' changed @_-
Posted by John[H4Y], 12-05-2006, 06:36 PM
What I don't understand is why the backup link issue is not being investigated further. Waiting for the power company to get out of there at 8pm is a bad option. Why resign to the fact that the backup link cannot be fixed? Why are you not trying to get the backup link going in the meantime? What is Commonwealth's stance on that issue? For all the money you must be paying them, you would think that they would be sending all the workers who are waiting to work on the fiber cut over to try and fix the redundent link issue...?
Posted by sysc, 12-05-2006, 06:42 PM
At a minimum looking at another 4-8 hours down is what they are saying.
Posted by rileez, 12-05-2006, 06:43 PM
Please elaborate on why you think its not being investigated. I think burst is doing a fine job letting us know whats going on. If you read the whole thread you would see what happened to the backup link. Find the part where burst mentioned "Legal Actions".
Posted by JenniH, 12-05-2006, 06:44 PM
The question I ask myself is why Burst? Meaning: just take a look at this forum. How many other DCs do you see mentioned with anything like the frequency of Burst (for a whole DC issue)? Compare them with Servint or HE or anyone else. This is why we too only have a small presense with them. Sorry, it is a diversion from the current issue: but surely sooner or later Burst has to resolve these sort of things? Anyhow: any update Dan? Hopefully you can see why everyone is upset.
Posted by quadix, 12-05-2006, 06:46 PM
Actually, Yes, BurstNET has been down since 11AM this morning. It turns out their main provider CTSI had a few fiber circuits go down in Philadelphia near the POP. We have been on the phone with them (BurstNET) throughout the day and the latest is, CTSI is restoring power at around 8PM and then can perform the Fiber repairs and expect to be back online between 11PM - 1AM EST. Hope this helps.
Posted by viGeek, 12-05-2006, 06:48 PM
Still down... The thing with Burst is it's always something, theres always something doomed for failure. Having two fiber paths with the same company. They need to eliminate any single area of failure whethers its a one company, one fiber path, they appear to have improved some of it. But they really need to get more fiber paths from more then one company IMO. I have been registered on this board since 2002 and nobody has went down more often and longer then Burst, thats something to think about.... Theres plenty of better options out there just do a search
Posted by rileez, 12-05-2006, 06:49 PM
A Whole DC issue has happened before. And I am sure will happen again elsewhere. Remember when a tech dropped a screwdriver into a power box over when it (ev1servers.net) was called Rackshack? How do you get power redundancy?
Posted by kapot, 12-05-2006, 06:50 PM
1 ISP = single point of failure = no redundancy This is a very basic principle of networking. 1 ISP = 50% chance of total downtime 2 ISP = 25% chance of total downtime 3 ISP = ...
Posted by HostJedi, 12-05-2006, 06:53 PM
<< What I don't understand is why the backup link issue is not being investigated further. Waiting for the power company to get out of there at 8pm is a bad option. Why resign to the fact that the backup link cannot be fixed? Why are you not trying to get the backup link going in the meantime? What is Commonwealth's stance on that issue? For all the money you must be paying them, you would think that they would be sending all the workers who are waiting to work on the fiber cut over to try and fix the redundent link issue...? >> I could not agree with your more. From what they tell us all of their engineers are working/involved in the fiber cut issue, and we will not have any answer whatsoever on the redundancy failure until tomorrow at the earliest. That is what concerns/infuriates me the most. At the very least, even if the redundancy failed, they should have been able to manually make modifications to get us up and running on the redundant side. We keep pushing and pushing this point to them, but it is getting us nowhere. There is no excuse for this from my perspective, and this is a serious breach of contract from them. We will absolutely be exploring other options, as we can obviously no longer rely on CTSI's redundancy provisioning to protect our client base. Unfortunately provisioning new fiber/transport service is not a quick thing, especially with a minimal amount of available options to us. We will be pursuing this, but the immediate resolution is finding the problem with CTSI and fixing it in the meantime so it functions properly and as expected, while we proceed on an additional solution/implementation at the same time. . .
Posted by gschmidtccs, 12-05-2006, 06:53 PM
What I don't understand is if the people that are slamming them as always having these problems are still hosting with them, WHY ARE YOU STILL THERE? Is it because it's not as bad as you make it seem and you are just frustrated?
Posted by Mekhu, 12-05-2006, 06:55 PM
Do you guys have any sort of access to the servers? Do you have any staff willing to pull HDD data for clients?
Posted by rileez, 12-05-2006, 06:56 PM
because its just all bark no bite or a majority of them are just to lazy to move once the server is back up. just my opinion and I reserve the right to be wrong
Posted by hostlab, 12-05-2006, 06:57 PM
Exactly... It is not there fault that someone has damaged a fiber, correct. However, is it their fault that they are only using a single ISP to carry their entire network? Yes.. it is. Is it their fault that they don't have a fallover plan, maybe a deal with another ISP who will handle 'temporary traffic' whilst the main connection is being repaired - even if it is highly restricted and slow backup connection?
Posted by glace, 12-05-2006, 06:58 PM
So if I understand that right 8 PM EST is not the time when this issue will be fixed but the time they will only even start working on it and from then on X hours, right ? And 8 PM EST is in 2 hours, right ?
Posted by JenniH, 12-05-2006, 06:59 PM
Clearly it has. The term I used was 'frequency'. Go ahead and search this forum of WHT (Outages). I would be amazed if any other major DC had anything like as many whole DC outages as Burst. If you prove me wrong I will hold my hands up. I would be very surprised indeed.
Posted by rileez, 12-05-2006, 07:00 PM
Their network map is on the same site that clients signed up through. If they didnt like the network, then why purchase a server there?
Posted by rileez, 12-05-2006, 07:02 PM
You may find many threads that have Burst Down but please read through all of them and read why they were down. I can recall seeing a few where the client didnt know how to manage a server. etc.
Posted by glace, 12-05-2006, 07:03 PM
Because some content doesn't move easily. I.e. if your customers are using third party nameservers. You need to contact them and ask them to point to the new DC's IP address. Even if you never tell your customers to use third party nameservers some will still do it and then if they do not respond your email you can't just turn their site off. They may not be receiving your email and just turning their site off may be a violation of your contract. Moving hundreds of hosting accounts from one server to another can be days of work or weeks of work if you have multiple servers. Last edited by glace; 12-05-2006 at 07:07 PM.
Posted by ultra.net, 12-05-2006, 07:04 PM
I will perhaps go back if needed, but didn't I read that there had been an outage some time (weeks/months) ago where the redundant link was used? Under the circumstances, having seen the service they're paying for work OK, what extra do you expect them to do for redundancy? I'm sure we'll all enjoy a "master class" from you, when you have time (there's an hour at least while waiting for the connection to come back, so it might be that you will come up with a 'foolproof' redundancy solution, then again, someone will spot a problem with a 1 in a million chance of it failing, perhaps!) Oh, dear.... Do you consider your 'outburst' to be "professional"?
Posted by gschmidtccs, 12-05-2006, 07:09 PM
I understand the concept. However, once again I ask is that if the service is bad and has been bad wouldn't you have developed a strategy for moving to another provider. At our other co-location facility we have 15+ servers that we own and manage and if the service was bad I would certainly plan to move irregardless of how long it took because I would consider it essential to our business to do this even if it took weeks. However, if you are saying that some of your customers don't respond to email, I would really question the value of that customer and how they would expect you to provide quality service to them. Last edited by gschmidtccs; 12-05-2006 at 07:15 PM.
Posted by glace, 12-05-2006, 07:14 PM
I did develope a strategy. 85% of my customers have already been transferred away from Burst. As a result of this more than 3000 domains are online right now that would be down if I had not moved.
Posted by Mekhu, 12-05-2006, 07:18 PM
Any other providers located in PA with more then 2 fibers!?
Posted by gschmidtccs, 12-05-2006, 07:20 PM
Anyone ever hear the old "people in glass houses...". If you are sitting here hammering burst.net because they are down which in turn has caused yourself and/or your customers downtime, does it mean that you don't have a fail-safe backup/redundancy plan? SAY IT ISN'T SO! Is it possible that you are guilty of the same issues that you are accusing burst of?
Posted by AlexAT, 12-05-2006, 07:24 PM
I use with them Dual Xeon 2GHZ with unlim bandwidth, cPanel&Fantastico and 10 IPs for $139.11 (with TP/SM I have Celeron 2.4 with 1.5T bandwidth, cPanel&Fantastico and 8 IPs for $127 - feel the difference). Do you think with any other provider I can recieve near the same taking into account that in general they support is good?
Posted by Cirtex, 12-05-2006, 07:25 PM
Hey Shawn I'm sure it's been a pretty hectic day for all of us and we'll definitely be processing quite few refunds and cancellations tomorrow morning. It's already dinner time here in New York, I really hope after I get a chance to have some dinner, things will be back up and running. Thanks though for keeping us updated, but in the end it still be best to tell us an ETA, keeping everyone in doubt is probably the worst and when at first I thought this will just be another routing issue temporary one, it turned out to be much more complex. Keep us updated and anything that we can relay to our customers would be the best. Cheers John
Posted by HostJedi, 12-05-2006, 07:28 PM
<< I will perhaps go back if needed, but didn't I read that there had been an outage some time (weeks/months) ago where the redundant link was used? Under the circumstances, having seen the service they're paying for work OK, what extra do you expect them to do for redundancy? >> And that is exactly the case from our perspective. We went out an implemented such redundancy after the major outage 2-3 years ago, spending nearly 1.75x the cost of non-redundant service to make sure our clientele were properly protected, and such a major outage would never occur again. We had no reason to believe that this redundancy would not work based on multiple prior outages in the past two years where it did work successfully. (NOTE: If you have ever seen our latency between Scranton and Philadelphia go from 5ms avg to 8ms for 24-48 hours, that was due to us operating on the redundant side and taking the longer route thru Southwest PA...) We are at a complete loss as to why such did not work this time, and CTSI has some explaining to do. Regardless of the reason however, it is clear to us that even this level of redundancy is not good enough, and we will be pursuing even further redundancy solutions moving forward. This is something that we had plans for in 2007 anyways, but obviously more of an immediate need from here forward. . . Last edited by HostJedi; 12-05-2006 at 07:34 PM.
Posted by spiritedaustralia, 12-05-2006, 07:33 PM
I am based in Australia, and it is currently business hours here, which means our websites would normally be expected to be alive right now. It is only 10.30 in the morning... which means I will have a REALLY long day...
Posted by RobM, 12-05-2006, 07:35 PM
The only thing I can think of is where the cut is located both fibers are in the hole causing both down... anyway any possibility of doing on site work to a server? ( and yes they are colo boxes...
Posted by HostJedi, 12-05-2006, 07:43 PM
<< anyway any possibility of doing on site work to a server? ( and yes they are colo boxes... >> Rob, Give Benji a call on his cell phone...he says you have the #. He'll help you out with this---several other co-location client's have the same idea to make use of the downtime to accomplish something... . .
Posted by dgynepa2, 12-05-2006, 07:45 PM
Like everyone else, this downage is causing me lots of problems, too. I really think that BURST.NET has to rethink and re-engineer their redundancy plan - its not redundancy when relying soley on a single provider and relying on THEIR redundancy. Anyway... it seems as though they are as concerned about this as we are - so we will see how they handle it. REGARDLESS.. I think that it is VERY VERY VERY important for EVERYONE to understand a real simple concept when it comes to the "internet" -- you CANNOT *RELY* on the internet for 100% uptime business services. It is simply not the nature of the internet to have any guarantee whatsoever for functionality. Sure, we try to control what we can, and have redundancy in place as much as reasonable, but the simple bottom line is that there is too much relying on "others" throughout the internet system to be able to count on full functionality all the time. This goes for not only connectivity but simple things such as important business emails getting delivered properly to the desired recipient. Sure, we have all come to RELY on the internet - and businesses exist soley ON the internet, so when internet goes down, it's like shutting down your business, but it's just the nature of the beast. YES IT STINKS when it happens, but it happens; and it happens with EVERYONE. BTW, in all my contracts I clearly state that circumstances beyond my control may cause downages and even data-loss, and if you do not make that clear to your clients, you haven't been working professionally with computers or the internet for very long. As a matter of fact, I have built into some of my mission critical systems the capabilities to handle down-time occurrances as well -- because we KNOW it will happen - and at the worst possible time! Anyway, for one, I would like ALL available resources working on correcting the problem, and THEN, and only then, do I care about dealing with the issues of what went wrong, why, and how to prevent it in the future. While having an ETA would be wonderful, sometimes you just have no idea exactly how long things will take, and I am sure hopeful the repair engineers are not on dinner break while we sit here hitting refresh all night. JMHO, -Dave
Posted by bteeter, 12-05-2006, 07:47 PM
Look where Scranton, PA is on a map. If you've ever driven through there, and I have - guess what - there is NOTHING around. All the other NOC's are in major cities with multiple options for connections. Burst doesn't have that, so they go with the best they can. Unfortunately, they keep getting screwed over by problems that affect their only or primary carrier. I'm sure they are doing all the can. I'm glad I only host for pleasure now and not for a business. I can remember how my clients used to be during outages. It was never fun... Take care, Brian
Posted by RobM, 12-05-2006, 07:50 PM
yep good time to do some server upgrades as there down already .
Posted by geoapps, 12-05-2006, 07:51 PM
Or else the power outage took out power to redundant equipment in two separate-but-physically-close locations. According to BurstNet they allegedly had two separate paths and fully redundant circuits in different locations, at least that's what I read into the messages that say what they were buying from CTSI.
Posted by dgynepa2, 12-05-2006, 07:52 PM
oh.. i thought u meant we would all meet in Scranton and go drinking!
Posted by jbiquez, 12-05-2006, 07:53 PM
Hello all. My server has been un reachable since 8 hours ago. All my clients are VERY angry witha reason for sure. My server is with a company that has service with NOCSTER and they told me there is a problem with a fiber cut and they estimate at least 3 more hours. All business hours are gone. Do others that have servers with Nocster are having the same problem? Thanks for the comments
Posted by examnsz, 12-05-2006, 07:58 PM
hi , I just want to know when the problem will be solved
Posted by shade88, 12-05-2006, 08:01 PM
Search for nocster. a big thread about burstnet outage. www dot webhostingtalk dot com/showthread.php?t=567387 cant post live link yet
Posted by glace, 12-05-2006, 08:01 PM
As far as I understand it they will start working to fix it in one hour from now so I guess 5 hours if we are lucky.
Posted by glace, 12-05-2006, 08:03 PM
Man I can't believe my eyes !!! They are back online !
Posted by Sphosting.com, 12-05-2006, 08:04 PM
all my servers are back up
Posted by hostrequest, 12-05-2006, 08:05 PM
I have connection to my server... HHHHOOOORRRAAAAY
Posted by server4sale, 12-05-2006, 08:05 PM
Not a bad idea
Posted by geoapps, 12-05-2006, 08:07 PM
My websites are back online at 17:06 MST -- 19:06 Philadelphia time
Posted by flashware, 12-05-2006, 08:09 PM
I hear you loud and clear...! I'm over in Perth, so the business day is only just beginning here... Again, its going to be a loooooooong day
Posted by sebby, 12-05-2006, 08:09 PM
Hello all- what about emails that we sent to our servers all day. Were they queued somewhere or returned back to senders (as all my DNS were down all day...)? Thanks, Seb
Posted by CNnames, 12-05-2006, 08:10 PM
Fortunally we were prepared for a situation like this and have a failover scenario in place for our sites so -AT LEAST- we can communicate with our customers and inform them about what's happening. Why does burst not have a failover solution for their websites? (THIS SO YOU CAN INFORM YOUR CUSTOMERS) It's unreasonable that we have to search on worldwide forums to get the information we NEED so we can inform our customers. It will cost you a few $ to have a failover server at a different datacenter a failover DNS solution at fx. dnsmadeeasy.com and a mysql replication script running. With a simple solution like this we will be able to reduce the damage by keeping our customers informed.
Posted by fleed, 12-05-2006, 08:11 PM
It's back up!!! Yes!!!
Posted by geoapps, 12-05-2006, 08:11 PM
E-mails should be re-sent for up to 5 days by 'proper' mail servers, but I've heard that some ISPs (e.g. AOL, MSN) violate standards and don't re-send as they should.
Posted by HostJedi, 12-05-2006, 08:11 PM
The fiber cut has not been repaired, they have not even started yet most likely. We managed to get CTSI to re-route our circuit around the fiber cut, to get things back online for our clients. I'm sure they realized the seriousness of the situation with the redundancy failure, and figured it was in their best interest to mitigate the downtime. I just thank you all for you patience, understanding, and support during the issue---much appreciated. I'm not sure if there will be any minor outages when things return back to normal or redundancy is fixed, but we will keep you all posted as things progress. Our immediate concern was getting this service restored as quickly as possible, without waiting for fiber splicing, which has now been accomplished. more information will be provided as made available to us... . .
Posted by TCP/IP Warrior, 12-05-2006, 08:11 PM
I feel for all of you who are going through this. Just to let you know... you are now accessible from Florida USA as well Good luck!
Posted by flashware, 12-05-2006, 08:13 PM
I take that back - we're back online! Awesome! Hopefully it will stay up though... we didn't loser power, server is showing 31 day uptime, so just network loss..
Posted by cianuro, 12-05-2006, 08:13 PM
Yeay! Looking forward to the report and your public *** whooping of CTSI!
Posted by flashware, 12-05-2006, 08:15 PM
Thanks, it's great to see that a workaround has been able to be put in place.
Posted by flashware, 12-05-2006, 08:22 PM
Awesome - mail from the past few hours is starting to trickle in now.
Posted by Arsalan, 12-05-2006, 08:23 PM
Thanks a bunch for your efforts towards getting service restored!
Posted by sebby, 12-05-2006, 08:24 PM
Thanks geoapps. It was my understanding that when both primary and secondary DNS were unreachable, MTAs were interrupting the sending process and returning mail back to senders (i.e. mail was not queued)? Is there something I missed? seb
Posted by Cirtex, 12-05-2006, 08:33 PM
Thanks Shawn, Be sure to update us regarding CTSI Issue tomorrow and how it goes with them Just send an email over. Cheers!
Posted by glace, 12-05-2006, 08:48 PM
Paypal doesn't give a damn either. If your mailserver is down you'll lose any Paypal email so check it for orders manually.
Posted by Acossta, 12-05-2006, 09:07 PM
Back online Movilpoint.net Latin America, Thanks for your work Burst!!
Posted by RicardoG, 12-05-2006, 09:20 PM
It's great that they are back online but I still would like to know if there will be any compensation. At least it should be logical that we do not need to pay for this day. As my server is not for commercial use it didn't to much damage to me but I guess that others who use the servers for business purposes this was very damaging and cost them a lot of money. Anyway anyone know what for a compensation they will give etc? Thanks
Posted by bcs1a, 12-05-2006, 09:48 PM
Well Guys, all i have to say is "Thanks to the Information you've provided to me, not only here in this forum, but during my Phone call to your office earlier, We here at the company i work for, have a factual (up to date) Notice in our network status page for our clients, and have an alternate server available for our clients to use" this would not have been possible if you guys had shirked any of this issue and not responded to Phone calls and made the postings here (while at the same time trying to deal with your providers to get answers and updates) So while there may be a bunch of people "busting your chops" so to speak, I (and the company i work for) are thankful for the information. By your taking the time to give it to us, and make it (updates) available here, we have been able to kick in our own redundancy stratagy, and i currently have NO tickets in my support queue regarding this issue from my clients. Thank you Bill Darkstar Communications
Posted by hostpc.com, 12-05-2006, 11:37 PM
Glad to see everything is back online. Hopefully they can do the fiber repairs overnight and do so without any other downtime. Tomorrow, time to investigate options, ask questions and hopefully get more answers. Personally, I'd like to applaud Shawn and crew for keeping us updated here. I'll be contacting you personally regarding another matter. Joe
Posted by sebby, 12-05-2006, 11:39 PM
I agree with bcs1a. I called three times today (before I found this thread!) and Burst's staff was taking all calls (even after 6pm). I prefer to hear bad news than no news at all! However, as suggested by someone else earlier in this thread, I would recommend the 'worst case scenario' to be better organized : Burst should have a remote server in another DC to provide updates during such crisis. Moreover, Burst should have a copy of its own client database to email us with updates on the situation (not all Burst clients have a third party watching the uptime and it is never a good thing when you learn about this situation by your own clients....) Hope this is all behind us, seb
Posted by hostpc.com, 12-05-2006, 11:43 PM
I just emailed Burst with just such a plan. I hope to hear back from them soon so we can all move forward.
Posted by sebby, 12-05-2006, 11:52 PM
What about a RSS feed that we could display on our own 'Plan B's for the benefit of our clients... seb
Posted by RobM, 12-06-2006, 12:00 AM
They already know about this and the word is it can be a security hole and I agree that it does leave a security hole being a big company like them with customer info housed in another DC Now I don't know if they have a place down in philly that is secure where they have some gear, as thats the only place I can think of if they decide to do it The most secure place other then there DC I can think of now. Now they did do there best today and communication was super and let's all hope if anything like this happens again they do the same with communication. I mean what DC do you know of that will allow you access to your gear during a time they had, none that I can think of or been with, most say not right now too busy
Posted by ultra.net, 12-06-2006, 12:11 AM
Indeed. One reason I have reseller accounts with multiple services, some cheap and nasty, but useful as secondary MX handlers, and to put some websites on *in an emergency*. It is 4am here but if I had not seen sites up again, I've still time enough to have everything running for my clients from other servers before start of business over here. Thing is, sometimes a client cannot access a particular site (their own!) because of routing problems which are between them and the server... nothing I can do about that, except use it as an example of how the net works 99.9% of the time, and give them advice on chasing their own ISP, but with another server or two, then it is at least possible to use alternative hosting they can reach/view. Same for DNS... Paid use of ZoneEdit but also using a second service "just in case". Well put, Dave, having worked in a service department 15+ years ago, always the hardest question is "when will it be back up" (multi-user VAX/VMS cluster, approx 400 connections in the machine room, with some 1000+ remote terminals - so there were always queues of users waiting for a free port, as various terminal rooms had ~250 ports reserved in advance). I think anyone expecting Burst.net to be able to give an answer when they (B.net) are not getting any firm info is heading for a coronary... I took a break, and came back with everything OK.
Posted by GordonH, 12-06-2006, 04:10 AM
Can I just say that after 6/7 years in this industry I have become much more relaxed about these sort of issues. Years ago I would have been climbing the walls, but all this was going on at around 10pm our time. I just prayed about it and went to bed. An hour or so later it was all working and our inbox is actually not full of complaints this morning. We had emailed customers and were updating a blog page on the issue until about 10:30pm, plus our tech support were on duty during the night. There was not any more we could do. In the end these things always get fixed because the telecom companies know they will lose business if they don't do them quickly. The worst aspect of this was it stopped a server move we were doing between boxes at Burst (for about the fifth time). That job is jinxed.
Posted by bhaputi, 12-06-2006, 12:33 PM
http://forums.burst.net/showthread.p...3966#post23966
Posted by geoapps, 12-06-2006, 01:20 PM
Do you have a link to the RFC where this is covered?
Posted by BurstMikeM, 12-06-2006, 01:58 PM
http://tools.ietf.org/html/rfc1123 5.3.1.1 Sending Strategy The general model of a sender-SMTP is one or more processes that periodically attempt to transmit outgoing mail. In a typical system, the program that composes a message has some method for requesting immediate attention for a new piece of outgoing mail, while mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender. A mail queue entry will include not only the message itself but also the envelope information. The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the sender-SMTP can determine the reason for non- delivery. Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days. The parameters to the retry algorithm MUST be configurable. A sender SHOULD keep a list of hosts it cannot reach and corresponding timeouts, rather than just retrying queued mail items.
Posted by sysc, 12-06-2006, 04:30 PM
10hrs of downtime and this is the reply you get when you request a credit Hello, I'm sorry to say but due to our SLA policy, credit will not be provided as indicated on our site for the following reasons. You may view the following at: http://www.nocster.com/policy/sla.shtml
Posted by sysc, 12-06-2006, 04:51 PM
I should have added this part as well.. BurstNET plans on pursuing alternative redundancy scenarios within our local area at no additional cost to our clientele. This will ensure network stability and avoid future outages from reoccurring. And he did paste me the entire SLA.
Posted by bhaputi, 12-06-2006, 04:58 PM
sysc, I am glad to see you tack on that second note, as you were definitely given more information than you originally posted.
Posted by HostJedi, 12-06-2006, 05:04 PM
BTW, until now there has not been sufficient capacity in our city to service our network, and hence our need to backhaul everything to Philadelphia. Most of the backbones in this area operate on a single OC12 circuit, which is not enough for us whatsoever. Our own network usage is larger than a few of these backbones combined actually. We would have to combine multiple carriers with multiple transport options, and it would not be cost effective as a budget hosting provider for a redundancy solution, and would not have been able to be implemented without signifigant price increases to our client base. We had opted therefore for a completely redundant path with CTSI, which was supposed to operate properly, while we awaited a local high-capacity transit provider to come into town. We are currently working on a redundancy solution with a carrier that has just recently moved into the region by taking over another network, that should have the capacity we require at a reasonable price level. This is something we have been waiting for at our company quite some time. We expect to be able to implement this without any additional expense to our client base, which is important to us, especially considering the outage just experienced. Further information on this will be provided once we have details available. . . Last edited by HostJedi; 12-06-2006 at 05:13 PM.
Posted by bhaputi, 12-06-2006, 05:22 PM
Just an update, which is that there is not yet an update. We were told this morning that many conference calls and meetings were going on, and I would be notified when they were done. I was told that again later today. The meetings/calls are still going on, so hopefully we will have more info on what happened either later tonight or tomorrow.
Posted by RossH, 12-07-2006, 04:37 PM
This literally almost made me spit coke on my laptop
Posted by RossH, 12-09-2006, 06:14 PM
I just also wanted to chime in here and say that this downtime was very unfortunant. Should Burst have used a different company for their backup fiber, yes. However during this entire downtime they were very reponsive on here and to all customers and I have to say that impressed the hell out of me. Kudos to Shawn and team for showing they truly care about their customers...