Wholesale Internet down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Wholesale Internet down?

Posted by hD-Scott, 08-06-2013, 02:07 PM
Both my servers at WSI are down. Anyone know if this is a systemwide problem. Unable to access my dashboard as well.

Anyone else having issues?

Thanks!

Scott
Posted by ryguy222, 08-06-2013, 02:10 PM
All my colo, and all my dedicated servers at Datashack are down too even though they are spread through different facilities. Although I was able to login to the dashboard and put in a ticket.

Datashack gets its bandwidth from Wholesale, so I'm guessing wholesale is having a power outage or just a network outage.
Posted by 0x3g3n, 08-06-2013, 02:11 PM
Mine is down as well. I was able to get a emergency ticket in before the customer panel crapped out.
Posted by Justin, 08-06-2013, 02:12 PM
I'm down as well there. Can't get into their ubersmith panel either. Let's hope this is short lived...
Posted by brianz, 08-06-2013, 02:12 PM
Both of my servers at Wholesale and datashack are down too. It seems like the whole datacenter is down
Posted by shnoc, 08-06-2013, 02:14 PM
My server is also offline more than 25 minutes
Posted by rveram, 08-06-2013, 02:17 PM
Mine is also offline. I had a lot of trouble creating the support ticket, I got and DB Connect timeout error.
Posted by microz, 08-06-2013, 02:17 PM
same here, from 30min
Posted by NE-Andy, 08-06-2013, 02:18 PM
My at DataShack is also down since 10:20 PST or so, so it is almost an hour now.
Posted by John0092Ojard, 08-06-2013, 02:21 PM
Down here as well. Managed to submit a ticket, thought the server just freezed and then the support board went down again. I guess we will just need to wait

Got a response:

Quote:

We are aware of an issue with our network at this time.
We are working to resolve the issue now.

So be patient guys
Posted by iPole, 08-06-2013, 02:22 PM
All my servers are down as well. I submitted a 911 ticket 25 minutes back, but their usually quick response is missing this time.

Hope they atleast update the status here.
Posted by ryguy222, 08-06-2013, 02:23 PM
Got a reply from support:

"We are aware of an issue with our network at this time.
We are working now to resolve the issue.
Once resolved your server should return to normal."
Posted by iPole, 08-06-2013, 02:26 PM
Oh well, I just got a reply to the ticket:

Quote:

We are aware of an issue with our network at this time.
We are working to resolve the issue now.

--
Thank you,
Spencer
Tech Support

I made the mistake of moving even my primary nameserver to WI network, and now all my customer domains have stopped resolving.
Posted by JPerkster, 08-06-2013, 02:29 PM
My dedi's at DS are down as well. I tried to put in a ticket about 30 minutes ago, but I got DB errors from the client portal. Glad to hear it's not just me
Posted by Justin, 08-06-2013, 02:39 PM
Appears their support panel is at least somewhat working now. I was able to place in ticket without getting the critical db failure error at this point. (Good news for anyone attempting to get in touch with support since their network is having problems).
Posted by pechspilz, 08-06-2013, 02:48 PM
DOH, fiber cut! That might take a while.
Posted by spencerocks, 08-06-2013, 02:53 PM

Quote:

Originally Posted by pechspilz

DOH, fiber cut! That might take a while.

Why do you think that
Posted by pechspilz, 08-06-2013, 02:55 PM
Because I wasn't given an ETA and splicing fibers is not that easy?
Posted by spencerocks, 08-06-2013, 02:55 PM

Quote:

Originally Posted by pechspilz

Because I wasn't given an ETA and splicing fibers is not that easy?

Sorry, I mean how do you know it was a fiber cut?
Posted by pechspilz, 08-06-2013, 02:57 PM
Ticket response. I have servers both at WII & The Shack.
Posted by RSNET-John, 08-06-2013, 02:59 PM
I would recommend not putting in tickets. They are aware of the issue and most likely just responding to tickets with canned responses, which is most likely utilizing valuable tech time.

Good luck to WSI/WII and DS, hopefully they get everything back up quickly!
Posted by pechspilz, 08-06-2013, 03:00 PM
By the way, did anyone experience network related problems yesterday at the same time? I had like 90% packet loss from/to one specific server and around 30% to other servers on a different prefix.
Posted by codatory, 08-06-2013, 03:02 PM
This is one of those cases where an official status page comes in handy. Or a twitter account. Or anything really. Since their website was up, and I don't know of any other servers around there I had to put in a ticket and got that fairly unhelpful reply, and if I want a status update I'm stuck opening another ticket (because they immediately closed my original ticket).
Posted by pechspilz, 08-06-2013, 03:03 PM

Quote:

Originally Posted by RSNET-John

I would recommend not putting in tickets. They are aware of the issue and most likely just responding to tickets with canned responses, which is most likely utilizing valuable tech time.

Good luck to WSI/WII and DS, hopefully they get everything back up quickly!

You're right, I was thinking about this myself when I put in the ticket. But OTOH they have this announcement area in the client dashboard would save them some time in such situations.
Posted by pechspilz, 08-06-2013, 03:04 PM

Quote:

Originally Posted by alxconn

This is one of those cases where an official status page comes in handy. Or a twitter account. Or anything really. Since their website was up, and I don't know of any other servers around there I had to put in a ticket and got that fairly unhelpful reply, and if I want a status update I'm stuck opening another ticket (because they immediately closed my original ticket).

I think it gets reopened once you log an answer to a closed ticket.
Posted by bdwebservices, 08-06-2013, 03:05 PM
my server is down for 1+ hrs and counting
Posted by HocusPocus, 08-06-2013, 03:10 PM
Same with both my servers. Yesterday they also had issues (around the same timeframe), my servers had 99% packet loss for approx 30mins
Posted by pioneernetworks, 08-06-2013, 03:14 PM
I was just told by DataShack there is a fiber cut that is causing the network issues. Although I think its funny if you read this on datashacks website

Quote:

All DataShack facilities are located on a private fiber ring connecting them to downtown Kansas City. This ring is self healing and includes fiber on diverse river crossings. In the event where the fiber is not owned by us, we use fiber from multiple providers.

Posted by httpEasy, 08-06-2013, 03:18 PM

Quote:

Originally Posted by bdwebservices

my server is down for 1+ hrs and counting

Same here. Just put in a ticket before looking here.
Posted by zacharooni, 08-06-2013, 03:18 PM
My server at WSI is down, but I can still get into the dashboard. For now, I will let them be, nothing mission critical, but I know they probably have a lot of people unaware of this thread. Hurry back!
Posted by Holoshed, 08-06-2013, 03:19 PM
I also received a response saying a fiber was cut and there is no ETA. It is the no ETA part I hate since it leaves you wondering how far along they may even be at this point with fixing it.
Posted by arthur8, 08-06-2013, 03:20 PM
Still offline..
Posted by lfxx, 08-06-2013, 03:26 PM
such threads are a lot more useful than my ping monitor

but oh well, a status page would be even better
Posted by httpEasy, 08-06-2013, 03:37 PM
Canned or not, I received a response to my ticket within minutes, confirming the fiber cut (externally).
Posted by RSNET-John, 08-06-2013, 03:40 PM

Quote:

Originally Posted by httpEasy

I received a response to my ticket within minutes, confirming the fiber cut (externally).

Communication is the best thing you can have in these sort of situations. Kudos to them.
Posted by chaudharymic, 08-06-2013, 03:46 PM

Quote:

Originally Posted by pechspilz

By the way, did anyone experience network related problems yesterday at the same time? I had like 90% packet loss from/to one specific server and around 30% to other servers on a different prefix.

Yup, 90%+ loss to a sever there for about 30 min or so sometime in the last few days.
Posted by ramnet, 08-06-2013, 03:48 PM
My colo at WSI in oak tower hasn't had any outages today.

Must be limited to their north kansas city location?
Posted by WII-Aaron, 08-06-2013, 03:50 PM
There is a fiber cut. So far we haven't been able to give an eta because we don't have one ourselves. We have some other dark fiber we can reroute to but we're not sure that will be any quicker than just waiting for them to resplice it.

This was an AT&T crew that hit it so there are supervisors and supervisor supervisors here. Ugh...

We're working as fast as we can and trying to get them to do the same.
Posted by Techno, 08-06-2013, 03:51 PM

Quote:

Originally Posted by ramnet

My colo at WSI in oak tower hasn't had any outages today.

Must be limited to their north kansas city location?

My Datashack dedicated is in Oak Tower - no downtime.
Posted by rveram, 08-06-2013, 03:51 PM
Thank you for the update Aaron, hope you're able to restore everything promptly.
Posted by duran2003, 08-06-2013, 03:53 PM
Yes, same here:

Quote:

We are aware of the issue and are working as fast as we possibly can to get the
network issue resolved.
When the issue is resolved your server should return to normal.
We experienced a fiber cut and currently don't have an ETA on when it will be
fixed, sorry for the inconvenience.

This is my first "fiber cut" experience using DC in EUA.
Here in Brazil we have this problem every week.

well... we will see how many time this is take...
Posted by codatory, 08-06-2013, 03:57 PM
So what you're saying is that your datacenter does not have redundant fiber?
Posted by XanCharb2, 08-06-2013, 03:59 PM
From wholesaleinternet.net/our-facility/ : (I couldn't make that a link because I don't yet have 5 posts in this account, and I can't get to my old one because my email lives on a server at Wholesale!)

"Wholesale internet is focused on providing highly reliable, ultra low latency network connectivity, suitable for any application. Our datacenter is fed by redundant fiber feeds from Hurricane Electric, Cogent Communications, and the Kansas City Internet Exchange (KCIX). We utilize Foundry routers and switches in a parallel redundant design, eliminating any one point of failure."

How can this be true if a single fiber cut causes such damage?
Posted by RSNET-John, 08-06-2013, 04:08 PM

Quote:

Originally Posted by XanCharb2

How can this be true if a single fiber cut causes such damage?

They most likely have the redundant lines coming to their main facility (Oak Tower?) and then feeding the other facility from there via a dark fiber loop.
Posted by XanCharb2, 08-06-2013, 04:10 PM

Quote:

Originally Posted by RSNET-John

They most likely have the redundant lines coming to their main facility (Oak Tower?) and then feeding the other facility from there via a dark fiber loop.

Could be. If that's the case, then they shouldn't sell servers in the other facility with the "no single point of failure" line.
Posted by duran2003, 08-06-2013, 04:12 PM

Quote:

Originally Posted by Techno

My Datashack dedicated is in Oak Tower - no downtime.

Here all my Datashack dedicated servers are down.
Posted by Microlinux, 08-06-2013, 04:25 PM

Quote:

Originally Posted by XanCharb2

How can this be true if a single fiber cut causes such damage?

Redundancy does not imply diversity. No idea if that is the case here, just an example.
Posted by XanCharb2, 08-06-2013, 04:32 PM

Quote:

Originally Posted by [CTI] Todd

Redundancy does not imply diversity. No idea if that is the case here, just an example.

hmm, that sounds pretty dubious to me. Then what about the phrase "eliminating any one point of failure"? A single fiber cut has caused an enormous failure. Sounds like they didn't actually get, you know, ALL the single points of failure taken care of.
Posted by kirkhilles, 08-06-2013, 04:48 PM
Yeah, it sucks, but this sounds like this is a very rare occurrence. Its not like a lot of cheap providers that I've used in the past that had power strip issues, really old hardware and so premature hard drive failures, etc. Not to mention all of those "cloud" solutions where there seemed to be something new every month where they'd have to replace this or that and had hours of downtime.

Don't get me wrong, I'm not letting DS off the hook, but I'm certainly willing to give them the doubt (although a one time discount might be nice) as I can't recall a single issue I've had with the 7 months of uptime from my first server and 4.5 months on the second server.

How about you guys? Good uptime?
Posted by WII-Aaron, 08-06-2013, 04:49 PM
Well, first off, we are not completely down. Our NKC facility is off line which comprises less than half of our total server count.

Second, both lines that head to NKC cross each other in the street right outside Oak. Both conduits were cut in half by this giant man hole cutting drill.

The solution is to move the second leg of our NKC run onto the fiber we currently use for cogent and our cogent onto one leg of the fiber we go to NKC on. That conduit is on the other side of the street and was not hit. So that's the plan to prevent this in the future.

As for an update: the splicing crews are pulling new fiber now, then they'll start splicing. Hopefully we'll be back in 1-2 hours.
Posted by XanCharb2, 08-06-2013, 04:52 PM
Aaron,

Thanks for the explanation and update. That makes me feel less like going totally berserk. :-)
Posted by ZacUSNYR, 08-06-2013, 05:13 PM
Chit happens, it's all about how it's communicated and handled. You guys are doing a great job at that, so keep it up and here's to hoping we come back soon.
Posted by Phiberop, 08-06-2013, 05:15 PM
Sounds like it's all under control then, good news. I have three servers through DataShack and really can't complain. This is the first unscheduled outage since I acquired my first server sometime back in November 2012. Support is always great and the price can't be beat for what you get.

I always keep a variety of options at my disposal. Non-mission critical sites are very well suited to places like DataShack where you can save a lot of money but perhaps don't get the same level of reliability as you might expect elsewhere for two or three times the bill (if not more).
Posted by Dedispec, 08-06-2013, 05:24 PM
Thanks for the update Aaron - definitely unfortunate but sounds like everything that can be done is being done quickly.
Posted by Microlinux, 08-06-2013, 05:26 PM

Quote:

Originally Posted by XanCharb2

hmm, that sounds pretty dubious to me. Then what about the phrase "eliminating any one point of failure"? A single fiber cut has caused an enormous failure. Sounds like they didn't actually get, you know, ALL the single points of failure taken care of.

What I said was not in direct reference to WSI's fiber issue, but the matter of "single points of failure" is really a subjective determination.

I've been in the same situation as them, it's out of their hands at this point -- and that can be frustrating.
Posted by WojonsTech, 08-06-2013, 05:26 PM
The only thing that would have been nice if they emailed us to let us know that the fiber got cut. For me I woke up to my server not being online. I know werid unproductable issues liket his happen no matter what provider you use but it would always be nice for providers to test the unpredicable
Posted by duran2003, 08-06-2013, 05:41 PM

Quote:

Originally Posted by WII-Aaron

Well, first off, we are not completely down. Our NKC facility is off line which comprises less than half of our total server count.

Second, both lines that head to NKC cross each other in the street right outside Oak. Both conduits were cut in half by this giant man hole cutting drill.

The solution is to move the second leg of our NKC run onto the fiber we currently use for cogent and our cogent onto one leg of the fiber we go to NKC on. That conduit is on the other side of the street and was not hit. So that's the plan to prevent this in the future.

As for an update: the splicing crews are pulling new fiber now, then they'll start splicing. Hopefully we'll be back in 1-2 hours.

Thanks for the update Aaron.
Posted by rbstern, 08-06-2013, 05:52 PM
Aaron, thanks for staying on top of it.

And give our best regards to the AT&T crew who cut the fiber. "Call before you dig!"
Posted by CBH-Nick, 08-06-2013, 06:06 PM
Mine is also down. Let's hope they get this cable plugged back in soon.
Posted by WojonsTech, 08-06-2013, 06:07 PM

Quote:

Originally Posted by rbstern

Aaron, thanks for staying on top of it.

And give our best regards to the AT&T crew who cut the fiber. "Call before you dig!"

And at least check city plans and records to see if there is a fiber line on the street ur working on.
Posted by WII-Aaron, 08-06-2013, 06:15 PM
Well there seems to be a dispute about weather it was properly located or not. I don't know. I just told them to fix it and fight about it later. There are 3 crews working on it right now. (1 on each hand hole and one where the cut happened.) They've pulled the one leg and are splicing it. There was an issue with blockage in the conduit on the other leg. They've cleared it and are trying to catch up.

I'm pretty much just standing there trying to look as annoyed as I can so they hurry up. This is my first fiber cut.
Posted by WII-Aaron, 08-06-2013, 06:17 PM
On a related note: We have backup lines into every facility for OOB network management. The tech staff is on these answering support tickets. I'm at the office at Oak which is unaffected so if you have questions your support tickets are still being answered.
Posted by lfxx, 08-06-2013, 06:18 PM

Quote:

Originally Posted by kirkhilles

How about you guys? Good uptime?

Yes, first downtime in 4 months or so. I thought that uptime robot was not working since it didn't report any WSI downtime but it was just fine.

Aaron, thanks for the updates.
Posted by WojonsTech, 08-06-2013, 06:35 PM

Quote:

Originally Posted by WII-Aaron

Well there seems to be a dispute about weather it was properly located or not. I don't know. I just told them to fix it and fight about it later. There are 3 crews working on it right now. (1 on each hand hole and one where the cut happened.) They've pulled the one leg and are splicing it. There was an issue with blockage in the conduit on the other leg. They've cleared it and are trying to catch up.

I'm pretty much just standing there trying to look as annoyed as I can so they hurry up. This is my first fiber cut.

Yeah i hate when stuff like that happens just digg up all the paper work you have and who installed it and that they are lisnces by city and state to be doing that and if they forgot a step or what ever you can drag them with u
Posted by WII-Aaron, 08-06-2013, 06:39 PM
So apparently they didn't have it cleared and they're opening up another section of the street to get to the blockage....

Wojons: this fiber belongs to the city.
Posted by Servode, 08-06-2013, 06:45 PM
Im a fiber optic splicer my self hopefully its a small fiber cable and not like a 428 or its going to be down a while
Posted by lifeisjustabout, 08-06-2013, 06:49 PM
thanks Aaron for update

Quote:

Originally Posted by WII-Aaron

The solution is to move the second leg of our NKC run onto the fiber we currently use for cogent and our cogent onto one leg of the fiber we go to NKC on. That conduit is on the other side of the street and was not hit. So that's the plan to prevent this in the future.

as far as my understanding if we have that redundant fiber in the future lies to facility we will not be affected for next stupid digging attempt am I right ? and if so is there any ETA for this redundant fiber i am more interested with this part rather than fixing this issue actually.
Posted by WII-Aaron, 08-06-2013, 07:07 PM

Quote:

Originally Posted by lifeisjustabout

thanks Aaron for update

as far as my understanding if we have that redundant fiber in the future lies to facility we will not be affected for next stupid digging attempt am I right ? and if so is there any ETA for this redundant fiber i am more interested with this part rather than fixing this issue actually.

This took out 2 lines. We'll move one to the other side of the street so they'll be harder to hit at once.
Posted by 8inghost, 08-06-2013, 07:12 PM

Quote:

Originally Posted by WojonsTech

The only thing that would have been nice if they emailed us to let us know that the fiber got cut. For me I woke up to my server not being online. I know werid unproductable issues liket his happen no matter what provider you use but it would always be nice for providers to test the unpredicable

I got an email. Not from wholesaleinternet, but from a third party monitor. Shortly after a bunch of tickets and phone calls came pouring in.

Thank you Aaron for the updates and having techs answer tickets so fast. I was freaking out thinking that my server got hacked or hard drive broke or KC exploded or something and it was nice to know so quickly that it is simply nothing more than an inconvenience and that everything will be fine once the network is brought back up.
Posted by nutN2Lewz, 08-06-2013, 07:14 PM

Quote:

Originally Posted by WII-Aaron

There was an issue with blockage in the conduit on the other leg.

I hate it when that happens!!!

I hope you weren't planning on going to the Royals/Twins game tonight.

Barry
Posted by Justinwiz, 08-06-2013, 07:19 PM

Quote:

Originally Posted by WII-Aaron

This took out 2 lines. We'll move one to the other side of the street so they'll be harder to hit at once.

Thanks for the updates Aaron.

Post some pictures
Posted by icedemo, 08-06-2013, 07:40 PM
still down for the past 5 hours.
Posted by httpCORE, 08-06-2013, 08:37 PM
Damn still down...
Posted by WII-Aaron, 08-06-2013, 08:38 PM
Should be back very shortly now.
Posted by 8inghost, 08-06-2013, 08:43 PM
Yeah.. This outage is going on 7 hours. Lost a big account I just sold two weeks ago due to this. Sometimes I guess looking in that mirror screaming, WHY!?!? of GAWD WHY?!? helps. But not this time. lol

I'm pretty sure that AT&T will hopefully get it together soon. I've spliced fiber before too but never underground. It's much nicer to do in a building with almost zero safety gear required.
Posted by osgregs, 08-06-2013, 08:43 PM

Quote:

Originally Posted by WII-Aaron

Should be back very shortly now.

cables have been repaired?
Posted by arthur8, 08-06-2013, 08:49 PM
That is not good. Still offline.
Posted by omar_ramon, 08-06-2013, 08:50 PM
Thanks Aaron, communication really makes a difference. Today we had a crazy day, tons of calls, fights, etc.

I really want this day to END
Posted by JordanAB, 08-06-2013, 08:56 PM
As many have already said, thanks for the communication.

Although you are probably busy, I do have a small question.
Do you plan to offer any compensation after all the legal stuff is sorted with AT&T?
Posted by Servode, 08-06-2013, 08:57 PM

Quote:

Originally Posted by 8inghost

Yeah.. This outage is going on 7 hours. Lost a big account I just sold two weeks ago due to this. Sometimes I guess looking in that mirror screaming, WHY!?!? of GAWD WHY?!? helps. But not this time. lol

I'm pretty sure that AT&T will hopefully get it together soon. I've spliced fiber before too but never underground. It's much nicer to do in a building with almost zero safety gear required.

Lol im a fiber optic splicer for ATT most of the work we do is under ground, You would be surprised how many telecom companies, cable companies, until companies get crammed in one man whole that is 40 ft deep.

From my experience here in Dallas/Ft Worth anyway.

Fiber repairs can be costly, And take for ever i do a lot of work in infomart too, Its getting better though most fiber cables don't have icky pick in them now days.
Posted by Dedispec, 08-06-2013, 08:57 PM

Quote:

Originally Posted by omar_ramon

Thanks Aaron, communication really makes a difference. Today we had a crazy day, tons of calls, fights, etc.

I really want this day to END

I can only imagine the day Aaron and his staff are having
Posted by webheaded, 08-06-2013, 08:58 PM
Thank you Aaron for keeping in touch with us. I've been monitoring this thread and thought I'd throw in my thanks as well. Been down for most of the day now, hoping they finish up soon.

Stupid AT&T. Screwing things up even when I'm not their customer. :p
Posted by HostHoarders-Mike, 08-06-2013, 09:05 PM

Quote:

Originally Posted by webheaded

Thank you Aaron for keeping in touch with us. I've been monitoring this thread and thought I'd throw in my thanks as well. Been down for most of the day now, hoping they finish up soon.

Stupid AT&T. Screwing things up even when I'm not their customer. :p

That's part of the reason I stay away from anything AT&T. I have never liked their company and now I see them interfering with another business. I really do hope WSI is compensated by AT&T as this is beyond ridiculous. All of our servers in Kansas City are offline from the downtime but we are being patient as we know this is in no way WSI's fault. Thanks again for all the hard work Aaron and WSI's staff!
Posted by webheaded, 08-06-2013, 09:10 PM

Quote:

Originally Posted by HostHoarders-Mike

That's part of the reason I stay away from anything AT&T. I have never liked their company and now I see them interfering with another business. I really do hope WSI is compensated by AT&T as this is beyond ridiculous. All of our servers in Kansas City are offline from the downtime but we are being patient as we know this is in no way WSI's fault. Thanks again for all the hard work Aaron and WSI's staff!

I thought this was ironic considering I had just been telling one of my coworkers that when I used to work with the Canadians that they had outages all the time and it was always a fiber cut and I remember thinking "wtf, do you guys not mark these lines or what?" Now here I am. x)

AT&T is a baby bell. THE CANADIANS ARE CONSPIRING AGAINST US!!
Posted by chasingsol, 08-06-2013, 09:15 PM
Glad to have found this thread. I opened a support ticket at Datashack some hours ago, it was closed with no response. Reopened it, closed again with no response.

Good to have information, although unceremoniously closing tickets isn't very nice.
Posted by Microlinux, 08-06-2013, 09:17 PM

Quote:

Originally Posted by HostHoarders-Mike

That's part of the reason I stay away from anything AT&T.

It could (and does) happen to anyone.
Posted by webheaded, 08-06-2013, 09:18 PM

Quote:

Originally Posted by chasingsol

Glad to have found this thread. I opened a support ticket at Datashack some hours ago, it was closed with no response. Reopened it, closed again with no response.

Good to have information, although unceremoniously closing tickets isn't very nice.

I'm with Datashack as well. The support area was down earlier and then it just randomly came back up. I had emailed the support address and gotten nothing but they actually replied very quickly to the ticket I put in. It is times like this that I wish hosts would just have a status page so we didn't have to flood their support queues with questions during an outage.
Posted by icedemo, 08-06-2013, 09:19 PM

Quote:

Originally Posted by webheaded

I'm with Datashack as well. The support area was down earlier and then it just randomly came back up. I had emailed the support address and gotten nothing but they actually replied very quickly to the ticket I put in. It is times like this that I wish hosts would just have a status page so we didn't have to flood their support queues with questions during an outage.

well at least they bother to respond to you. i have send in 2 tickets and none have any response from them till i bother to do some proper searching on my own.
Posted by webheaded, 08-06-2013, 09:21 PM

Quote:

Originally Posted by icedemo

well at least they bother to respond to you. i have send in 2 tickets and none have any response from them till i bother to do some proper searching on my own.

Maybe they're all outside right now supervising the line being fixed.
Posted by HostHoarders-Mike, 08-06-2013, 09:23 PM

Quote:

Originally Posted by webheaded

Maybe they're all outside right now supervising the line being fixed.

Is it mildly strange that I want to see pictures? I have it played out in my head as a pretty interesting sight from the details Aaron has given.
Posted by TupaHost, 08-06-2013, 09:24 PM
Anyone know how long it will take?
Posted by flyah, 08-06-2013, 09:25 PM
Woke up this morning to find out my server is down.

Well, If I'm in Kansas, I'll go buy you guys pizza. hehe
Posted by dlabrie, 08-06-2013, 09:25 PM

Quote:

Originally Posted by webheaded

I'm with Datashack as well. The support area was down earlier and then it just randomly came back up. I had emailed the support address and gotten nothing but they actually replied very quickly to the ticket I put in. It is times like this that I wish hosts would just have a status page so we didn't have to flood their support queues with questions during an outage.

Agreed, it is always times like that I wish I had at least one failover node offshore, so I could still provide my hosting service without interruption. But so far, I can only offer one page.
Posted by TupaHost, 08-06-2013, 09:30 PM
Anyone know how long it will take?
Posted by Servode, 08-06-2013, 09:30 PM
There is a lot to fiber optic splicing you guys,

i mean a lot;

Heres a few pictures from a few fiber jobs i have been on maybe it will give you guys a better idea of the process.

Fiber Tray: http://i.imgur.com/qLlw3P7.png

ATT man whole "Downtown dallas" http://i.imgur.com/alOHXPf.png

Logic Communications: http://i.imgur.com/h1rgDEW.jpg

Some more fiber: http://i.imgur.com/ypc5JuU.jpg

Anyway fiber optic splicing as a long and lengthy process when it comes to prep, And actual splicing the correct way ive always spliced using a fusion machine
Posted by Magiobiwan, 08-06-2013, 09:30 PM
I remember the Fiber Cut up in Canada that hit OVH. That was a mess too. Have fun, splicing crews!
Posted by dlabrie, 08-06-2013, 09:31 PM

Quote:

Originally Posted by TupaHost

Anyone know how long it will take?

None here knows better than Aaron, he's probably supervising the fiber installation outside.
Posted by TupaHost, 08-06-2013, 09:33 PM
8 hours already passed all my servers ones outside. : (
Posted by dlabrie, 08-06-2013, 09:35 PM

Quote:

Originally Posted by Servode

There is a lot to fiber optic splicing you guys,

i mean a lot;

Heres a few pictures from a few fiber jobs i have been on maybe it will give you guys a better idea of the process.

Fiber Tray: http://i.imgur.com/qLlw3P7.png

ATT man whole "Downtown dallas" http://i.imgur.com/alOHXPf.png

Logic Communications: http://i.imgur.com/h1rgDEW.jpg

Some more fiber: http://i.imgur.com/ypc5JuU.jpg

Anyway fiber optic splicing as a long and lengthy process when it comes to prep, And actual splicing the correct way ive always spliced using a fusion machine

Thank you, I thought fiber would be much bigger.
Posted by 8inghost, 08-06-2013, 09:38 PM

Quote:

Originally Posted by dlabrie

Thank you, I thought fiber would be much bigger.

I thought it would be much dirtier! lol
Almost looks comfortable in those pics.
Posted by Servode, 08-06-2013, 09:39 PM
It all depends, Some fiber is actually air blown now days.
The fiber in those screen shots actually have a gel in the buff tube that contains 12 fibers, its called icky pick smells like syrup

but if it gets dirty its very difficult to clean.

And the actual fiber tube is big, It really depends on how much fiber is in a tube, some tubes go from 1 fiber buff x12 fibers, to 468 fiber ,
Posted by dlabrie, 08-06-2013, 09:44 PM
I could imagine that headache you get working on those.
Posted by CBH-Nick, 08-06-2013, 09:48 PM
Funny how with all these backup systems in place, the one thing that shuts down the data center is a fiber optic cable being cut.
Posted by osgregs, 08-06-2013, 09:53 PM
what bothers most is the backup
Posted by DPI, 08-06-2013, 09:53 PM
Fiber cuts happen everyday, all the time, anywhere. Sounds like it just so happened that the tunnel bore took out two paths at the same time.
Posted by osgregs, 08-06-2013, 09:57 PM
fix this soon <<snipped>>
Posted by WojonsTech, 08-06-2013, 09:57 PM

Quote:

Originally Posted by WII-Aaron

So apparently they didn't have it cleared and they're opening up another section of the street to get to the blockage....

Wojons: this fiber belongs to the city.

Wow your lucky about that I wish my city had fiber already in place and would be easy to hook into.
Posted by CN-Jeremy, 08-06-2013, 10:02 PM
Yeah fiber splicing can be a PIA at times.. Lots of cleaning strands over and over to get them perfectly clean, making good cuts at the end, etc.. The actual splice machine is pretty quick (at least the one we had). You had to get everything all cleaned, cut correct and then put it in the machine, aligned correctly and press the button... It then pushes the ends together and a bright light(the laser) and then you see one strand.. Very cool to do..

Your also right about the iky pick.... it sucks... especially on the larger cables..

For those that don't know what iky pick is, its basically like petroleum gelly inside fiber and copper telecom cabling and its a PIA to get off anything it touches.
Posted by dlabrie, 08-06-2013, 10:10 PM
It is really surprising that such small fibers can transfer Gbps/Tbps of data.
Posted by TupaHost, 08-06-2013, 10:18 PM
My apologies for your inconvenience, like you we anxiously await a resolution to this issue.
Unfortunately we are at the mercy of the local Telco who is making the repairs.

--
Thanks!

Gragg
Posted by Justin, 08-06-2013, 10:19 PM
It's definitely getting to the point where even I can't wait to hear Aaron tell everyone it's been fixed. Last update (from Aaron) was almost 2 hours ago. Guessing they hit another snag with getting the line lit back up.
Posted by jonsparks, 08-06-2013, 10:27 PM
AT&T better compensate WSI well for this, this wasted a whole day of development time on two of my projects since we couldn't use my git server. I'm kinda hoping DataShack offers some kind of service credit.
Posted by DPI, 08-06-2013, 10:32 PM
Sounds like there is a few crews working, and had to open another hole to remove a blockage, so they ARE working on it, but it does take time. The guys working this break have (probably) been at work since 6-7AM and are on OT. It takes time to get fiber spliced properly with no obstacles, but when you run into one, it can add significant time.
Posted by jonsparks, 08-06-2013, 10:36 PM
I think the break was sometime around 1pm CST, it's 9:30 now. I know Aaron mentioned a couple obstacles they had run into. Hopefully it's all spliced back soon though. If it's not up soon I'll just end up getting woke up as soon as the network is fixed.
Posted by WojonsTech, 08-06-2013, 10:37 PM

Quote:

Originally Posted by jonsparks

AT&T better compensate WSI well for this, this wasted a whole day of development time on two of my projects since we couldn't use my git server. I'm kinda hoping DataShack offers some kind of service credit.

I know how you feel i am currently working on a project that uses large data sets and my test servers are out in datashack so my git is fine its lives in the cloud and have a fall back but for db its hopeless thinking about just getting a rack of servers in my house.
Posted by CampinasPublicidade, 08-06-2013, 10:37 PM
How broke this fiber?
Posted by WII-Aaron, 08-06-2013, 10:38 PM
It's definitely taking longer than I thought it would. The construction crews left a few hours ago and now it's just the splicers. I was expecting to be the first ones to be spliced back in but a few minutes ago they told me that they were just starting on the runs from City Hall. (We're on those) Then there are guys with fiber maps who I think are trying to figure out which strands go to who... It's a messed up deal.

I'm hoping to get a better ETA soon.
Posted by brianz, 08-06-2013, 10:44 PM
Looks like they will work on it all night
Posted by alect, 08-06-2013, 10:45 PM
thanks for the updates,
and my server id down for 537 minutes..
Posted by jonsparks, 08-06-2013, 10:46 PM

Quote:

Originally Posted by WojonsTech

I know how you feel i am currently working on a project that uses large data sets and my test servers are out in datashack so my git is fine its lives in the cloud and have a fall back but for db its hopeless thinking about just getting a rack of servers in my house.

I have everything on a private Gitlab server rather than paying for private repos on GitHub, and all of the sites I host for my web design clients are down. Luckily only a couple have called me, and I sent out a mass email to all of them earlier explaining the situation and waving their next invoice. So no harm done other than my dev team ended up not working today.
Posted by CampinasPublicidade, 08-06-2013, 10:52 PM
how did this happen?
Posted by digdugdog, 08-06-2013, 10:59 PM
I've had a server with them for a year or so, and I think this is the first issue I've had with their connectivity. For me it's a redundant server, so it's not urgent for me to regain sevice... just concerned that one backhoe can take out their entire connectivity.
Posted by musfiq, 08-06-2013, 11:21 PM
any update on the fiber cut??

when it will be restored??
Posted by IgorSantos11, 08-06-2013, 11:24 PM
Estatísticas do Ping para 69.30.200.xxx:
Pacotes: Enviados = 6297, Recebidos = 0, Perdidos = 6297 (100% de perda).
Posted by Linn_Boyd, 08-06-2013, 11:40 PM
What this sounds like is a PTP fiber that is being covered up, how can you not look and make sure that the east and west or north and south sections of the ring aren't going out in the same fiber duct. What this does boil down to is that it is your fault.

We were all under the impression that you had complete redundancy in the datacenter. This kind of downtime in the middle of a weekday is not acceptable. Even Cogent makes darn sure that the fibers go out completely different ducts all the way to the rack, and if it is a HUB they have east/west and north/south and they are usually on different providers.
Posted by mainpipe, 08-07-2013, 12:20 AM
I'm back up!!!

Special thanks to Aaron and the entire Wholesale Internet team.
Posted by hD-Scott, 08-07-2013, 12:20 AM
Back up. Woo hoo!
Posted by rbstern, 08-07-2013, 12:21 AM
My server just came up.

Aaron and crew, I know that was a lot of stress. Hope you guys get some sleep!
Posted by HostHoarders-Mike, 08-07-2013, 12:21 AM

Quote:

Originally Posted by mainpipe

I'm back up!!! Thanks Wholesale Internet.

This exactly. You all beat me to it. Hopefully this never happens again but thanks for getting it fixed!
Posted by TupaHost, 08-07-2013, 12:25 AM
My server returned however am not having ping response.
Posted by duran2003, 08-07-2013, 12:27 AM
Some server still down here.
Posted by rveram, 08-07-2013, 12:28 AM
Back online, thanks!
Posted by Justinwiz, 08-07-2013, 12:31 AM
Partially back online
Posted by arthur8, 08-07-2013, 12:31 AM
Seems to be the network is overloaded. Ping times are 177 (normal for Brazil) but for some moments it gets ~1000-1400.

Thanks, i hope this gets resolved. Here is some ping LOG:

Quote:

Resposta de 74.91.xx.xxx: bytes=32 tempo=1338ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=2117ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=207ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=494ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=342ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=602ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=1437ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=1850ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=884ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=219ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=303ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=223ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=234ms TTL=48
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Esgotado o tempo limite do pedido.
Resposta de 74.91.xx.xxx: bytes=32 tempo=177ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=177ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=177ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=177ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=177ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=177ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=319ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=487ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=176ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=780ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=785ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=1230ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=1093ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=1603ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=1702ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=298ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=466ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=216ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=437ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=562ms TTL=48
Resposta de 74.91.xx.xxx: bytes=32 tempo=599ms TTL=48
Esgotado o tempo limite do pedido.

Posted by gigist, 08-07-2013, 12:31 AM
Hopping off and on.
Posted by icedemo, 08-07-2013, 12:34 AM
up for me now. time wasted= 10hrs
Posted by Dedispec, 08-07-2013, 12:35 AM

Quote:

Originally Posted by icedemo

up for me now. time wasted= 10hrs

Well geez, you could have driven to Kansas City and helped AT&T with some Fiber work it sounds like?

Glad to see connectivity starting to pop back. Saw some of our misc. boxes come back up
Posted by Microlinux, 08-07-2013, 12:36 AM

Quote:

Originally Posted by Linn_Boyd

how can you not look and make sure that the east and west or north and south sections ring aren't going out in the same fiber duct.

You'd be shocked at how many "rings" are built this way . . . and I'm not sure they claim to have a ubiquitous ring topology.

Quote:

Originally Posted by Linn_Boyd

Even Cogent makes darn sure that the fibers go out completely different ducts all the way to the rack, and if it is a HUB they have east/west and north/south and they are usually on different providers.

WSI is not a carrier backbone, they are a budget colo facility. And let me tell you about Cogent's standards for physical transport . . .
Posted by prandah, 08-07-2013, 12:37 AM
yes now up
Posted by WII-Aaron, 08-07-2013, 12:45 AM
Some of our fiber is back online. You may see some increased latency while things even out. We're still here working on it.
Posted by webheaded, 08-07-2013, 01:03 AM
Everything is working great, either way. It's been a long day.
Posted by duran2003, 08-07-2013, 01:10 AM
All servers are UP now.
Posted by arthur8, 08-07-2013, 01:11 AM

Quote:

Originally Posted by webheaded

Everything is working great, either way. It's been a long day.

Yes.

I hope this never happen again. I personally really like WII/DS guys service.
Posted by spottraining, 08-07-2013, 01:14 AM
Good timing

My server was down around 21:30 by Estonian time - so it was not any more work time and back 8:00 by Estonian time - just beginning of the workday

Most of my customers do not notice the interruption
Posted by gigist, 08-07-2013, 01:25 AM
My DataShack server is still bouncing around (on and off).
Posted by httpEasy, 08-07-2013, 01:31 AM

Quote:

Originally Posted by arthur8

Yes.

I hope this never happen again. I personally really like WII/DS guys service.

Their support is really good and I haven't seen any issues before. Still their network is nothing to host critical stuff on.
Posted by WojonsTech, 08-07-2013, 02:20 AM

Quote:

Originally Posted by WII-Aaron

Some of our fiber is back online. You may see some increased latency while things even out. We're still here working on it.

I am guessing the fiber that is up is not able to handle all the traffic untill all fiber lines are back on line.
Posted by WojonsTech, 08-07-2013, 02:21 AM

Quote:

Originally Posted by jonsparks

I have everything on a private Gitlab server rather than paying for private repos on GitHub, and all of the sites I host for my web design clients are down. Luckily only a couple have called me, and I sent out a mass email to all of them earlier explaining the situation and waving their next invoice. So no harm done other than my dev team ended up not working today.

yeah i have a private git server that i use for perosnal projects then i have a github private repos for a few things that are related to working with a few clients this way i can give them access to github and not my private server never wanna be in the situcation like this where they cant work or see my work because the my server is down. if github is down its a different level.
Posted by JPerkster, 08-07-2013, 02:34 AM
Yep back up and running.

**** is UP again at 08/07/2013 05:27:19AM, after 10h 40m of downtime.
Posted by Several Companies, 08-07-2013, 03:14 AM
Just checked, and my server there is still down. So I guess not everything is fixed yet. Hopefully it will be back up soon.

Aaron's team has always been excellent, I can't imagine the headaches they are getting from today.
Posted by Linn_Boyd, 08-07-2013, 03:28 AM

Quote:

Originally Posted by [CTI] Todd

You'd be shocked at how many "rings" are built this way . . . and I'm not sure they claim to have a ubiquitous ring topology.

WSI is not a carrier backbone, they are a budget colo facility. And let me tell you about Cogent's standards for physical transport . . .

They definitely make it sound like they have ubiquitous ring, I can also tell you a thing or two about cogent, but if a large number of servers are on a "ring" it shouldn't be where some id10t can hit it out in a street. Also WSI claimed to have two fiber hits by the same boring machine. Making it seem that they did have a true ring. I just can't see running a secondary facility on a single carrier fabric.
Posted by bdwebservices, 08-07-2013, 03:35 AM
my server is up after 10hrs
Posted by Several Companies, 08-07-2013, 03:47 AM
Just as an update, now my server is up as well.
Posted by deshanshashi, 08-07-2013, 05:15 AM

Quote:

Originally Posted by Several Companies

Just as an update, now my server is up as well.

that's good
Posted by osgregs, 08-07-2013, 07:34 AM
server up
Posted by Ankheg, 08-07-2013, 08:38 AM
While our affected server is back up and reachable from the outside world, it's not reachable from our other, unaffected, servers at oak tower, sigh. I hope that gets fixed sometime today.
Posted by codatory, 08-07-2013, 08:54 AM
I didn't even know I was in a secondary facility. And certainly wasn't aware that as part of that I didn't have diverse fiber service.

Does this secondary facility have UPSes, generators or anything else? What about access control? Fire suppression?

Quote:

Wholesale Internet houses its 14,000 square foot facility within Oak Tower

The website says nothing about the space across the street, so until I hear otherwise I have to assume it has zero of the facilities of a normal datacenter.
Posted by shnoc, 08-07-2013, 09:05 AM

Quote:

Originally Posted by codatory

I didn't even know I was in a secondary facility. And certainly wasn't aware that as part of that I didn't have diverse fiber service.

Does this secondary facility have UPSes, generators or anything else? What about access control? Fire suppression?

The website says nothing about the space across the street, so until I hear otherwise I have to assume it has zero of the facilities of a normal datacenter.

Is correct, I wanted my server in the structure that is announced on the website of the datacenter.
Posted by EjeetNetworks, 08-07-2013, 10:15 AM
Servers is back on line after 10~ hours, since the staff just closes out any tickets regarding the outage I have to ask will there be any SLA.
Posted by ryguy222, 08-07-2013, 11:13 AM

Quote:

Originally Posted by codatory

I didn't even know I was in a secondary facility. And certainly wasn't aware that as part of that I didn't have diverse fiber service.

Does this secondary facility have UPSes, generators or anything else? What about access control? Fire suppression?

The website says nothing about the space across the street, so until I hear otherwise I have to assume it has zero of the facilities of a normal datacenter.

Datashack has multiple facilities secondary to Oak, and they receive bandwidth from oak. These facilities have backup/ups power and diverse fiber. The problem yesterday was ATT hit BOTH lines at the one point where the two lines cross. There have been power outages at the secondary facilities and backup generators worked fine in the past.

The newest place (Clay facility I think its called?) Might be shared by WSI / DS but I'm not certain. All of our colo is at the Cable facility, and we've never had any problems until yesterday.

If you aren't sure where your stuff is at open a ticket and ask.
Posted by codatory, 08-07-2013, 12:58 PM
If a single backhoe can take out all the connectivity to a building, that's not redundant fiber. I suppose shame on me for believing that their website conveyed complete and accurate information about their facility, I'll have to open a support ticket later to find out what the actual datacenter is like.
Posted by ryguy222, 08-07-2013, 01:01 PM
It wasn't a back hoe, it was a drill that hit two lines that were 5 feet apart. Aaron already said they were going to move them farther apart now that everything was cut open.

Ever hear of you get what you pay for? You aren't paying a premium like you would at most datacenters.
Posted by httpCORE, 08-07-2013, 02:23 PM
Down again
Posted by Justinwiz, 08-07-2013, 02:26 PM
I've been stable since 8:17PM PST last night.
Posted by IgorSantos11, 08-08-2013, 07:04 PM
Ok, all servers it's online now. But, who will pay for the downtime?
Posted by Dougy, 08-09-2013, 09:26 AM

Quote:

Originally Posted by IgorSantos11

Ok, all servers it's online now. But, who will pay for the downtime?

Nobody. It's a fact of life. Especially when you are in a budget facility.
Posted by pechspilz, 08-17-2013, 01:31 AM
Is it just me or did Wholesaleinternet just pull the plug on their local DNS resolvers?
Google DNS is 22ms away...
Posted by ryguy222, 08-17-2013, 01:34 AM
We saw it too, and just changed over to google. Did you open a ticket?
Posted by pechspilz, 08-17-2013, 02:46 AM
Yes. They said its permanent. Since they preinstall their name server in their OS templates, I wonder how many servers are without DNS right now. Even though they're a budget provider, this is far from professional.
Posted by Techno, 08-17-2013, 02:48 AM

Quote:

Originally Posted by pechspilz

Is it just me or did Wholesaleinternet just pull the plug on their local DNS resolvers?
Google DNS is 22ms away...

Same here, I'm on Datashack. It caused my in/out email to bog down. I switched to Google DNS last week. Didn't open a ticket.
Posted by ryguy222, 08-17-2013, 02:49 AM
Some of our stuff from Datashack used it, however the newer servers were setup with google from the start.