Wholesale Internet

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Wholesale Internet

Posted by subigo, 04-19-2011, 03:59 AM
Heads up... http://wholesaleinternet.com is down. All of my servers with them are unreachable.
Posted by chasebug, 04-19-2011, 04:07 AM
Down here also!
Posted by ramnet, 04-19-2011, 04:08 AM
It started about 2 hours ago in the colocation section and it has now expanded to the entire datacenter.

All our stuff is down as well however only IPv4 is down - IPv6 to them is working fine.
Code:
We are currently looking into possible issues, I will keep you updated.

-- 
Best Regards,
AJ Whitehead
Wholesale Internet Support
Posted by Bios, 04-19-2011, 04:10 AM
Yep servers down here.

Subscribing to thread
Posted by AvailHosting-Jeff, 04-19-2011, 04:11 AM
Just got off the phone with Aaron, he reports they're having an issue with an upstream provider. Didn't ask him many questions--didn't want to tie up his time.
Posted by chasebug, 04-19-2011, 04:32 AM
Still down and cannot get to their website to submit any tickets.
Posted by amusive.com, 04-19-2011, 04:37 AM
https://customer.wholesaleinternet.net/ does still work, but I don't see why you'd bother filing a ticket if their entire network is down, and they're aware of it.
Posted by mpkapadia, 04-19-2011, 04:41 AM
Did they give any Estimated time for resolving the problem .
I have one server there that isnt accessible.
Posted by JB_404, 04-19-2011, 04:44 AM
"Good" morning,

I am on the same boat, down for 2+ hours now. (I find this VERY long).

Quote:

Originally Posted by AvailHosting-Jeff

he reports they're having an issue with an upstream provider.

Aren't connections of DataCenters supposed to be redundant to avoid one provider to take down the entire network?
Posted by GeekMinister, 04-19-2011, 04:52 AM
Servers DOWN too, here!!!
Posted by Blurple, 04-19-2011, 04:54 AM
Got a server down here to, don't see a need to submit a ticket since it is affecting everyone, got a little worried this morning though...
Posted by AvailHosting-Jeff, 04-19-2011, 04:57 AM

Quote:

Originally Posted by mpkapadia

Did they give any Estimated time for resolving the problem .
I have one server there that isnt accessible.

No ETA yet--could be a long one.

No point in opening tickets folks, everyone at WSI is well-aware of the issue. Best to sit tight and let the people at WSI do their work.
Posted by cashadv, 04-19-2011, 05:00 AM
ok thats good that they are aware of the issue
hope they solve it real fast
Posted by blacktulip, 04-19-2011, 05:06 AM
they are not running away, are they? ... I got a server with them and it is down now
Posted by cashadv, 04-19-2011, 05:10 AM
can anyone call them and ask for ETA thats very long time
Posted by chasebug, 04-19-2011, 05:14 AM
Over 1 hour down now.
Posted by subigo, 04-19-2011, 05:15 AM
Why would you guys think they would have an ETA? It's an upstream issue. It's not planned maintenance. They have no ETA and won't have one.
Posted by JOEsDC, 04-19-2011, 05:15 AM
I spoke with Aaron and he asked me to come let everyone know what was going on. They had a piece of equipment fail and have had to go to another facility to get a replacement part. They are in the process of getting this fixed and have brought additional staff in as well to make sure everything is done as quickly as possible. They should be able to get everything back to normal operational status shortly and everyone's tickets will get answered. Obviously things are back logged and their highest priority right now is getting everything fixed. I will let you know if I hear any additional information.
Posted by subigo, 04-19-2011, 05:17 AM

Quote:

Originally Posted by JOEsDC

I spoke with Aaron and he asked me to come let everyone know what was going on. They had a piece of equipment fail and have had to go to another facility to get a replacement part. They are in the process of getting this fixed and have brought additional staff in as well to make sure everything is done as quickly as possible. They should be able to get everything back to normal operational status shortly and everyone's tickets will get answered. Obviously things are back logged and their highest priority right now is getting everything fixed. I will let you know if I hear any additional information.

Thank you.
Posted by ramnet, 04-19-2011, 05:21 AM

Quote:

Originally Posted by AvailHosting-Jeff

No point in opening tickets folks, everyone at WSI is well-aware of the issue. Best to sit tight and let the people at WSI do their work.

As someone who just finished answering ~100 tickets from my clients about this, I heartily agree.

Don't take valuable time away from them fixing the problem

----

JOEsDC, thanks for passing along an update.
Posted by ramnet, 04-19-2011, 05:26 AM

Quote:

Originally Posted by chasebug

Over 1 hour down now.

For some of us this started 2 hours before this thread appeared.

It would seem they may have had multiple equipment failure, or one piece of equipment started to fail and gradually took down everything over the course of 2 hours.

The last thing to fail was IPv6 which finally went down about 40 minutes ago.
Posted by cashadv, 04-19-2011, 05:34 AM
4 hours down for us
Posted by ramnet, 04-19-2011, 05:37 AM
IPv6 just came back up....here's hoping IPv4 is close behind
Posted by JB_404, 04-19-2011, 05:38 AM

Quote:

Originally Posted by JOEsDC

I spoke with Aaron /.../ I will let you know if I hear any additional information.

Thank you Joe!
Posted by noworg, 04-19-2011, 05:49 AM
~JoesDC work fine.....why the WII .......
Posted by chasebug, 04-19-2011, 05:51 AM
Now over 2 hours down.
Posted by cashadv, 04-19-2011, 05:58 AM
5 hours down now
Posted by cashadv, 04-19-2011, 06:00 AM
looks like all is back
Posted by blacktulip, 04-19-2011, 06:01 AM

Quote:

Originally Posted by cashadv

looks like all is back

Mine is still down
Posted by JB_404, 04-19-2011, 06:01 AM
Back online after a litte more than 3 hours of downtime.
Posted by ramnet, 04-19-2011, 06:02 AM
Yup looks like we're back up on IPv4 now

Edit: and as soon as I post it all goes poof again
Posted by cashadv, 04-19-2011, 06:03 AM
now down again
Posted by JB_404, 04-19-2011, 06:04 AM
Hmm, seams like I spoke too fast...
Posted by cashadv, 04-19-2011, 06:21 AM
did anyone hear from them on status?
Posted by MrPiang, 04-19-2011, 06:25 AM
My server DOWN too... I guess they facing big issue right now. Its DOWN more than 2Hrs for me.
Posted by noworg, 04-19-2011, 06:27 AM

Quote:

Originally Posted by cashadv

did anyone hear from them on status?

Aaron is going to get the part we need as I type this. I don't know how long it
will take to get us up fully once we have the part.
Posted by JB_404, 04-19-2011, 06:37 AM

Quote:

Originally Posted by noworg

Aaron is going to get the part we need as I type this. I don't know how long it
will take to get us up fully once we have the part.

Thank you for the update noworg.
Posted by cashadv, 04-19-2011, 06:40 AM
i have 3 servers with them am going to move them
thats enough of bs
there cant be anything that explains six hours downtime for us
Posted by blacktulip, 04-19-2011, 06:43 AM

Quote:

Originally Posted by noworg

Aaron is going to get the part we need as I type this. I don't know how long it
will take to get us up fully once we have the part.

offtopic, but the website in your sig is hacked...
Posted by VirtualSRV, 04-19-2011, 06:47 AM
It's back up for me now.
Posted by ramnet, 04-19-2011, 06:48 AM

Quote:

Originally Posted by cashadv

i have 3 servers with them am going to move them
thats enough of bs
there cant be anything that explains six hours downtime for us

Judging by your past threads here, I'm sure they won't mind too much

http://www.webhostingtalk.com/showthread.php?t=1037723

We have over 20 servers with them and we have no intention of leaving over a one-time incident that will happen to every datacenter at some point.
Posted by blacktulip, 04-19-2011, 06:48 AM

Quote:

Originally Posted by vpsplug

It's back up for me now.

not for me...
Posted by cashadv, 04-19-2011, 06:54 AM
six hours downtime comeon )
Posted by cashadv, 04-19-2011, 07:00 AM

Quote:

Originally Posted by vpsplug

It's back up for me now.

what ip block you're on?
Posted by MrPiang, 04-19-2011, 07:10 AM

Quote:

Originally Posted by cashadv

what ip block you're on?

NOT for me too... Im from 173.208.165.* My server still DOWN.
Posted by Bios, 04-19-2011, 07:14 AM
One of my servers is back up.

IP range: 204.12.231.XXX
Posted by @Matt, 04-19-2011, 07:15 AM
Were still also experiencing downtime, but AJ just notified us that partial parts of the network are coming back online.

Quote:

We are currently experiencing networking issues, however we have partial network functionality. We are still working on the issue but parts of the network are online now.

--
Best Regards,
AJ Whitehead
Wholesale Internet Support
support@wholesaleinternet.net

Posted by cashadv, 04-19-2011, 07:16 AM
looks like only a few servers are connected back i just spoke to friend that has with them over 25 servers and he says all of them are down
Posted by ramnet, 04-19-2011, 07:17 AM
3 of ours are back so far.
Posted by MrPiang, 04-19-2011, 07:22 AM
Hmmm.. Anybody know the WII got any Network SLA etcs?
Posted by cashadv, 04-19-2011, 07:23 AM
our 3 still down
Posted by twhiting9275, 04-19-2011, 07:30 AM

Quote:

Originally Posted by cashadv

i have 3 servers with them am going to move them

With that attitude , I'm sure they won't mind

Quote:

Originally Posted by cashadv

thats enough of bs
there cant be anything that explains six hours downtime for us

Actually there can. When major issues happen in the middle of the night, it can take a while for proper responses to be made. I've been a customer of one of their network resellers on and off for years and haven't seen anything this huge in that time.

Some people just like to complain
Posted by Bios, 04-19-2011, 07:30 AM
All my servers are back up. Only 3 hours downtime for me which is probably the first time I've experienced something like this since I joined back in December 09.

Another IP range: 173.208.133.XXX
Posted by ramnet, 04-19-2011, 07:31 AM
All of mine just came back online
Posted by blacktulip, 04-19-2011, 07:32 AM
Great, mine is up now
Posted by Hoppa66, 04-19-2011, 07:35 AM

Quote:

Originally Posted by ramnet

All of mine just came back online

Check!

Let's hope it stays up ...
Posted by cashadv, 04-19-2011, 07:35 AM
great my 3 as well
thanks god
Posted by ErikD, 04-19-2011, 07:36 AM

Quote:

Originally Posted by cashadv

our 3 still down

And you coming to post back every few minutes that there are still issues does absolutely nothing to help the problem. Wait till your servers are up.

I have some servers there too, sure the downtime is a pain, but there are some real simple ways to deal with it. If you really do want to movae to a host that a enure you will have no such downtimes start looking, but you better increase your budget by about 10x at least. Or you could of course just go to a carrier neutral datacenter and build out your own fault tolerant network, how hard could it possibly be to design and maintain a network with 99% uptime?
Posted by MrPiang, 04-19-2011, 07:38 AM

Quote:

Originally Posted by Bios

All my servers are back up. Only 3 hours downtime for me which is probably the first time I've experienced something like this since I joined back in December 09.

Another IP range: 173.208.133.XXX

My server back online too =) from same IP block.
Posted by cashadv, 04-19-2011, 07:43 AM
is there anymore downtime 2 come or thats it?
Posted by blacktulip, 04-19-2011, 07:43 AM

Quote:

Guaranteed Services
99.9% Uptime Guarantee
BGP4 Bandwidth (KCIX, Hurricane Electric, Cogent)
24/7 Emergency Support
Secure Locking Cabinets
Remote Hands
400KW Diesel Backup
UPS Battery Systems

that's from their website. But I know **** happens, I wont move for one incident like this, unless it keeps coming.
Posted by @Matt, 04-19-2011, 07:54 AM
I just found out the problem from AJ and he stated it was 10GB card that failed. Just incase if anyone here is looking for an answer to what exactly went wrong and info to pass over to your clients.

Quote:

We had a 10gb card fail, causing these issues.

--
Best Regards,
AJ Whitehead
Wholesale Internet Support
support@wholesaleinternet.net

Posted by VirtualSRV, 04-19-2011, 08:46 AM
My server is on the following IP: 173.208.141.*
Posted by ErikD, 04-19-2011, 08:51 AM

Quote:

Originally Posted by blacktulip

that's from their website. But I know **** happens, I wont move for one incident like this, unless it keeps coming.

99.9% uptime equates to just under 9hrs unplanned downtime in a year, so technically they are still within this margin.

My server is back online too.
Posted by WII-Aaron, 04-19-2011, 09:18 AM
We had an issue with one of the core routers that caused our internal routing table to go wacky. (Wacky is the technical term by the way). We replaced the faulty equipment pretty quickly but it took us a while to figure out why no internal routes were getting passed around.

We believe the issue is resolved and do not expect any additional downtime from this incident.

If you have specific questions please feel free to open a ticket and we'll be happy to answer them.
Posted by skullbox, 04-19-2011, 10:02 AM
A lot of folks bitched about this outage. Look at it this way, at least it wasn't during business hours. I can accept this as none of my customers even noticed. I do B2B stuff, so for all you B2C sucks for you.

However, at least this was an equipment failure and not decision to upgrade power on a Monday morning EST like the 4 hour hostdime outage several months back.
Posted by Patrick, 04-19-2011, 10:11 AM
Was Joe's also affected by this outage?
Posted by @Matt, 04-19-2011, 10:28 AM

Quote:

Originally Posted by Patrick

Was Joe's also affected by this outage?

It was a hardware failure related to only WSI...
Posted by Patrick, 04-19-2011, 10:30 AM

Quote:

Originally Posted by PogiWeb

It was a hardware failure related to only WSI...

Thanks. For some reason I keep thinking Joe's uses WII's network infrastructure but I guess everything is separate now?

(Planning on moving a backup server there...)
Posted by chasebug, 04-19-2011, 12:22 PM
3 and 1/2 hour downtime. First downtime since moved to WII, will see if this is going to be an ongoing issue.
Posted by Blurple, 04-19-2011, 01:01 PM
I have been with WII for a while, maybe 2-3 years im guessing, and it is the first downtime I have had or atleast noticed. Once again it was good that it happened during the morning hours and I am NOT planning on moving to anyone else. The way I see it, **** happens and they will learn from this incident and hopefully it wont happen in the future, or at least they will be able to get it fixed a lot quicker.

Keep up the good work WII!