wholesaleinternet.com down

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > wholesaleinternet.com down

Posted by csgo, 02-25-2012, 03:19 PM
Our server at wholesaleinternet.com has been down for about 10 minutes. Can't reach their web site either so I guess their whole system is down as well.

Anyone have any info?
Posted by totalgeek, 02-25-2012, 03:20 PM
From my limited knowledge they host in the same building as Datashack? Because datashack.net is unreachable as well and so is my server on there.
Posted by The Universes, 02-25-2012, 03:20 PM
Yeah they are down. I just put in a ticket to get more info.
Posted by csgo, 02-25-2012, 03:21 PM

Quote:

Originally Posted by The Universes

Yeah they are down. I just put in a ticket to get more info.

How did you put in a support ticket??
Posted by The Universes, 02-25-2012, 03:22 PM

Quote:

Originally Posted by csgo

How did you put in a support ticket??

http://customer.wholesaleinternet.net/
Posted by Patrick, 02-25-2012, 03:22 PM
I think it's related to the power upgrade they were doing today. Service wasn't supposed to be impacted as they were going to run on a generator, but one of our boxes just went down.

https://customer.wholesaleinternet.net/client/

Already put in a ticket.
Posted by csgo, 02-25-2012, 03:23 PM
Dear Wholesale Customer,

This Saturday, February 25th, Oak Tower will be performing
maintainence on it's main power distribution system. Power is
scheduled to be off from 6am to 6pm Central time.

During this time we will be running on generator and do not expect the
event to have a major impact. Please note, however, that building
services will be unavailable such as elevators and running water. For
your safety we ask that you do not come to the facility while the
maintainence is under way.
If you are a colo customer you may call our network department in the
case of an emergency.

If you have any questions, please let me know.

Aaron Wendel
Wholesale Internet, Inc.
Posted by The Universes, 02-25-2012, 03:24 PM
We know about that already. WII has generators though and that power maintenance shouldn't have affected them.
Posted by Patrick, 02-25-2012, 03:24 PM
The network came back online for a minute, just dropped off again though. Safe to assume they are working to get service restored ASAP.
Posted by csgo, 02-25-2012, 03:25 PM
Mine just came back up.
Posted by chasebug, 02-25-2012, 03:27 PM
Servers down again?
Posted by csgo, 02-25-2012, 03:31 PM

Quote:

Originally Posted by chasebug

Servers down again?

Yes, down again.
Posted by nixcom, 02-25-2012, 03:35 PM
down here too...
Posted by chasebug, 02-25-2012, 03:44 PM
I have servers at both WI and Datashack but both servers down even though they are in different facility.
Posted by csgo, 02-25-2012, 03:46 PM
Back online... for now anyway.
Posted by pubcrawler, 02-25-2012, 03:49 PM

Quote:

Originally Posted by chasebug

I have servers at both WI and Datashack but both servers down even though they are in different facility.

There's ample confusion about what is and isn't shared between these providers and throw Datashack in there too.

WII seems to be the main conduit for everything.

WII has it's own gen sets, Joe's does not.

So check your Joe's server for uptime to see if that box went off with power. Repeat for Wholesale box.

Should see Joe's server rebooted and WII did not.

Odd though, that connectivity to the facility went offline. Wondering if WII lost power somehow.
Posted by Patrick, 02-25-2012, 03:50 PM

Quote:

Originally Posted by pubcrawler

Odd though, that connectivity to the facility went offline. Wondering if WII lost power somehow.

I can confirm that our boxes at WII did lose power.
Posted by csgo, 02-25-2012, 03:52 PM

Quote:

Originally Posted by pubcrawler

There's ample confusion about what is and isn't shared between these providers and throw Datashack in there too.

WII seems to be the main conduit for everything.

WII has it's own gen sets, Joe's does not.

So check your Joe's server for uptime to see if that box went off with power. Repeat for Wholesale box.

Should see Joe's server rebooted and WII did not.

Odd though, that connectivity to the facility went offline. Wondering if WII lost power somehow.

Our server at Wholesale rebooted... so power was out there too.
Posted by pubcrawler, 02-25-2012, 03:53 PM
WII power loss makes sense and why we saw connectivity down at Datashack remote facility.

Concerning of course this is, since routers seem to be central to these three companies and something needs done to elongate runtime via batteries perhaps or another genset just for core equipment.

Hopefully, there is a post incident wrap up to document what happened and to remedy things going forward.
Posted by nixcom, 02-25-2012, 03:55 PM
hope it is back for a while now
Posted by nixcom, 02-25-2012, 03:56 PM
never good to go down when you are upgrading kernel
Posted by pubcrawler, 02-25-2012, 04:00 PM

Quote:

Originally Posted by nixcom

never good to go down when you are upgrading kernel

Today was NOT the day to be mucking with your server.

Most certainly going to end up broken on reboot.
Posted by PrHosting, 02-25-2012, 04:02 PM
sysinfo shows a reboot and up for 15 minutes. I had a downtime of around 30 minutes.
Posted by Techno, 02-25-2012, 04:03 PM

Quote:

Originally Posted by nixcom

never good to go down when you are upgrading kernel

Did you get the WII or datashack emails about this scheduled outage?
Posted by desynced, 02-25-2012, 04:03 PM

Quote:

Originally Posted by nixcom

never good to go down when you are upgrading kernel

If you lost power during the upgrades. I would put in a ticket and have them choose the previous kernel in Grub, and upgrade later this evening when the building power is restored (6:00PM CST or earlier). I had a server lock up during a kernel upgrade once and after a manual reboot, it wouldnt boot. Had to boot off the previous kernel and force yum to remove new kernel and then installed again.
Posted by ryguy222, 02-25-2012, 04:03 PM
Was hoping I would have gotten an email with that announcement. I think they posted it, but didn't send out an email. I haven't logged into the portal for a good week so I never saw it. >_<

The power on my boxes at wholesale went down. Connectivity at datashack's north Kansas city facility was lost. Everything seems to be back up now.
Posted by pubcrawler, 02-25-2012, 04:09 PM
This is as per Virpus' Facebook posts:

"There was an issue with the voltage of the generator that caused it to shut down to protect itself. It has permanently been fixed."

9 minutes ago

We are now running off generator.
7 hours ago

Oddly, Virpus' own website is failing to load with a 502 Nginx error.

Feel back for all the customers out there wondering what is going on.
Posted by The Universes, 02-25-2012, 04:13 PM
From WII:

Quote:

Per the email that was sent out last week, we were going to have a power outage
but we would be running on a backup generator. We ran into an issue with the
generator causing us to lose the power feed. The generators are back online and
we are working now to restore your service.

Please be patient and we sorry for any inconvenience.

post length 123
Posted by nixcom, 02-25-2012, 04:32 PM

Quote:

Originally Posted by Techno

Did you get the WII or datashack emails about this scheduled outage?

Only got it TODAY, when decided to login on my customer account.

They should send it by email instead...

Everything back to normal, booted on the last kernel.
Posted by iTom, 02-25-2012, 04:46 PM
How many generators do they have?...
Posted by WII-Aaron, 02-25-2012, 04:47 PM
We sent the announcement Monday via e-mail in addition to posting it in our support system.

The issue at about 1pm was that the Generator, which had been running since 6am had a loose over voltage pot. The vibrations had caused it to move down and when it hit 480 the generator sensed it was over voltage and shut itself down.

One of our techs went out and cleared the faults which is why it came back on but then went down again in a few minutes.

Our generator guy dialed it back up to the correct over voltage setting and applied nail polish to keep it there. He is also checking it every 30 minutes to make sure there are no other issues.

As an added safety, we've moved the HE and Cogent building feeds onto a smaller, secondary generator so if we experience any other issues the other building tenants that still have power will be unaffected.

There were several issues this morning when the building tried to shut power down that delayed their maintainence. We aren't sure if utility power will actually be restored by 6pm.

I can't stress enough: PLEASE do not come to the building. The elevators are down, the lights are out and the emergency lights in the stairway are out. It is not a safe situation for people unfamiliar with the building layout.
Posted by nixcom, 02-25-2012, 04:49 PM
Keep on good work! But please check because I did not receive any announcement by email sincerely...
Posted by cd/home, 02-25-2012, 04:50 PM

Quote:

Originally Posted by WII-Aaron

We sent the announcement Monday via e-mail in addition to posting it in our support system.

The issue at about 1pm was that the Generator, which had been running since 6am had a loose over voltage pot. The vibrations had caused it to move down and when it hit 480 the generator sensed it was over voltage and shut itself down.

One of our techs went out and cleared the faults which is why it came back on but then went down again in a few minutes.

Our generator guy dialed it back up to the correct over voltage setting and applied nail polish to keep it there. He is also checking it every 30 minutes to make sure there are no other issues.

As an added safety, we've moved the HE and Cogent building feeds onto a smaller, secondary generator so if we experience any other issues the other building tenants that still have power will be unaffected.

There were several issues this morning when the building tried to shut power down that delayed their maintainence. We aren't sure if utility power will actually be restored by 6pm.

I can't stress enough: PLEASE do not come to the building. The elevators are down, the lights are out and the emergency lights in the stairway are out. It is not a safe situation for people unfamiliar with the building layout.

How often do you test that generator at full load and for how long?

Sounds like an upgrade is in order.
Posted by WII-Aaron, 02-25-2012, 04:53 PM

Quote:

Originally Posted by cd/home

How often do you test that generator at full load and for how long?

Sounds like an upgrade is in order.

Once a week for an hour.

Tuesdays at 10.
Posted by subigo, 02-25-2012, 05:00 PM

Quote:

Originally Posted by WII-Aaron

Once a week for an hour.

Tuesdays at 10.

Are there still servers without power right now?
Posted by The Universes, 02-25-2012, 05:00 PM
Thanks for the updates Aaron!

Its also good to hear the generator is tested regularly.
Posted by WII-Aaron, 02-25-2012, 05:10 PM

Quote:

Originally Posted by subigo

Are there still servers without power right now?

If your server is offline please submit a ticket. We don't have anything without power right now.
Posted by chasebug, 02-25-2012, 05:14 PM
Why were servers at datashack at another building affected by this power outage?
Posted by WII-Aaron, 02-25-2012, 05:18 PM

Quote:

Originally Posted by chasebug

Why were servers at datashack at another building affected by this power outage?

When we lost power Datashack lost it's network connectivity, so did a lot of other people.

Believe me, no one wants this to be done faster than I do.
Posted by pubcrawler, 02-25-2012, 05:53 PM
Aaron is the man behind the curtain for sure. Big hand clap for him finding the time to update folks... It's unexpected, but very proactive.

Everyone down again too?
Posted by subigo, 02-25-2012, 05:56 PM
Sigh. Everything is down again.
Posted by mcdl, 02-25-2012, 05:58 PM
Yep, so much for it being fixed.
Posted by subigo, 02-25-2012, 06:00 PM
Honestly, I would have rather just had a scheduled outage without trying to run the generator at all at this point. At least that way I could have cleanly shutdown my servers and not worried about data corruption.
Posted by WII-Aaron, 02-25-2012, 06:07 PM

Quote:

Originally Posted by subigo

Honestly, I would have rather just had a scheduled outage without trying to run the generator at all at this point. At least that way I could have cleanly shutdown my servers and not worried about data corruption.

You and me both.

Now we've sprung a coolant leak causing the generator to overheat. I had my techs haul water jugs from the water cooler down to the generator to refill the radiator and the gen tech is patching the hole.
Posted by pubcrawler, 02-25-2012, 06:07 PM
Yep, running from generators like this is rather risky. I'd never want to be involved in such a thing on site at this scale.

Mind you in other countries, they run whole towns on rickety old gensets.

Has to be issues with this up down cycling and massive surge load. Servers are big initial surge load.

My bet is on the electrician and the nail polish hackery. Temporary fix might have went and undid itself or worse. The generators have been running oh 8+ hours at this point, so anything is possible.

And... *PRESTO* we are back online in KC.
Posted by WII-Aaron, 02-25-2012, 06:09 PM

Quote:

Originally Posted by pubcrawler

Aaron is the man behind the curtain for sure. Big hand clap for him finding the time to update folks... It's unexpected, but very proactive.

Everyone down again too?

I appreciate that however I sure do wish there were someone else behind the curtain today.

Posting these updates from my phone.
Posted by subigo, 02-25-2012, 06:15 PM

Quote:

Originally Posted by WII-Aaron

You and me both.

Now we've sprung a coolant leak causing the generator to overheat. I had my techs haul water jugs from the water cooler down to the generator to refill the radiator and the gen tech is patching the hole.

I vote for just shutting down everything, taking a nap, and waiting for power to be restored to the building. Who's with me?
Posted by pubcrawler, 02-25-2012, 06:19 PM
Naps, that would be nice.

But I can't find my blanky and binky.
Posted by lamnea, 02-25-2012, 06:21 PM
Had an email about colorhost doing an power upgrade (was a client of them a while back) apparently a generator they were using whilst the main power was down failed.

Maybe the same datacenter?
Posted by chasebug, 02-25-2012, 06:26 PM
My databases got corrupted! How do I fix this?
Posted by cd/home, 02-25-2012, 06:28 PM

Quote:

Originally Posted by chasebug

My databases got corrupted! How do I fix this?

Restore from your upto date backups?

Am just amazed at all the problems they are having with these generators, Wasnt they ever tested for long haul jobs like this or simply on the hopes they would never have to run for some 12 hours.

Ive seen/had several generators running 24 hours for 7 days solid without any hiccups also a massive rolls royce 1000+ killowatt generator running for some 12 hours per day without any issues either...
Posted by Patrick, 02-25-2012, 06:31 PM

Quote:

Originally Posted by chasebug

My databases got corrupted! How do I fix this?

Does myisamchk make a difference?

http://www.thegeekstuff.com/2008/09/...ing-myisamchk/
Posted by pubcrawler, 02-25-2012, 06:36 PM
Whole lot of folks are going to have MySQL corruption. MySQL needs to fix their d@mn software already to be more graceful about power failures.

This command will work, given you don't have an enormous database:
mysqlcheck -u root -p --auto-repair --check --optimize --all-databases
Posted by chasebug, 02-25-2012, 07:21 PM
I used phpmyadmin to repair database and it is fine now. The other thing this screwed up is PayPal IPN, I got a couple rejected payments because the server after power reboot rolled back 20 minutes in time. Everybody should check the server time make sure it is correct.

I just took 1 more backup of the repaired database.
Posted by Patrick, 02-25-2012, 07:27 PM

Quote:

Originally Posted by chasebug

I used phpmyadmin to repair database and it is fine now. The other thing this screwed up is PayPal IPN, I got a couple rejected payments because the server after power reboot rolled back 20 minutes in time. Everybody should check the server time make sure it is correct.

I just took 1 more backup of the repaired database.

You should setup a cron job to make a backup of the database and email it to you every night.
Posted by chasebug, 02-25-2012, 07:35 PM
I have R1SOFT MySQL but haven't got to setting it up yet, seems difficult.
Posted by WII-Aaron, 02-25-2012, 08:52 PM
Utility power has been restored. We have cut back from generator.

If you have equipment that's down please submit a ticket. We have several additional techs on hand to clean up any messes.

I'd like to say a special thank you to AT&T who just threatened to call the EPA for my coolant spill. I'll get right on that Mr. I have an unpermitted generator sitting in the street with 2 extention cords strung across the sidewalk.
Posted by chasebug, 02-25-2012, 08:53 PM
It is almost 7PM CST the maintenance at the building is finished? Is it now running on power instead of generator?
Posted by cd/home, 02-25-2012, 08:59 PM

Quote:

Originally Posted by WII-Aaron

I'd like to say a special thank you to AT&T who just threatened to call the EPA for my coolant spill. I'll get right on that Mr. I have an unpermitted generator sitting in the street with 2 extention cords strung across the sidewalk.

Eh?
Posted by F-DNS, 02-25-2012, 09:13 PM

Quote:

Originally Posted by cd/home

Eh?

LOL. I suspect Aaron is contemplating shoving the hot ends of those 2 extension cords somewhere where the AT&T dude's sun don't shine
Posted by WII-Aaron, 02-25-2012, 09:35 PM

Quote:

Originally Posted by chasebug

It is almost 7PM CST the maintenance at the building is finished? Is it now running on power instead of generator?

Yes. It is finished.
Posted by WII-Aaron, 02-25-2012, 09:38 PM

Quote:

Originally Posted by F-DNS

LOL. I suspect Aaron is contemplating shoving the hot ends of those 2 extension cords somewhere where the AT&T dude's sun don't shine

I was probably a little off topic. Yes. nerves are a little frayed around here. For everyone. No one in the building got out unscathed. We all had issues someplace.

I just couldn't believe the AT&T tech would stop me and whine about that with everything we've all been through today.
Posted by cd/home, 02-25-2012, 09:38 PM

Quote:

Originally Posted by WII-Aaron

Yes. It is finished.

What was they doing exactly for the power to be turned off for that long?
Posted by WII-Aaron, 02-25-2012, 09:46 PM

Quote:

Originally Posted by cd/home

What was they doing exactly for the power to be turned off for that long?

They were load testing, cleaning and recertifying the breakers in the basement.
Posted by cd/home, 02-25-2012, 10:04 PM

Quote:

Originally Posted by WII-Aaron

recertifying the breakers in the basement.

Do they do this every year or so then?
Posted by rajapatr1, 02-26-2012, 12:30 AM
I don't receive your email sent to me now my server come back and hdd was problem so bad for me it problem on data hdd
Posted by mpkapadia, 02-26-2012, 02:22 AM
I have a server that is still not accessible . Aaron if you are reading this then my Ticket id is 157665
Posted by mpkapadia, 02-26-2012, 03:16 AM
Anybody else is still down or its just me . I am quite sure that we have not even recieved any advance notification of the scheduled outage .

Tried to contact them on phone / Pager Email id . But looks like noone available right now.
Posted by Ilya74, 02-26-2012, 03:40 AM
They responded to your ticket?
Posted by mpkapadia, 02-26-2012, 03:53 AM
They havent responded as yet
Posted by Ilya74, 02-26-2012, 03:55 AM
If much time has passed, try to open a new emergency ticket.
Posted by mpkapadia, 02-26-2012, 03:59 AM
Just got an update a few minutes back from Bob Region that they will try to get to my server asap.

Just hoping for services to be back online

I had already sent a couple of mails to their Pager Id. So their emergency support staff should also be knowing about the issue.
Posted by mpkapadia, 02-26-2012, 06:21 AM
4 hours into my emergency ticket they are still waiting for a server crash cart with Keyb / Monitor to check what is the status of machine .

I dont know if they have had a lot of dead servers tonight due to this power issue or are just plain unstaffed this night . But its very frustrating to not know what is happening with our server.

I am hoping WII-Aaron sees this post and can do something to expedite things .
That i cannot call any phone number to find any status update also does not help.
Posted by mpkapadia, 02-26-2012, 06:57 AM
Phew , Back online now . A silly reason like pressing one F1 key on the keyboard resulted in this downtime .

The fact that reason was so silly it sucks.
But i am happy now that its back online
Posted by pubcrawler, 02-26-2012, 08:00 AM
AT&T needs to go jump of a nearby bridge - the entire company. They've done plenty of misdeeds over the years. Tell neb-nose to go spend his energy lashing his employer instead of the independent hard working small business man.

Pitiful little creature and his coolant enviro-toxin over-reaction. GROWL!