Gigenet Outage?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Gigenet Outage?

Posted by Curious Too, 10-09-2013, 04:35 PM
I have 13 servers at Gigenet that are unreachable:

tracert manage.gigenet.com

Tracing route to manage.gigenet.com [69.65.30.3]
over a maximum of 30 hops:

1 3 ms 1 ms 2 ms 192.168.1.254
2 10 ms 9 ms 9 ms adsl-99-189-61-254.dsl.sfldmi.sbcglobal.net [99.
189.61.254]
3 11 ms 9 ms 9 ms 67.36.65.2
4 * * * Request timed out.
5 * 12.83.32.161 reports: Destination net unreachable.

Trace complete.

Anyone else having problems with Gigenet?
Posted by afallison, 10-09-2013, 04:38 PM
same here. i tried calling the toll free number and there was no answer. prob because they're using voip..
Posted by Scott.Mc, 10-09-2013, 04:39 PM
Same we have servers down also.
Posted by gtithesexy, 10-09-2013, 04:41 PM
same our servers down
Posted by gtv6, 10-09-2013, 04:42 PM
same here
Posted by marcosv, 10-09-2013, 04:44 PM
My servers are also down.
Their website also does not work this time.

PING gigenet.com (69.65.17.3) 56(84) bytes of data.

--- gigenet.com ping statistics ---
15 packets transmitted, 0 received, 100% packet loss, time 14011ms
Posted by xnpu, 10-09-2013, 04:44 PM
Route is gone completely so I'm guessing it's a network issue and our servers are fine, just lonely perhaps.
Posted by GigenetKristin, 10-09-2013, 04:48 PM
Hello we have an outage. We are aware of it. We are all scrambling to fix things as fast as possible for you.

We apologize for the inconvenience and will have answers for you as soon as we have some more details regarding what has happened.

For now please bear with us as our entire staff is doing their best to get you back online.
Posted by the_pm, 10-09-2013, 04:48 PM
We're completely offline as well.
Posted by Tmonster, 10-09-2013, 04:52 PM
Just called them. They have some problem with the electrical equipment which caused the fire alarm and fire department forced them to shut the electricity down...They hope to get everything back in order in 50 minutes.
Posted by GigenetKristin, 10-09-2013, 04:54 PM
Hello WHT!

We are experiencing an outage. We are working on getting everything back up as soon as possible.

Thank you for your understanding.
Posted by MrTony, 10-09-2013, 04:56 PM
Outages might happen with any network, the good point is someone from gigenet already answered in this post.
Posted by Tmonster, 10-09-2013, 05:01 PM
Right I agree. But it makes no sense to me now to use their Cloud solution because it is in fact "Cloud" within a single DC. Even their site and ticket system is down. Worse putting the ticket system to Amazon.
Posted by proclanh, 10-09-2013, 05:02 PM
We have several servers down there right now as well but there support has always been top notch with us. More than I want to say than other providers we used in Chicago in the past. Thanks for the quick response Kristin!
Posted by xnpu, 10-09-2013, 05:02 PM

Quote:

Originally Posted by Tmonster

Right I agree. But it makes no sense to me now to use their Cloud solution because it is in fact "Cloud" within a single DC.

You'll find this to be true for most clouds. Even if you use something like Amazon you have to specifically create instances on multiple clouds to get redundancy across DC's.
Posted by SeriesOfTubes, 10-09-2013, 05:03 PM
I have seven severs at GigeNET and have been a long time client of theirs.

My question is, how can a datacenter with so called redundant power just have to "shut off everything"? What about the on-site generators? Any competent datacenter would have switched to another power source.

Isn't gigenet performing regular power redundancy checks with their generators to verify that a crisis like this would be averted?

I can go to any number of data centers in this area and not have to worry about losing contact with my servers because they had to "shut down everything".

This is pathetic-- I demand a more technical response to this issue. Why hasn't ameen posted as a response to this thread?
Posted by xnpu, 10-09-2013, 05:04 PM
As far as I understand the power didn't fail. They were instructed to shut it down because of smoke.
Posted by MrTony, 10-09-2013, 05:05 PM

Quote:

Originally Posted by Tmonster

Right I agree. But it makes no sense to me now to use their Cloud solution because it is in fact "Cloud" within a single DC. Even their site and ticket system is down. Worse putting the ticket system to Amazon.

In this case try to use a CDN service.
Posted by Tmonster, 10-09-2013, 05:08 PM

Quote:

Originally Posted by MrTony

In this case try to use a CDN service.

We DO use CDN from NetDNA and Cloudflare but the DB servers are on Gigenet Cloud. As well as DNS for the domain name points to their origin servers We have no other choice, but wait.
Posted by freddyh, 10-09-2013, 05:11 PM

Quote:

Originally Posted by GigenetKristin

Hello WHT!

We are experiencing an outage. We are working on getting everything back up as soon as possible.

Thank you for your understanding.

I'm a little concerned here because you gave and ETA then removed it.

Also if you're turning power back on now, I don't understand why that would take 2 hours.

Can you state if the 2 hour ETA you gave a moment ago is correct? And if indeed power is going back on now?

Also I'd like to know if any of the sprinklers went off.
Posted by MrTony, 10-09-2013, 05:14 PM

Quote:

Originally Posted by Tmonster

We DO use CDN from NetDNA and Cloudflare but the DB servers are on Gigenet Cloud. As well as DNS for the domain name points to their origin servers We have no other choice, but wait.

Yes its terrible to have the services down, in this case I really recommend to have a cloud in other datacenter too, so in extreme cases like this you dont have a long downtime.

It will increase a little your budget but sometimes its very necessary to prevent losses.
Posted by Steven, 10-09-2013, 05:14 PM

Quote:

Originally Posted by GigenetKristin

Hello WHT!

We are experiencing an outage. We are working on getting everything back up as soon as possible.

Thank you for your understanding.

What happened to the talk of smoke?
Posted by GigenetKristin, 10-09-2013, 05:15 PM

Quote:

Originally Posted by SeriesOfTubes

I have seven severs at GigeNET and have been a long time client of theirs.

My question is, how can a datacenter with so called redundant power just have to "shut off everything"? What about the on-site generators? Any competent datacenter would have switched to another power source.

Isn't gigenet performing regular power redundancy checks with their generators to verify that a crisis like this would be averted?

I can go to any number of data centers in this area and not have to worry about losing contact with my servers because they had to "shut down everything".

This is pathetic-- I demand a more technical response to this issue. Why hasn't ameen posted as a response to this thread?

We have UPS units and generators.

They are checked on a daily basis. The generator is tested on a monthly basis.

Ameen has not posted in this thread because he is working on getting you back online.
Posted by keliix06, 10-09-2013, 05:19 PM
Just so you guys at Gigenet know, removing/editing posts does get noticed by us. It's not helping the situation.
Posted by the_pm, 10-09-2013, 05:20 PM
We relayed an ETA of 2 hours to our clients immediately after you gave us this information. Can you tell us if we need to retract this statement? Please let us know - it's a matter of credibility between us and our clients. Thank you!
Posted by GigenetKristin, 10-09-2013, 05:22 PM
Sorry for the inconvenience. We are working on getting everyone back up as soon as possible. Some customers should be back up shortly, and others will back online in the next hour or two.
Posted by freddyh, 10-09-2013, 05:23 PM

Quote:

Originally Posted by the_pm

We relayed an ETA of 2 hours to our clients immediately after you gave us this information. Can you tell us if we need to retract this statement? Please let us know - it's a matter of credibility between us and our clients. Thank you!

I've been telling my clients they were turning power back on, now I see that's been edited.
Posted by freddyh, 10-09-2013, 05:26 PM

Quote:

Originally Posted by GigenetKristin

Sorry for the inconvenience. We are working on getting everyone back up as soon as possible. Some customers should be back up shortly, and others will back online in the next hour or two.

Was equipment hard powered off? If so there is likely some that will not come up on its own and/or have drive errors.
Posted by GigenetKristin, 10-09-2013, 05:32 PM

Quote:

Originally Posted by freddyh

Was equipment hard powered off? If so there is likely some that will not come up on its own and/or have drive errors.

There was a shutdown process, we will go from there.

All systems should have been shutdown properly before the electricity was turned off.
Posted by tamouh, 10-09-2013, 05:34 PM
We've few servers at Gigenet and their support/customer relation has been superior. I'm sure this incident will make us reconsider them depending on the cause and how well they can handle it.

Single DCs do go down, but to have a smoke and a fire then all power shutdown. What happened to fire suppression systems and redundant power?
Posted by Steven, 10-09-2013, 05:35 PM

Quote:

Originally Posted by GigenetKristin

There was a shutdown process, we will go from there.

All systems should have been shutdown properly before the electricity was turned off.

What? How long did that take? How did you do this? Everything seem to die at the same time.
Posted by freddyh, 10-09-2013, 05:39 PM
Can you PLEASE give us some kind of running status here?

Your making these random ambigous statements evading questions while editing your posts, which isn't helping this situation.
Posted by cb1000rider, 10-09-2013, 05:41 PM

Quote:

Originally Posted by GigenetKristin

There was a shutdown process, we will go from there.

All systems should have been shutdown properly before the electricity was turned off.

I think that's *very unlikely*. We have monitoring off all of our servers in that datacenter and they all went dark almost at once.

How many machines do you have there? How long would it take to power them all down one at a time?
Posted by freddyh, 10-09-2013, 05:42 PM

Quote:

Originally Posted by Steven

What? How long did that take? How did you do this? Everything seem to die at the same time.

Agreed, I don't even think they have the login info for most of the servers to power them down safely.
Posted by layer0, 10-09-2013, 05:45 PM
What's with all the evasive answers?

Where's Ameen?

Where's Chris?
Posted by freddyh, 10-09-2013, 05:45 PM

Quote:

Originally Posted by layer0

What's with all the evasive answers?

Where's Ameen?

Where's Chris?

They're prolly not online, if the power is off there.
Posted by cb1000rider, 10-09-2013, 05:47 PM
Someone at Gigenet is also deleting tweets relating to fire/smoke.

Look, be honest or don't answer.... We need to know what to expect in order to prepare to bring systems back.

Remember your 100% SLA!
Posted by Tmonster, 10-09-2013, 05:51 PM
Ameen and Chris are in the facility and working to get things back. I don't think it would be better if they come here and post instead of fixing things first.

However, somebody from Gigenet might post real time updates here so we won't sit in the darkness and wait.
Posted by freddyh, 10-09-2013, 05:53 PM

Quote:

Originally Posted by Tmonster

However, somebody from Gigenet might post real time updates here so we won't sit in the darkness and wait.

Agreed, and on twitter it seems they're mitigating this saying "just a smoke issue". And shifting the blame to the fire department, saying they made them shut off their power.
Posted by benj114, 10-09-2013, 05:53 PM

Quote:

Originally Posted by tamouh

Single DCs do go down, but to have a smoke and a fire then all power shutdown. What happened to fire suppression systems and redundant power?

My guess is the fire department said turn off all power.

You can have all the redundant systems and fire suppression systems, but when the fire department says turn off all power, you have no say in the matter.

The life of fire fighter(s) is not worth a servers uptime...
Posted by gtv6, 10-09-2013, 05:55 PM

Quote:

Originally Posted by cb1000rider

Someone at Gigenet is also deleting tweets relating to fire/smoke.

its looks more like clarifying the situation to me https://twitter.com/gigenet

anyway I agree, it'd be nice to have a more solid update on their procedure to recover from this so we know what to expect
Posted by GigenetKristin, 10-09-2013, 05:56 PM

Quote:

Originally Posted by cb1000rider

Someone at Gigenet is also deleting tweets relating to fire/smoke.

Look, be honest or don't answer.... We need to know what to expect in order to prepare to bring systems back.

Remember your 100% SLA!

Update: The smoke alarms went off for a brief time in our data center, but there was no fire in the building. All systems were properly shut down prior to turning off the power to the building. As of right now, we are still estimating a one to two hour downtime as we begin to start all the servers back up in phases. We will continue to update as we get more details.
Posted by Steven, 10-09-2013, 05:57 PM

Quote:

Originally Posted by GigenetKristin

Update: The smoke alarms went off for a brief time in our data center, but there was no fire in the building. All systems were properly shut down prior to turning off the power to the building. As of right now, we are still estimating a one to two hour downtime as we begin to start all the servers back up in phases. We will continue to update as we get more details.

Please explain how you shut down systems you don't have login details for, I need to know what kind of damage control I am looking at.
Posted by the_pm, 10-09-2013, 05:57 PM
Take their story at face value folks. No DC would risk putting a false story about smoke, fire dept. and forced shutdown into the wild if it wasn't true. These sorts of things are all public record; you don't spin a lie like this, and for all the years I'm known Ameen, I've never know him to spin a lie or authorize someone else to do so. Let them work, and sort out the RFO later.
Posted by Tmonster, 10-09-2013, 06:08 PM

Quote:

Originally Posted by the_pm

Take their story at face value folks. No DC would risk putting a false story about smoke, fire dept. and forced shutdown into the wild if it wasn't true. These sorts of things are all public record; you don't spin a lie like this, and for all the years I'm known Ameen, I've never know him to spin a lie or authorize someone else to do so. Let them work, and sort out the RFO later.

Paul I completely agree. I just don't understand why it would take 1-2 hours to put things back...
I fully trust Ameen and Chris they've always been straightforward and honest partners. We just need more clarity and transparency.

I can't imagine what other customers think if they don't know about this forum and don't read their twitter... They might think this is a beginning of US Government shut down
Posted by gigenet_customer, 10-09-2013, 06:08 PM

Quote:

Originally Posted by Steven

Please explain how you shut down systems you don't have login details for, I need to know what kind of damage control I am looking at.

Linux machines generally enter graceful shutdown if the power button is pressed, so running through the DC tapping power buttons could get down hundreds of servers from a single person in just a couple minutes.

That being said, I was actually SSH'd in to one of my machines doing a Debian 6 -> 7 update and I did not see any console message indicating a graceful shutdown. I had 2 different SSH consoles opened and as far as I can tell, it was a hard shutdown. Thankfully in my case, the actual package installation hadn't begun, so provided I don't have any hardware or file system errors, I expect no major problems if/when the servers come back up. That being said, I have to seriously question the information being put out by gigenet when it's clear they didn't perform a graceful shutdown on my particular server.

Probably I'll be called a competitor or troll or whatever because this is a new account, but I don't normally post on forums and only reason I bothered coming to WHT was because I found a link to this thread on twitter when trying to find details of the outage.

I've been fairly happy with gigenet's quality of service, although one of their techs that has responded to a couple of my tickets was beyond incompetent, most are extremely helpful and knowledgeable. I don't intend to move any of my servers from gigenet due to this incident. Bad things happen at all data centers, including Amazon's cloud, but if I find out that the info put out was false(aside from the shutdowns as my server could have been missed by accident), then I'll have to reconsider.

Edit: I have dedicated servers with gigenet (I do not use cloud or any other services). I know both of my servers have IPMI, so I suppose for those servers they may have a way to send a mass shutdown via IPMI.
Posted by xnpu, 10-09-2013, 06:08 PM

Quote:

Originally Posted by the_pm

Take their story at face value folks. No DC would risk putting a false story about smoke, fire dept. and forced shutdown into the wild if it wasn't true. These sorts of things are all public record; you don't spin a lie like this, and for all the years I'm known Ameen, I've never know him to spin a lie or authorize someone else to do so. Let them work, and sort out the RFO later.

I don't think anyone doubts that the fire dept. shut them down.

What gets people doubting the story is:
* Removing information from previous posts, rather that just posting a correction if one is needed.
* Claiming there was time to shut everything down properly, but not to send out an e-mail or tweet.
Posted by Steven, 10-09-2013, 06:11 PM

Quote:

Originally Posted by gigenet_customer

Linux machines generally enter graceful shutdown if the power button is pressed, so running through the DC tapping power buttons could get down hundreds of servers from a single person in just a couple minutes.

Sure. Most of the time yes, but in some cases no.

Quote:

That being said, I was actually SSH'd in to one of my machines doing a Debian 6 -> 7 update and I did not see any console message indicating a graceful shutdown. I had 2 different SSH consoles opened and as far as I can tell, it was a hard shutdown. Thankfully in my case, the actual package installation hadn't begun, so provided I don't have any hardware or file system errors, I expect no major problems if/when the servers come back up. That being said, I have to seriously question the information being put out by gigenet when it's clear they didn't perform a graceful shutdown on my particular server.

If it was shutdown, it was done after the network was killed because everything dropped at the same time.
Posted by freddyh, 10-09-2013, 06:11 PM

Quote:

Originally Posted by gigenet_customer

Linux machines generally enter graceful shutdown if the power button is pressed, so running through the DC tapping power buttons could get down hundreds of servers from a single person in just a couple minutes.

Most servers have a 4 sec override to protect from accidental shutdown. After holdling in the power button for 4 secs linux will NOT gracefully shut down.
Posted by cb1000rider, 10-09-2013, 06:12 PM

Quote:

Originally Posted by xnpu

* Claiming there was time to shut everything down properly, but not to send out an e-mail or tweet.

I need details on this point. It was tweeted and posted here.
From our perspective, we went black across the board at one time.

I understand under the threat of smoke/fire if they have to hard power down machines. That makes perfect sense to me. However, indicating graceful shutdown changes our procedures on recovery... We need to be 100% sure. And we'll be able to tell via logs.

No phone calls. No one answered the phone. The should have someone in corporate office calling customers immediately.
Posted by Steven, 10-09-2013, 06:13 PM

Quote:

Originally Posted by freddyh

Most servers have a 4 sec override to protect from accidental shutdown. After holdling in the power button for 4 secs linux will NOT gracefully shut down.

Yeah this is a bios setting.
Posted by WII-Aaron, 10-09-2013, 06:16 PM

Quote:

Originally Posted by Tmonster

Paul I completely agree. I just don't understand why it would take 1-2 hours to put things back...

If you have a panel catch on fire and have to replace it. That could take hours, best case. Days if you can't source a new one locally and depending on the size of the panel, a lot of them are custom made.
Posted by proclanh, 10-09-2013, 06:21 PM

Quote:

Originally Posted by WII-Aaron

If you have a panel catch on fire and have to replace it. That could take hours, best case. Days if you can't source a new one locally and depending on the size of the panel, a lot of them are custom made.

I also do not think any DC will go in and turn everything on at once, just imagine the power surge which could cause more issues.
Posted by sam0, 10-09-2013, 06:22 PM

Quote:

Originally Posted by WII-Aaron

If you have a panel catch on fire and have to replace it. That could take hours, best case. Days if you can't source a new one locally and depending on the size of the panel, a lot of them are custom made.

There was no fire:

Quote:

Originally Posted by GigenetKristin

Update: The smoke alarms went off for a brief time in our data center, but there was no fire in the building. All systems were properly shut down prior to turning off the power to the building. As of right now, we are still estimating a one to two hour downtime as we begin to start all the servers back up in phases. We will continue to update as we get more details.

Posted by cb1000rider, 10-09-2013, 06:22 PM
I understand that they need to take fast action if facing a possible electrical fire. I just need clarification on the shut down procedure, as it alters our recovery steps.

At a minimum, we'll have systems that go into disk check if hard powered down. Others will be in unknown states and databases may need recovery.

If shut down properly, this isn't an issue.

We still haven't received a phone call from Gigenet in terms of notification. I understand technical staff is busy, but they have additional staff in other departments.
Posted by Tmonster, 10-09-2013, 06:23 PM

Quote:

Originally Posted by WII-Aaron

If you have a panel catch on fire and have to replace it. That could take hours, best case. Days if you can't source a new one locally and depending on the size of the panel, a lot of them are custom made.

Nothing was on fire...it was just a smoke that caused the alarm. So the downtime as I understand was demanded by the fire dept. Now they are plugging everything back, so my question was why it takes 2 hours to plug it back?
Posted by GigenetKristin, 10-09-2013, 06:23 PM
Update: This issue is only effecting our Chicago data center. Our Los Angeles data center is fully operational.
Posted by layer0, 10-09-2013, 06:26 PM

Quote:

Originally Posted by GigenetKristin

Update: This issue is only effecting our Chicago data center. Our Los Angeles data center is fully operational.

Cool, nice "update" - glad you just figured that out!
Posted by freddyh, 10-09-2013, 06:27 PM

Quote:

Originally Posted by GigenetKristin

Update: This issue is only effecting our Chicago data center. Our Los Angeles data center is fully operational.

That wasn't really the update we were hoping for.
Posted by gigenet_customer, 10-09-2013, 06:30 PM

Quote:

Originally Posted by cb1000rider

I understand that they need to take fast action if facing a possible electrical fire. I just need clarification on the shut down procedure, as it alters our recovery steps.

At a minimum, we'll have systems that go into disk check if hard powered down. Others will be in unknown states and databases may need recovery.

If shut down properly, this isn't an issue.

We still haven't received a phone call from Gigenet in terms of notification. I understand technical staff is busy, but they have additional staff in other departments.

Unless you can be 100% sure of a clean shutdown, the only safe course of action is to proceed as if it was a hard shut down. Even if gigenet had intended to do a clean shutdown of all servers, it's possible that yours took an extended period of time for whatever reason, such that power was killed prior to the clean shutdown completing. Therefore, even if they can confirm that they had initiated the clean shutdown, that isn't enough to assume that clean shutdown actually occurred.
Posted by freddyh, 10-09-2013, 06:46 PM
Kristen, can we please have an update here?

Its been 2 hours since you initially gave an ETA of 2 hours and you said you started turning things back on.

gigenet.com is still down, and I'd hope that would be the first site you'd turned back on, so it seems nothing is happening.
Posted by xnpu, 10-09-2013, 06:49 PM

Quote:

Originally Posted by freddyh

Kristen, can we please have an update here?

Its been 2 hours since you initially gave an ETA of 2 hours and you said you started turning things back on.

gigenet.com is still down, and I'd hope that would be the first site you'd turned back on, so it seems nothing is happening.

Their webserver may be up and running already. As far as I can tell their routes aren't being announced. So until they get their router(s) back up we won't see thing.

EDIT: whoops, and there it is! (route, not website)
Posted by Steven, 10-09-2013, 06:53 PM

Quote:

Originally Posted by xnpu

Their webserver may be up and running already. As far as I can tell their routes aren't being announced. So until they get their router(s) back up we won't see thing.

EDIT: whoops, and there it is! (route, not website)

where are you seeing a route from?

edit:
I see routes are hitting chicago now:

Quote:

Host Loss% Snt Last Avg Best Wrst StDev
1. 204.11.61.78 0.0% 144 0.0 0.0 0.0 0.0 0.0
2. border2.te8-2.ionity-3.dal006.pn 0.0% 144 0.3 4.1 0.2 219.4 25.5
3. core1.pc2-bbnet2.ext1a.dal.pnap. 0.0% 144 1.4 1.4 1.3 1.7 0.1
4. 207.88.185.73.ptr.us.xo.net 0.0% 144 1.1 8.1 1.0 233.3 32.2
5. vb1001.rar3.chicago-il.us.xo.net 0.0% 144 31.4 33.1 28.5 60.9 6.4
6. ae0d0.mcr1.chicago-il.us.xo.net 0.0% 144 27.5 29.8 27.4 90.0 8.3
7. ae1d0.mcr2.chicago-il.us.xo.net 0.0% 144 27.2 29.8 27.1 103.8 10.2
8. ???

Posted by tamouh, 10-09-2013, 06:58 PM
We are two hours into the outage, what is the update? I still see Gigenet site down and our servers down. Anyone has anything come up for them?
Posted by Walk5000MilesAnHour, 10-09-2013, 06:59 PM
I use or have used dozens of hosts, big and small. I don't hesitate to blast those hosts who have downtime due to incompetence, ignorance, etc., so I understand the temptation. But Gigenet has been a juggernaut of reliability for me, and I didn't even really realize how much so until this happened.

So far, all of the information presented seems plausible.

Smoke, even a small amount, setting off a fire alarm; plausible.

Smoke without fire; plausible - happens all the time (despite the aphorism).

Fire department insisting that power must be shut down; plausible. (Arlington Heights is a paid department, not a volunteer one - that makes a difference.)

1-2 hour estimated downtime (maybe even a few more hours); plausible. (Making such a public announcement if they, otherwise, KNEW it would take days would be a marketing disaster that would, expensively, taint their reputation for years to come.)

A complete power shutdown affecting their phones functioning; plausible.

It also affecting routing; plausible. (If their routers can't do "BGP advertising", route-to information can go out to/from a multi-homed environment.)

Having all of their technical staff working solely on this situation; plausible - obviously.

I routinely have so-called "hard shutdowns" all the time, particularly on my home system. All that a "clean shutdown" means that all open files are closed prior to power-off. I've had crappy servers, and still never had any disaster result from a hard shutdown. 'fsck' (or whatever the windows equivalent of that is) takes care of that in a few minutes.

Now, I'm as crippled and adversely affected by this downtime as anyone. Frustrated, yes, but there's no one to blame, except whomever manufactured whatever part smoked.
Posted by Steven, 10-09-2013, 07:01 PM
And a server has come to life.
Posted by Steven, 10-09-2013, 07:04 PM
For the record, it was not shut down cleanly.
Safe to assume people, you need to handle this like a hard shutdown situation.
Posted by the_pm, 10-09-2013, 07:10 PM
Let's stick to discussions about the outage and use this thread for Gigenet to reach clients, as is the purpose of this forum. Posts about Gigenet's business affairs are not appropriate in this thread.

<< a little cleanup completed >>
Posted by xnpu, 10-09-2013, 07:17 PM
Back up here too. No clean shutdown. Downtime 2 hours and 16 minutes.
Posted by GigenetKristin, 10-09-2013, 07:20 PM
Update: We have been turning clients back on at a measured pace for the last hour, and should have all of our services back up and running shortly. Please be patient with us, as we have to bring everyone back online in small groups. We will continue to update as we bring everything up. Thank you
Posted by nero16, 10-09-2013, 07:28 PM
If fire department request a full shutdown. You got no choice... Once they at the scene, I assume they take full control. Nor gigenet or ur mum can make a difference her.

Now that said... The two hours to put things back up does sound very long... Now am not sure if we talk about 1000 or 10000 servers... But I know for the fact, it take minutes to break a wall.. Years to build.
Aka... Shutting down forcefully no big deal.... But putting servers back online has to be done controlled and limited at the time. To avoid burned down hardware fuse etc.

Now the bad stuff.

What the heck?????
No failoverfor gigenet website??? They own multiple DC.... How can their main site go down.

Even worst their ticket system

And even worst than that... Their blog.....

The very first think should be a page redirect with status update.
Gigenet failed here... They failed to reach their customers!

I am not worried about few hours downtime... Stuff like this happens.... That's why I have my own precautions.... Gigenet is still a great company.... I am sure gigenet will take home a lot of knowledgeable information from this experience and it will only enhance their future major outage.

No point pointing fingers at them.... Am sure, these guys know what they do. Mistake happens... But when did we see a major outage last from these guys?
Posted by kamal_s, 10-09-2013, 07:41 PM
still down here.. multiple servers
Posted by tamouh, 10-09-2013, 07:45 PM
We are back up. Thank you GigeNet
Posted by kamal_s, 10-09-2013, 07:49 PM
Coincidentally one of largest telecom in Canada Rogers facing cellular network outtage over an hr now.. perfect timing
Posted by double76, 10-09-2013, 07:51 PM
still not up, more than 3 hours. don't know how long to wait. quite depressed.
Posted by NetDepot - Terrence, 10-09-2013, 07:52 PM

Quote:

Originally Posted by kamal_s

Coincidentally one of largest telecom in Canada Rogers facing cellular network outtage over an hr now.. perfect timing

cellular network outage? where? I have rogers and my service is fine.
Posted by kamal_s, 10-09-2013, 07:54 PM

Quote:

Originally Posted by NetDepot - Terrence

cellular network outage? where? I have rogers and my service is fine.

http://canadianoutages.com/status/rogers
Posted by gtv6, 10-09-2013, 08:07 PM

Quote:

Originally Posted by double76

still not up, more than 3 hours. don't know how long to wait. quite depressed.

I was thinking the same then BAM, it came back up, and even better all data seems intact, so that's something... hopefully the same happens for you soon
Posted by SRobbins, 10-09-2013, 08:18 PM
Anyone know of alternatives? We need a Chicago area Cloud server that uses advanced routing like Internap. That was Gigenet. They not only had advance routing but also have internap.

On of our clients that uses our SaaS is on a deadline and fuming that we are down. We need to get our software up and on something else right now till Gigenet is back up. Anyone know of some good providers?

So disappointing, in the 5 years we have been operating out of 4 major US cities, we have never been down from a provider till now.
Posted by GigenetKristin, 10-09-2013, 08:25 PM
We're working with haste to get everyone back online.

If you're not online yet you will be shortly.

Every member of our staff is here checking each machine one at a time to make sure it powers back on and pings.

I will continue to keep you updated with more information as I receive it.
Posted by Steven, 10-09-2013, 08:31 PM
http://lexiconnstatus.com/

Quote:

Wednesday, Oct. 9th, 4:54 PM ET - The main UPS for the datacenter had a failure that caused smoke and fire alarms to sound. The fire department forced the datacenter to cut power and evacuate as a precaution. Preps are being made now to resume power and begin restoring networking and services. We do not have an ETA, but would expect things to be restored in the next hour to 90 minutes.

Posted by afallison, 10-09-2013, 09:00 PM
I can finally ping my gateway but my ip is still not reachable...
Posted by SRobbins, 10-09-2013, 09:04 PM
I can see my server listed as "On" in my panel but I'm getting "Network error: remote side closed connection" when I try to use the web console. And RDP just wont connect.
Posted by afallison, 10-09-2013, 09:05 PM

Quote:

Originally Posted by SRobbins

I can see my server listed as "On" in my panel but I'm getting "Network error: remote side closed connection" when I try to use the web console. And RDP just wont connect.

Same here. I had the same issue before and I had to call them to make it work for me. Should be fun trying to get hold of them tonight...
Posted by SRobbins, 10-09-2013, 09:08 PM

Quote:

Originally Posted by afallison

Same here. I had the same issue before and I had to call them to make it work for me. Should be fun trying to get hold of them tonight...

What is their number? I only have one and its not connecting(no ring or busy signal, just beeping like a fax machine), and being that their wordpress powered .com site is broken... I cant get anything else.
Posted by afallison, 10-09-2013, 09:10 PM
I just pulled a cached copy from google to get the numbers:

1-800-908-7115
1-224-265-9165

I have no idea if there is a better one to reach them at...
Posted by sfnz, 10-09-2013, 09:13 PM

Quote:

Originally Posted by SRobbins

What is their number? I only have one and its not connecting(no ring or busy signal, just beeping like a fax machine), and being that their wordpress powered .com site is broken... I cant get anything else.

1-800-561-2656 is the number they've advertised on Twitter during the outage.
Posted by SRobbins, 10-09-2013, 09:21 PM

Quote:

Originally Posted by afallison

I just pulled a cached copy from google to get the numbers:

1-800-908-7115
1-224-265-9165

I have no idea if there is a better one to reach them at...

Quote:

Originally Posted by sfnz

1-800-561-2656 is the number they've advertised on Twitter during the outage.

1-800-561-2656 - No answer with a "Mailbox is full" message.
1-800-908-7115 - Goes to a message to "send a fax, or leave a message."
1-224-265-9165 - Busy signal.

They say everyone is there trouble shooting the problem. Could the techs start taking phone calls and get those of us left sorted out before they go home? I know its not their typical job, but some of us dont want to lose our own clients that use the software we have on Gigenet cloud devices.
Posted by sfnz, 10-09-2013, 09:23 PM

Quote:

Originally Posted by SRobbins

1-800-561-2656 - No answer with a "Mailbox is full" message.
1-800-908-7115 - Goes to a message to "send a fax, or leave a message."
1-224-265-9165 - Busy signal.

They say everyone is there trouble shooting the problem. Could the techs start taking phone calls and get those of us left sorted out before they go home?

I'd keep trying the 1-800-561-2656 number if you must, it would likely have gone to voicemail as everyone was answering calls. You must appreciate that they'll be run off their feet at this point.

Similarly, I can guarantee they will have staff answering phones but it's also crucial to have staff on the floor getting your services back online. I'd doubt they'd leave the premises without servers being back online.
Posted by GigenetKristin, 10-09-2013, 09:34 PM
Hello,

Our phones were completely out due to our loss of power so we were forwarding them to one person.

He is answering as many calls as possible, and his mailbox filled up quickly.

As soon I can find someone to get into our pbx and flip the phone systems to their previous state I will get that done..... however at this time the higher priority is for everyone to keep working to get every customer back online ASAP.
Posted by RunRyder, 10-09-2013, 09:40 PM
runryder.com
66.252.2.7
Still down.
Posted by SRobbins, 10-09-2013, 09:44 PM

Quote:

Originally Posted by GigenetKristin

Hello,

Our phones were completely out due to our loss of power so we were forwarding them to one person.

He is answering as many calls as possible, and his mailbox filled up quickly.

As soon I can find someone to get into our pbx and flip the phone systems to their previous state I will get that done..... however at this time the higher priority is for everyone to keep working to get every customer back online ASAP.

I appreciate the situation you all are in Kristin. Thank you for remaining true to your customer service efforts. I dont mean to sound frustrated or anxious. We(our company) are getting pressure from our own clients to get out SaaS back up in the Chicago area. So I completely relate to your situation.

At the same time it's difficult to understand some of the oddities. For example, server showing "on" in the panel, but not connecting with RDP or web console. So we are just trying to be at the bleeding edge of when you all have us fully up.

Is there a site you are posting frequent updates to on Cloud status, phones, ext that we can follow?
Posted by GigenetKristin, 10-09-2013, 09:46 PM

Quote:

Originally Posted by SRobbins

I appreciate the situation you all are in Kristin. Thank you for remaining true to your customer service efforts. I dont mean to sound frustrated or anxious. We(our company) are getting pressure from our own clients to get out SaaS back up in the Chicago area. So I completely relate to your situation.

At the same time it's difficult to understand some of the oddities. For example, server showing "on" in the panel, but not connecting with RDP or web console. So we are just trying to be at the bleeding edge of when you all have us fully up.

Is there a site you are posting frequent updates to on Cloud status, phones, ext that we can follow?

I am posting here, on twitter, and on facebook.

Please give me an IP so I can pass it along to someone and try and escalate it for you.
Posted by SRobbins, 10-09-2013, 09:49 PM

Quote:

Originally Posted by GigenetKristin

I am posting here, on twitter, and on facebook.

Please give me an IP so I can pass it along to someone and try and escalate it for you.

<IP REMOVED>
TY

EDIT: Looks like something is happening. I removed my IP for security reasons before I lose Editing power on the post.
Posted by hostseeker, 10-09-2013, 09:50 PM
216.38.14.222 still down for me. Please check on this

Ticket system not working no one answering phone.
Posted by GigenetKristin, 10-09-2013, 10:01 PM
I will try and get you guys an update on GigeNET Cloud.

The guys here are working very hard to get you back online. I am juggling between wanting to get you updates as fast as possible, and not wanting to slow down their progress on fixing things for you.
Posted by SRobbins, 10-09-2013, 10:03 PM

Quote:

Originally Posted by GigenetKristin

I will try and get you guys an update on GigeNET Cloud.

The guys here are working very hard to get you back online. I am juggling between wanting to get you updates as fast as possible, and not wanting to slow down their progress on fixing things for you.

Thank you. Let me know if you need my IP again.

This forum locks you our of editing your post after 15 minutes or so and I didn't want it to remain up.
Posted by GigenetKristin, 10-09-2013, 10:03 PM
I hear phones ringing so they are back up, please be patient when calling in.

The staff is to and from their desks and into the datacenter checking on things as they come back online so we are limited on resources to pick up phones.
Posted by GigenetKristin, 10-09-2013, 10:05 PM

Quote:

Originally Posted by SRobbins

Thank you. Let me know if you need my IP again.

This forum locks you our of editing your post after 15 minutes or so and I didn't want it to remain up.

I did not know that.
I've saved both of your IP's to a notepad file.
You can remove them if you'd like.
Posted by RunRyder, 10-09-2013, 10:12 PM
My backdoor is now working.
runryder.com
The front door (66.252.2.7) proxy shield is still down.
Posted by GigenetKristin, 10-09-2013, 10:16 PM
Steven - I got your PM however I think I am under the minimum post count to PM people. Please gtalk me: kristin.andrus@gigenet.com
Posted by txf861, 10-09-2013, 11:31 PM
When will be cloud server back up?
Posted by SRobbins, 10-09-2013, 11:33 PM

Quote:

Originally Posted by txf861

When will be cloud server back up?

I just spoke with Chris(literally just hung up) and their expectation are all machines in 1 hour. However that could change if they encounter problems.

fingers crossed.
Posted by GigenetKristin, 10-09-2013, 11:42 PM
I am being told it varies depending on which SAN it is, but as little as 30 minutes. They are working hard on it!
Posted by Odd Fact, 10-10-2013, 12:32 AM
Can we get a current update? I am still offline.
Posted by SRobbins, 10-10-2013, 01:10 AM

Quote:

Originally Posted by Odd Fact

Can we get a current update? I am still offline.

I just got off the phone with "Rahjem"(forgive me if I spelt it wrong). He said cloud computers are taking slightly longer then expected. He explained they typically start only 30 at a time, but due to the power outage they are having to a larger number which means the read/write to disk is working over time. They are staggering them in groups so they dont all start in one big pig pile, but it still is taking longer. No ETA.

However good news for me... mine is starting, its just going to take over 20 minutes before I can log in.
Posted by RunRyder, 10-10-2013, 01:13 AM
RunRyder is back. Thanks
Posted by marcosv, 10-10-2013, 01:13 AM
Already complete more than five hours of downtime.
And none of the positions given for them, at least for my servers were not correct.
Posted by hostseeker, 10-10-2013, 01:18 AM
Over 8 hours of downtime for me and still down! I thought the Gcloud was supposed to be reliable and redundant and all that.
Posted by GhostMachine, 10-10-2013, 01:34 AM

Quote:

Originally Posted by hostseeker

I thought the Gcloud was supposed to be reliable and redundant and all that.

Ha! That's a good one. Cloud server is just another name for virtual machine, which means 100 instances running on a regular Dell server. The only thing "reliable" about the Gcloud is the profit it makes for Gigenet.
Posted by GhostMachine, 10-10-2013, 02:41 AM

Quote:

Originally Posted by hostseeker

I thought the Gcloud was supposed to be reliable and redundant and all that.

Our outage: 10 hours and counting! And remember, the power was only out for about 30 minutes. In what world is this acceptable?
Posted by chrisarmer, 10-10-2013, 03:25 AM
Attention GigeNET Customers,

I just want you all to know we are doing everything within our power to get all of the centralized storage devices back up and 100% operational.

Our top priority is data integrity and second to that is returning all Cloud Servers to their working state. We understand a few customers might still be experiencing downtime but please know that we will not stop until this is fixed. So far everything is coming back up as expected and we are going through each Cloud Server 1 by 1 to ensure the servers come back and do not require a filesystem check.

Please, if you have an issue currently with your cloud server, send us an email at support@gigenet.com so we focus on your server during this process.

We are extremely grateful to have such wonderful customers and we thank you for understanding the uncontrollable situation we had to deal with today. Please bare with us and the next message we post should be a positive one stating our progress we have made.
Posted by afallison, 10-10-2013, 03:40 AM
That's the same thing I've been hearing for 11 hours...

I think everyone else got fixed first, then they started working on the cloud...
Posted by hostseeker, 10-10-2013, 09:24 AM
I was down a total of 11 hours before coming back up. I also fail to see how a 30 minute power outage can bring a host down for 11 hours. Obviously Gigenet's disaster plan didn't work very well.

Since Gigenet has 2 data centers I would have expected that when the Chicago center went down that at least there would be a status page and a ticket system fall over to the other data center so we would know what is going on. Instead of just disappearing from the Internet.

I hope Gigenet learns from this, and I also hope they give all customers credit WITHOUT putting the burden on the customer to submit a ticket to get the credit, it would be the decent thing to do.
Posted by trolltuner, 10-10-2013, 11:22 AM
Still down

Think you forgot to hit the button on my dedicated...
Posted by bueno, 10-10-2013, 12:56 PM
Still down.

Ticket # 242464 is opened 5 hours ago and no response or resolution.
Posted by marcosv, 10-10-2013, 05:17 PM
Gigenet is getting to be worse than other companies "low cost".
I have a server in Chicago more than 15 hours offline.

I am currently migrating to Rackspace in Chicago.
Is a tip for those who need a server in the region.
Posted by cb1000rider, 10-10-2013, 06:04 PM
Kristen,
Can you provide more details on how you went about shutting down systems yesterday? You indicated:

Quote:

All systems should have been shutdown properly before the electricity was turned off.

It would also be helpful to know how you started systems back up. We've been fighting fires all day in terms of system recovery due to SANs not being available when hosts came up etc?

Apparently GigeNet has erased posts on clean shutdown on twitter. If you did not cleanly shut down, it alters our recovery procedures.
Posted by petrushka, 10-10-2013, 06:30 PM

Quote:

Originally Posted by cb1000rider

Kristen,
Can you provide more details on how you went about shutting down systems yesterday? You indicated:

It would also be helpful to know how you started systems back up. We've been fighting fires all day in terms of system recovery due to SANs not being available when hosts came up etc?

Apparently GigeNet has erased posts on clean shutdown on twitter. If you did not cleanly shut down, it alters our recovery procedures.

In my experience, our dedicated server MySQL database was cleanly shutdown, no issues coming back up.

Our Cloud VM database server was hard-shutdown, MySQL tables crashed and myisamchk repair was needed.
Posted by Steven, 10-10-2013, 08:35 PM
Anyone else seeing servers dropping? Just had a half dozen boxes go down.
Posted by JonL, 10-10-2013, 08:36 PM

Quote:

Originally Posted by Steven

Anyone else seeing servers dropping? Just had a half dozen boxes go down.

Yes. My cloud server went down a few minutes ago.
Posted by afallison, 10-10-2013, 08:37 PM
same here..
Posted by RavC, 10-10-2013, 08:38 PM
Yep at about 1930CST
Posted by Curious Too, 10-10-2013, 08:38 PM
Yes. For some reason the servers are rebooting.
Posted by gtv6, 10-10-2013, 08:39 PM
my dedi just went out again

yay for pingdom monitoring though
Posted by afallison, 10-10-2013, 08:40 PM
im on cloud. so dedi's and cloud is down. nice..
Posted by openhost, 10-10-2013, 08:41 PM
yep, down as well. VMs are showing as 'Frozen' same as the outage yesterday.
Posted by RavC, 10-10-2013, 08:41 PM
I see some boxes up.. rebooted
Posted by afallison, 10-10-2013, 08:58 PM
Now my panel says it's in Maintenance Mode with the following message:

The system is currently not available
Details: We're aware of the issue and we're working to resolve it.

The system will be back shortly. Please contact customer support with any questions or concerns.
Posted by hostseeker, 10-10-2013, 08:59 PM
Out here again too, this is disgusting.
Posted by petrushka, 10-10-2013, 09:31 PM
Our dedicated server has been unaffected.

All our cloud servers have no network connectivity.
Posted by keliix06, 10-10-2013, 10:35 PM
The silence is a bit deafening. In our case it appears to be a network issue. Traceroute failing at servernap.net.
Posted by Steven, 10-10-2013, 11:11 PM
Has anyone actually got anything out of support at all?
Posted by keliix06, 10-10-2013, 11:23 PM
Nope. Ticket open over 2 hours.
Posted by keliix06, 10-11-2013, 02:26 AM
Got a vague response to the ticket after it had been open 5 hours.
Posted by keliix06, 10-11-2013, 03:16 AM
Did anyone else come up just before 2:15am CST?
Posted by petrushka, 10-11-2013, 08:40 AM
Same here, that's around when we came back up on 2 of our cloud servers that were down.
Posted by gigenet_customer, 10-11-2013, 11:07 AM
Both of my dedicated servers were rebooted approximately 14.5hrs ago and appear to have been fine since then.

EDIT: They were rebooted by gigenet, not me.
Posted by petrushka, 10-11-2013, 11:11 AM
I will add our dedicated server was unaffected and was up.

We have a couple dozen cloud servers, all were down 8:31pm EST, 90% of them came back up 9:50pm EST though required Apache reboots. 2 of them were still down until 2:31am EST.
Posted by layer0, 10-11-2013, 11:33 AM
Prior to this, Gigenet has been very stable for a long time. It's surprising to see this issue crop up, but it does seem to happen to the best of providers sometimes. I do wish communication was better.

I am sure Chris/Ameen will have a more detailed RFO.
Posted by cb1000rider, 10-11-2013, 11:59 AM

Quote:

Originally Posted by petrushka

In my experience, our dedicated server MySQL database was cleanly shutdown, no issues coming back up.

Our Cloud VM database server was hard-shutdown, MySQL tables crashed and myisamchk repair was needed.

Gigenet indicated an orderly shutdown of systems.
Our windows systems indicate hard shutdown.
Our Linux systems indicate the same thing.

They're not responding to request for details about shutdown/startup procedures, after indicating "clean" shutdowns.

Order of operations matter for us. We've got SAN storage that is required to come up before other hosts otherwise we run into significant problems with VMs... Which is what we're seeing currently.
Posted by Amitz, 10-11-2013, 12:27 PM
Just received this eMail:

Quote:

Dear GigeNET Customers,

On October 10th at 7:33PM CST, while undergoing a refueling from Osco to our Diesel Generator, we experienced a brief power glitch resulting from the generator dropping revolutions per minute while the Battery Backups were in bypass. With the way the system is normally setup, refueling the Diesel Generator and momentarily lowering voltage would have not caused any problems due to the fact it would have been feeding the Battery Backup System instead of the Power Distribution Units that supply power to all of our servers and equipment. We are using the Generator to power the datacenter only temporarily while waiting for replacement parts for the Battery Backup System. The Generator is a much more stable, cleaner source of power than the power feed coming in to the building.

Thank You,

Chris Armer
Senior VP Of Operations

I don't know... Something does not feel right.
Posted by freddyh, 10-11-2013, 12:31 PM

Quote:

Originally Posted by Amitz

Just received this eMail:
I don't know... Something does not feel right.

I didn't notice any outage yesterday.
Posted by petrushka, 10-11-2013, 12:41 PM
It is strange to me they don't mention anything at all about misconfiguration of their Cloud servers.

When you compare the uptime the last few days between their dedicated servers and Cloud Servers, it is clear to me > 75% of the downtime is related to either Cloud misconfiguration, or perhaps their Cloud Servers hypervisors etc simply taking a long time to start-up (by long I mean hours, not minutes).
Posted by layer0, 10-11-2013, 01:37 PM

Quote:

Originally Posted by petrushka

It is strange to me they don't mention anything at all about misconfiguration of their Cloud servers.

When you compare the uptime the last few days between their dedicated servers and Cloud Servers, it is clear to me > 75% of the downtime is related to either Cloud misconfiguration, or perhaps their Cloud Servers hypervisors etc simply taking a long time to start-up (by long I mean hours, not minutes).

I guess this just comes down to the more complex nature of a cloud environment, especially when you have SANs involved. If some of the cloud infrastructure in fact encountered unclean shut downs, it is not a big surprise that this would take some work to fix properly, vs. dedicated servers which do not usually rely heavily on network based storage.

We only utilize their dedicated servers at this time, so I can't comment on cloud related issues, but they did have our machines up pretty quickly once power was restored.
Posted by petrushka, 10-11-2013, 01:43 PM

Quote:

Originally Posted by layer0

I guess this just comes down to the more complex nature of a cloud environment, especially when you have SANs involved. If some of the cloud infrastructure in fact encountered unclean shut downs, it is not a big surprise that this would take some work to fix properly, vs. dedicated servers which do not usually rely heavily on network based storage.

We only utilize their dedicated servers at this time, so I can't comment on cloud related issues, but they did have our machines up pretty quickly once power was restored.

That all makes sense, and I agree it is better to prioritize data consistency over restart time.

Also on our dedicated server side, it came back very quickly and cleanly (didn't even trigger our monitoring alert), it was only the Cloud Servers that seemed to have extended downtime.

Still, if there was a lengthy SAN / Cloud restart period involved, particularly if that period of time represented a majority of the downtime for Cloud customers, I feel it was an omission in their message to not speak to this.

I don't know their split, if say 95% of their customers are dedicated servers and their cloud has very few customers, that is one thing, but if they have a decent amount of cloud customers, any downtime related to that aspect should of been addressed in their communication too.
Posted by cb1000rider, 10-11-2013, 02:21 PM

Quote:

Originally Posted by Amitz

Just received this eMail:

This is the only communication I've received from Gigenet and it doesn't address the original outage.

We can't reach anyone on the phone - lines are now busy.

I'm aware that technical staff is busy and understand that they are having an emergency, but other staff should be sending status communications to customer accounts, even if generic.

Misinformation (clean shutdown) only exacerbates the problem for customers with complex environments that require specific restart procedures.

If you have a direct line of escalation to the executive level (even an email address) - please PM me.
Posted by layer0, 10-11-2013, 02:59 PM
cb1000rider

Are you having issues with their cloud platform, or dedicated servers?
Posted by cb1000rider, 10-11-2013, 03:58 PM
We don't use their cloud. We're co-lo.
Posted by GhostMachine, 10-11-2013, 06:09 PM

Quote:

Originally Posted by layer0

Prior to this, Gigenet has been very stable for a long time.

Not for us. FWIW, We've had half a dozen outages in the last year, and this fiasco is the last straw. Each and every time there's an outage, nobody answers the phone, nobody gives realistic recovery times, and nobody seems to be in any hurry to get our systems back online. We've hosted with many different companies over the years, and this is by far the worst experience we've had. If it wasn't so painful to move our systems, we would have done it long ago, but at this point enough is enough.
Posted by layer0, 10-11-2013, 06:10 PM

Quote:

Originally Posted by GhostMachine

Not for us. FWIW, We've had half a dozen outages in the last year, and this fiasco is the last straw. Each and every time there's an outage, nobody answers the phone, nobody gives realistic recovery times, and nobody seems to be in any hurry to get our systems back online. We've hosted with many different companies over the years, and this is by far the worst experience we've had. If it wasn't so painful to move our systems, we would have done it long ago, but at this point enough is enough.

Sorry to hear about your issues, just to clarify what services do you have with them?
Posted by GhostMachine, 10-11-2013, 06:12 PM
We're on cloud by the way. I think that the complexity of the cloud system is beyond the capabilities of Gigenet staff at this point, and explains why the stability isn't there and the downtime drags on. People with better experience are probably doing co-lo or dedicated hosting which relies less on Gigenet knowing what to do.
Posted by cb1000rider, 10-11-2013, 06:55 PM

Quote:

Originally Posted by GhostMachine

Each and every time there's an outage, nobody answers the phone, nobody gives realistic recovery times, and nobody seems to be in any hurry to get our systems back online. We've hosted with many different companies over the years, and this is by far the worst experience we've had.

This is the first major outage that I'm aware of. I can understand confusion if the fire department orders the power off.
However, after 12 hours, I'd expect some corporate level notifications. Limited notification (one email) and the phone are off the hook 24 hours later?

Guys, you're doing it wrong.
Posted by GhostMachine, 10-11-2013, 07:04 PM

Quote:

Originally Posted by cb1000rider

This is the first major outage that I'm aware of.

Then consider yourself lucky. I just went back and checked our logs, and we have experienced 10 outages in the last 12 months serious enough to open tickets for. Gigenet has been very good about giving us SLA credits every time we've asked for them, but credits do nothing to repair our reputation with our customers.
Posted by freddyh, 10-11-2013, 07:28 PM

Quote:

Originally Posted by GhostMachine

Then consider yourself lucky. I just went back and checked our logs, and we have experienced 10 outages in the last 12 months serious enough to open tickets for. Gigenet has been very good about giving us SLA credits every time we've asked for them, but credits do nothing to repair our reputation with our customers.

Are you one of those guys that write in the second someothing goes down?
Posted by GhostMachine, 10-11-2013, 07:37 PM

Quote:

Originally Posted by freddyh

Are you one of those guys that write in the second someothing goes down?

So you think it's okay for critical systems to crash as long as they're not down long? I expect more from Gigenet. We're not hosting some blog or comic, our customers lose money every minute these sites are down. We take it seriously.

Many of our outages were hard crashes, which is a terrible thing to do to a business-critical SQL server. Also, Gigenet's SLA policy does not issue credits until the outage lasts 15 minutes or more. Our outages range from about 20 minutes to over 17 hours.
Posted by cb1000rider, 10-11-2013, 07:58 PM
Read the SLA - it excludes quite a bit in terms of root cause.
Posted by cb1000rider, 10-11-2013, 08:02 PM

Quote:

Originally Posted by GhostMachine

Gigenet has been very good about giving us SLA credits every time we've asked for them, but credits do nothing to repair our reputation with our customers.

You shouldn't have to ask. They know when they are offline... Or at least they should.

Quote:

Originally Posted by freddyh

Are you one of those guys that write in the second someothing goes down?

We're talking about a facility with a 100% SLA. Redundant power. Backup generators. I understand if they run mom-and-pop websites, but that's not all they run.
I don't know who their biggest customers are, but expect $250M+/businesses that lose money if they are offline.
I understand if it's ok for some customers to be offline, but for others, it can quickly sink the business.
Customers shouldn't have to write in when something goes offline. They should be on the phone to the major customers and send email notification to everyone.