Snelserver down

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Snelserver down

Posted by shubhank, 03-19-2012, 11:00 AM
Well even our server was working all good and top class but suddenly our server, KVM, SNELpanel and everything went down ?

Is there any powercut or some problem ?
Posted by Vitaly, 03-19-2012, 11:07 AM
Snel IS, you are viewing this forum now, when the problem will be solved?
Posted by Snelinternet, 03-19-2012, 11:09 AM
We are working on it, as soon as we know the cause or have any update we'll let you know.
Posted by TBPrince, 03-19-2012, 11:10 AM
Confirmed that they are experiencing problems. Don't know which nature nor we have an estimate from them for service restoration.
Posted by OnTrackIT, 03-19-2012, 11:13 AM
Has been down now for 40minutes. Please can you elaborate on what the issue is?
Posted by chadm, 03-19-2012, 11:14 AM
Down for me as well, looks like their entire network as I cannot get to their site, control panel or my server.
Posted by shubhank, 03-19-2012, 11:14 AM
ah so till now even cause is not yet known ?

Also, looks like your whole DC or all racks are down as KVM and your own site and panels are down ?

Anyways, I hope things get back within an hour or such, servers were running so smooth till now
Posted by backtogeek, 03-19-2012, 11:16 AM
Just got off the phone with their support staff (Very polite and helpful) they have a network issue and don't believe it is a DDOS, I got off the phone quick so they can concentrate on fixing it rather than getting interrupted every 5 minutes.

Hopefully it wont be to long.
Posted by Snelinternet, 03-19-2012, 11:16 AM
The only thing we've been able to rule out is an issue on our equipment, the switches and core switches are up.
Our network provider is now looking into their routers and peering and have not yet given us an update.
Posted by Goldwing, 03-19-2012, 11:16 AM
Same here - did find it a bit strange that they have their own site on the same network as clients meaning there is no way to communicate or get information
Posted by OnTrackIT, 03-19-2012, 11:20 AM
Facebook https://www.facebook.com/#!/pages/Sn...74250975948545
Posted by backtogeek, 03-19-2012, 11:21 AM
I guess the reason for that is for the security when you consider how much automation the snelpanel provides, personally I don't like the idea of them having that hosted anywhere else.

Perhaps an offsite announcements page would be good though
Posted by rds100, 03-19-2012, 11:22 AM
"offsite announcements page" - sounds just like "twitter".
Posted by Snelinternet, 03-19-2012, 11:24 AM
Our provider has noticed a memory error on the line-card of the router but has not yet found the reason why it did not fail over or why it's not coming back online after resolving this.
As soon as we have more updates we'll let you know.
Posted by akisa, 03-19-2012, 11:25 AM
down for our website also, I'm a new customer with snel and joined them 3 months ago but this is the first time We faced unexpected downtime like this, down since 45 minutes
Posted by fevangelou, 03-19-2012, 11:29 AM
Listen to what your customers are saying and create an offsite announcements page...

This can happen again. Do we need to track WHT forums for responses or updates from SnelServer? Come on guys...
Posted by netofy, 03-19-2012, 11:30 AM

Work your magic, Wouter!
Posted by backtogeek, 03-19-2012, 11:33 AM

Quote:

Originally Posted by fevangelou

Listen to what your customers are saying and create an offsite announcements page...

This can happen again. Do we need to track WHT forums for responses or updates from SnelServer? Come on guys...

To be fair they always listen, by the sounds of it this is out of their direct control, I am sure as a result of this they will create an offsite announcements page.

Wouter will be screaming profanities in 8 different languages by now I bet!
Posted by netofy, 03-19-2012, 11:34 AM

Quote:

Originally Posted by fevangelou

Listen to what your customers are saying and create an offsite announcements page...

This can happen again. Do we need to track WHT forums for responses or updates from SnelServer? Come on guys...

File in the 'Lessons Learned' drawer to revist AFTER we get the sites back up.
Posted by StopForumSpam, 03-19-2012, 11:39 AM
I have to step in and defend SnelIS on this one. OK, WHT might not be the best platform for annoucements but when you have an emergency like this, you take what is at hand and inform the best you can while you are running around trying to fix things.

Ive found their service and support beyond anything that Ive ever experienced before and there is no way that I would like one hiccup cause any problem for me.

Their upstream provider will fix their issues and we will all be back online in no time but I will have 500 emails from people screaming that their boards are now filling up with spam... I guess that they should donate more so that I can afford a second backup server
Posted by backtogeek, 03-19-2012, 11:42 AM

Quote:

Originally Posted by StopForumSpam

I have to step in and defend SnelIS on this one. OK, WHT might not be the best platform for annoucements but when you have an emergency like this, you take what is at hand and inform the best you can while you are running around trying to fix things.

Ive found their service and support beyond anything that Ive ever experienced before and there is no way that I would like one hiccup cause any problem for me.

Their upstream provider will fix their issues and we will all be back online in no time but I will have 500 emails from people screaming that their boards are now filling up with spam... I guess that they should donate more so that I can afford a second backup server

Well said and agreed.
Posted by OnTrackIT, 03-19-2012, 11:50 AM
I feel Snel provide enough information, and the fact they respond on Social media very quickly is good enough for us. As already said by others its out of there control, but maybe having dual hardware rather than dual supervisor cards might be an option to prevent this again.
Posted by akisa, 03-19-2012, 12:04 PM
any update? we're in the 2nd hour with this downtime
Posted by lns2000, 03-19-2012, 12:12 PM
Too low response for the downtime. Do snelserver give special offer their customers because of the downtime. How much snelserver guarantee about the uptime?
Posted by Snelinternet, 03-19-2012, 12:19 PM
Our provider is going to replace several blades of the routers and hopes this will resolve the issue.
As soon as we know more, we'll let you know.
Posted by Goldwing, 03-19-2012, 12:31 PM

Quote:

Originally Posted by backtogeek

Well said and agreed.

Well actually Snel didnt come to the forum they responded to a message nearly an hour after the servers went down, up to that point no one knew what was going on ( I certainly didnt)

Personally I dont have the time to go chasing around social media sites and if a provider whether snel or anyone else has a problem my first place of call is their own site. I was constantly on the phone dealing with clients and could not find an update to let them know what was happening.

Sh** happens - unusual downtime is quite easy to explain to clients but they want to know what and when. I have been very happy with my server and the company
Right now I am a bit concerned about single points of failure since this happened
Posted by akisa, 03-19-2012, 12:32 PM
Snel IS: thanks for the update.. I think 2nd provider is a mandatory after this problem.
Posted by Snelinternet, 03-19-2012, 12:37 PM
I have no update on the issue at hand, but just to comment on the 2 things that have been mentioned:
1) An off-site NOC page is definitely a great idea and will be implemented in the near future.

2) While we have several peering and carriers, we are indeed relying on 1 network provider at this time.
Coincidence or not, we have already been in negotiations with other parties to introduce a second datacenter and network provider.
Issues like this confirm this is absolutely necessary and we will increase pressure and urgency to get this implemented much sooner than planned.

As soon as we have updates about the issue we'll let you know.
Posted by shubhank, 03-19-2012, 12:49 PM
Well thanks for letting us know about it, I can update the guys at top who owned the server(s) and post on their FB page so members donot whine about it

Also, it will be a good idea to have a twitter or FB page and update such issues separately, I also saw your FB update only in a comment and not as a status update so such helps in case your main site is down.

@People who asked about uptime or downtime policy or if they will get anything for it

Maybe now when site gets up you can read the TOS again and for real rather than asking such questions or whining about it
Posted by richooz, 03-19-2012, 12:59 PM
Guys , when your service will be back online , we are running voip over these servers that we rent from you, and now, every minute pass we loose a lot of money!!!!!!!!!!!!
Posted by Robert vd Boorn, 03-19-2012, 01:02 PM
It's fair to assume that their team is doing everything they can to keep the damage to a minimum, having the forum representative's subscription to this thread poke him all the time, distracting him from the recovery work, will certainly not speed up the process. And I'm also sure you'd get to know it as soon as there is something worth sharing...
Posted by backtogeek, 03-19-2012, 01:13 PM

Quote:

Originally Posted by richooz

Guys , when your service will be back online , we are running voip over these servers that we rent from you, and now, every minute pass we loose a lot of money!!!!!!!!!!!!

If you loose money by the minute perhaps you should have your own redundancy set up?

This issue was not caused by Snel directly.
Posted by 24x7group, 03-19-2012, 01:15 PM
I'd like to ask every customer to have patience in this matter. SnelIS is doing all they can to get their stuff online again.
Shouting does not resolve the issue any faster, so please bear with them
Posted by StopForumSpam, 03-19-2012, 01:22 PM
I have 25,000 users that cant use my server and its not worrying me. I have a backup up and running but no one is using it... maybe i should think harder about how they access my service in the future
Posted by mihaialdea, 03-19-2012, 01:25 PM
Don't want to be a bugger but is there an ETR on this?
Posted by kelso, 03-19-2012, 01:27 PM
Now all the servers down, and even their site is down so just have to wait for something to go up to at least send a ticket
Posted by Snelinternet, 03-19-2012, 01:35 PM
The blades have been replaced, the configuration is now being reloaded and you should see things coming up shortly.
Posted by netofy, 03-19-2012, 01:44 PM
And we're back! ty all!
Posted by Robert vd Boorn, 03-19-2012, 01:53 PM
, well done.
Posted by akisa, 03-19-2012, 01:54 PM
We're up again thanks.. The people want a backup provider>> The people want a backup provider>> The people want a backup provider>> The people want a backup provider
Posted by Unixboy, 03-19-2012, 02:00 PM
I think Snel already got this message, otherwise they didn't take any actions.

For the Snel guys: Consider building a RFO; with the information coming from your upstream provider, being open in communication helps!

Enjoy your uptime!
Posted by Snelinternet, 03-19-2012, 02:21 PM
Everything should be back online. If it's not, please open a ticket, call us or send us a message through WHT.
We've asked our provider for a full explanation of the issue/timeline and hope to be able to send this out tomorrow morning.
Posted by shubhank, 03-20-2012, 08:48 AM
Everything was up within 2 hours of downtime and back to super normal
Posted by Snelinternet, 03-20-2012, 10:39 AM
We have had a network outage on the 19th of March from 15:32 till 18:42 Central European Time.
This e-mail outlines the events that happened, future improvements and compensation.

- Events
Around 15:33 we were notified by both our on-site an off-site monitoring systems that the network was offline
After quickly verifying there was no power issue or issue on our core switches we contacted our network provider.
Our network provider quickly ruled out a fibre break and diagnosed a memory error in one of the blades.

This blade was restored and appeared normal, but the network did not come back online.
After some time of troubleshooting our provider further ruled out an issue with the optics or chassis.
The decision was made to replace a single entire blade and our provider proceeded to replace it.

Once they confirmed this did successfully resolve the issue, they replaced the other blades.
After configuring the VLANs and other settings the traffic restored to the usual levels.

- Future improvements
While we provided regular updates on Facebook and Webhostingtalk, this was not known to all clients.
To improve this we've implemented and launched an off-site NOC page located at http://www.snelnoc.com/
This will be used to announce all maintenance and outages with frequent updates on the current situation.

The other improvement we will be making is to introduce a second network provider with direct uplinks to them.
While we already have multiple carriers and peering, we're still reliable on a single network provider for our uplinks.
This outage has proven it is absolutely essential to have a secondary network provider with their own routers and transit.
Coincidentally we were already in negotiation with another network provider but this will be expedited and given high priority.

- Compensation
Because of this outage and our 99,9% uptime guarantee, we've credited all servers with a 1 week renewal.