Wiredtree down once again!

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Wiredtree down once again!

Posted by hosting_talk, 04-04-2012, 12:01 PM
Hi,

It looks like this month we have another downtime from wiredtree.
I can't access grove https://grove.wiredtree.com/ and when I'm pinging my website I have 25% pocket loss
Posted by khanbaba, 04-04-2012, 12:02 PM
Yeah i am noticing it too, its sucking big time.
Posted by huckyboy, 04-04-2012, 12:04 PM
same here for me.

time to move out i guess
Posted by khanbaba, 04-04-2012, 12:05 PM

Quote:

Originally Posted by huckyboy

same here for me.

time to move out i guess

Yep for sure this time, its quite routine with them now.
Posted by Flapadar, 04-04-2012, 12:08 PM
Seems up on my end?
Posted by WHU-Mike, 04-04-2012, 12:09 PM
My VPS is down for me too.
Posted by bpcbpc, 04-04-2012, 12:09 PM
I guess they are planning to go down for half-day at least once a month :/

Any good alternatives? What would you suggest?
Posted by hosting_talk, 04-04-2012, 12:14 PM
It looks like this time the problem is more serious then the previous one. Their main website is not loading for me...
Posted by davidpaj, 04-04-2012, 12:18 PM
Down for me too, my vps, and the main site. Can anyone get through to phone support, that's busy as well?
Posted by Yogesh Sarkar, 04-04-2012, 12:26 PM
Same here for my dedicated server, got a alert from pingdom and seeing packet loss.
Posted by stormy, 04-04-2012, 12:26 PM
I can't believe this is happening again. I did get Grove to load though, and Pingdom doesn't report downtime yet. It seems an intermittent issue. Again.
Posted by huckyboy, 04-04-2012, 12:30 PM
and the most frustrating part of that is, we still get no word from them on what's happening. their twitter page is now peppered with wiredtree down messages and yet no update from them.
Posted by Yogesh Sarkar, 04-04-2012, 12:33 PM
I have never claimed compensation till now, even though I could have. But these issues are embarrassing me again and again in front of my community members and I just cannot tolerate these any longer!

These guys better change the network equipment ASAP and bringing in redundancies, else they aren’t going to get my dollars anymore!
Posted by WiredTree James, 04-04-2012, 12:34 PM
We are aware of the issue affecting a few racks in our datacenter and our Network admins are stabilizing the issue as I write this. This would explain why some of you reporting no problems while a few others are seeing intermittent issues.

We appreciate your patience while the problem is mitigated. If you are continuing to have problems, please open a Support ticket or PM me and I will have a ticket opened for you immediately. All open Support tickets will be updated the minute we have additional information.
Posted by huckyboy, 04-04-2012, 12:43 PM
hi James! nice to hear an update. anyways my vps is now ok and i hope for most of us here too.

just wondering what is it this time? the symptoms seem to be the same from the last outage. been a happy customer for a few years now but lately things are not doing well.
Posted by bpcbpc, 04-04-2012, 12:47 PM
Alright, my server is back (tested it at host-tracker.com too) Hopefully this downtime is over :/
Posted by WiredTree James, 04-04-2012, 12:48 PM

Quote:

Originally Posted by huckyboy

hi James! nice to hear an update. anyways my vps is now ok and i hope for most of us here too.

just wondering what is it this time? the symptoms seem to be the same from the last outage. been a happy customer for a few years now but lately things are not doing well.

You are welcome Huckyboy. We should know more shortly and when we do, I will share this information with everyone. I apologize for the inconvenience and assure you our Network Admins acted very quickly.
Posted by WiredTree James, 04-04-2012, 02:15 PM
Thank you guys for your patience. I have posted the full RFO below for you. This is also available within the Grove for all of our clients to see. Feel free to PM me or open a Support ticket if you have any further questions and we will make sure you are taken care of.

During mid-morning on 4/4, we began experiencing noticing packet loss and intermittent network connectivity issues for a subset of our clients. Not all clients were affected as the problem was on a per connection basis which randomly impacted connectivity into the main WiredTree network. Our internal monitoring systems notified our Network Engineers as soon as the problem was detected and they went to work isolating the issue within Core Router 2 (CR2). During this time we quickly re-routed traffic from CR2 to Core Router 1 (CR1) to even out the connection issues and service returned to normal. The problem that impacted CR2 was identified as a software glitch that affects the Brocade IronWare routing OS.

There was an identical incident on 3/6 that hit Core Router 1 (CR1), the other Core Router in our core network. After that outage we have been working with Brocade to gather more information on the issue and they had us turn on verbose logging and monitoring within our Core Router pair to help track down the root cause of these issues. They explained that they have never seen this issue before and they have been actively working with our network team to track down the root cause. Fortunately with verbose logging and immediate recognition of the issue, we were able to mitigate the issue with CR2 in a much shorter time span. We have Brocade logged into CR2 right now and they are gathering information to help us fix this software glitch for good.

All of us at WiredTree understand that uptime is paramount in any hosting services that you buy. We hate downtime just as much as you and your frustrations with these network problems is not something we take lightly. We are doing everything in our power to ensure this never happens again. We appreciate your patience and while any downtime is unwelcome, we are confident that Brocade will have the information required fix and prevent future issues caused by this software glitch.
Posted by eva2000, 04-04-2012, 02:34 PM
never mind didn't see the new replies to this post