Rapidswitch / Iomart Maidenhead Issues

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Rapidswitch / Iomart Maidenhead Issues

Posted by PCS-Chris, 07-19-2012, 03:25 PM
We are seeing high latency to all of our kit (racks/colo) and their support desk at the moment. Looks like a local router issue from a traceroute.

Appologies to all of our clients, this couldn't have come at a worse time due to our own switching issues yesterday. Updates will be given on our blog as usual.

Chris
Posted by Adam H, 07-19-2012, 03:28 PM
I'm seeing around 30% packet loss to our rack in their Leicester datacentre also.
Posted by BELLonline, 07-19-2012, 03:31 PM
40-50% packet loss here hopefully it will get sorted quickly. Problems are pretty rare with RS though to be fair
Posted by JosephRedfern, 07-19-2012, 03:37 PM
Not seeing any packet loss here, just a very, very slow connection. Hope this gets resolved soon!
Posted by xftp, 07-19-2012, 03:41 PM
Currently seeing a 50-60% packet loss here
Posted by Kakashi, 07-19-2012, 03:44 PM
Yep same here.
Posted by BELLonline, 07-19-2012, 03:50 PM
Just phoned up, they don't know what the problem is yet but will be sending an email out to customers soon with details.
Posted by PCS-Chris, 07-19-2012, 03:52 PM
They really need a simple offsite status page for things like this.

From what I can see it's due to an issue with iomart's router at LINX (London Internet Exchange), as LINX's own public graphs all look normal.
Posted by xftp, 07-19-2012, 04:10 PM
Fingers crossed it will be resolved quickly as this on top of the near 9 hours downtime yesterday i'm going to have some unhappy customers.
Posted by Dan-CKS, 07-19-2012, 04:15 PM
They down again? What happened to 100% Uptime? Or is that just Sales BS
Posted by StreamGBcom, 07-19-2012, 04:20 PM
Their not down. Looks like a linx problem but linx graphs seem to look ok. Some people can access and some can't so must be some sort of external problem.
Posted by StreamGBcom, 07-19-2012, 04:22 PM
Also they are aware of the problem those who haven't had emails....

Some of your services with us are currently experiencing an interruption to connectivity. We are investigating and will provide further updates as soon as possible.

Date: 19/07/2012
Time: 20:00
Duration so far: < 60 minutes
Affected Service(s):
Posted by Dan-CKS, 07-19-2012, 04:22 PM
Then in that case they need to update there crappy network. Is it still only 10Gbps?

EDIT:

There own site wont load for me. Coming From TMOUK
Posted by BBGN Brian, 07-19-2012, 04:24 PM
http://www.internethealthreport.com/ Looks like things are getting back to normal.

http://internettrafficreport.com/europe.htm
Posted by wakka, 07-19-2012, 04:35 PM
Still have problem and called RS.
They are aware of the problem and need to get finger cross!
Posted by Adam H, 07-19-2012, 04:35 PM
Latency appears to be back to normal. But still high packet loss.
Posted by ThanosD, 07-19-2012, 04:38 PM

Quote:

Originally Posted by BBH Brian

Looks like things are getting back to normal.

I still have major dropouts coming from Greece and USA. Sites mostly inaccessible and SSH reminds me of a 300baud modem...
Posted by BELLonline, 07-19-2012, 04:56 PM
Looks like it's reasonably back to normal now.
Posted by wakka, 07-19-2012, 04:56 PM
seems fine now
Posted by StreamGBcom, 07-19-2012, 05:57 PM
We apologise for the connectivity interruption to your service.

Our engineers have identified the cause as a massive DDOS attack against a customer on our network.

We have re-routed traffic and services should be resuming back to normal now.
Posted by Adam H, 07-19-2012, 06:00 PM
Email from rapidswitch:

Quote:

Earlier this evening we started to experience intermittent connectivity issues with segments of our network. Our core networking team identified the issue as a very significant, distributed denial of service attack that was saturating some of our upstream providers, with some customers experiencing drops in connectivity .

The traffic has been identified and re-routed, with normal service being restored as quickly as possible.

The traffic will be intensely monitored throughout the night to ensure normal service continues and we will provide further updates as necessary. We do apologise for any inconvenience. Please be assured we are committed to preventing this attack from causing any further problems with your service.

Regards,

The RapidSwitch Team

Posted by Dan-CKS, 07-19-2012, 06:08 PM
They still have a crap network i see. a DDOS attack can cause that much problem. time to update your network.
Posted by PCS-Chris, 07-19-2012, 06:24 PM

Quote:

Originally Posted by Dan-CKS

They still have a crap network i see. a DDOS attack can cause that much problem. time to update your network.

Troll much?

They are not perfect (Trust me, we've been through numerous router issues and were one of the first into their new DC), but until today we've had 100% network uptime for around a year and a half.
Posted by Dan-CKS, 07-19-2012, 06:25 PM

Quote:

Originally Posted by PCS-Chris

Troll much?

They are not perfect (Trust me, we've been through numerous router issues and were one of the first into their new DC), but until today we've had 100% network uptime for around a year and a half.

Am not a fan of rapidswitch. The way they treat certain customers is disgusting. Granted the hardware is usually quite good (But a ripoff non the less)
Posted by StreamGBcom, 07-19-2012, 06:37 PM
I have to admit since their last major problem they have done well. Have 5 boxes with them and connectivity has been top notch. There sales guys done me a good deal when I bought more servers and customer support has been good.

I think they have dealt with things quite quickly today. Things do happen.

The key is definitely don't put all your eggs in one basket.
Posted by Sguk, 08-09-2012, 03:26 PM

Quote:

Originally Posted by xftp

Fingers crossed it will be resolved quickly as this on top of the near 9 hours downtime yesterday i'm going to have some unhappy customers.

Hi there, did the 9 hours downtime occur with Rapidswitch?
Just wanted to check as we are joining them soon.
Posted by BELLonline, 08-09-2012, 03:54 PM
I don't recall there being downtime the day before this happened and certainly not 9 hours. Here's an uptime report for our server with RapidSwitch

http://www.hyperspin.com/publicrepor.../16394/2/53416

They're generally pretty stable, this was the first outage since Jan 2011.
Posted by Sguk, 08-09-2012, 04:50 PM
thanks BELLonline

Great uptime, and I'm guessing from this thread that the 2hrs downtime for July 2012 wasn't truly downtime, just that hyperspin.com monitoring couldn't connect because of latency.

Good to see they dealt with issues quickly and kept customers informed. There's always some troll saying "crappy network" "they should do this and do that" - infact from what I've been told though they are spending millions on big upgrades IPV6, dual supplies.