UK2 and VPS.NET down again?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > UK2 and VPS.NET down again?

Posted by notjustpixels, 01-12-2012, 09:17 PM
Anyone else lost connectivity to VPS.NET or UK2?
Posted by markdrandall, 01-12-2012, 09:18 PM
UK2.net is 100% down. I just got through to midphase as we have a server with them and was advised this is a network issue.
Posted by IGobyTerry, 01-12-2012, 09:19 PM
There appears to be a network issue in our London datacenter. The VPS.NET website is hosted off site at SoftLayer however the nameservers for it are hosted at our London datacenter, which took the site offline. I will attempt to update this thread as we bring everything back online. Last edited by IGobyTerry; 01-12-2012 at 09:30 PM.
Posted by ramnet, 01-12-2012, 09:20 PM
Indeed, UK2 / 100TB is down in London for me as well. Going on 25 minutes now. Issue appears to be BGP related as all their IP's have dropped from the global BGP routing tables.
Posted by BuyYourHost, 01-12-2012, 09:20 PM
VPS.net is 100% down here. Hope they get things resolved quicky
Posted by markdrandall, 01-12-2012, 09:22 PM
UK2 doesn't even seem to announce a route to me past BT (I am located in the UK and am using BT right now) 1 2 ms 1 ms <1 ms 192.168.1.254 2 2 ms 39 ms 35 ms 213.123.109.58 3 39 ms 33 ms 173 ms 213.123.109.161 4 41 ms 39 ms 41 ms 213.1.69.122 5 42 ms 40 ms 39 ms 213.120.179.54 6 42 ms 41 ms 39 ms 213.120.179.182 7 * * * Request timed out. 8 * * * Request timed out.
Posted by IGobyTerry, 01-12-2012, 09:22 PM
Just to clarify, every location other than London is online now, which is experiencing a network issue.
Posted by iexo, 01-12-2012, 09:22 PM
Both also down for me. Aren't VPS.net on a cloud?
Posted by IGobyTerry, 01-12-2012, 09:25 PM
Nameservers are hosted at our London datacenter, which is why the site went down. We'll be changing that...
Posted by suhailc, 01-12-2012, 09:31 PM
Hi, What is the problem exactly? Regards, Suhail.
Posted by IGobyTerry, 01-12-2012, 09:32 PM
Just a network issue. We have our lead network engineers working on it now. No matter the issue we have software and hardware backups ready to be deployed if necessary, so I expect we'll have everyone back online very soon.
Posted by Snat, 01-12-2012, 09:33 PM
Thanks for the update, any idea how long roughly other then shortly?
Posted by Psycnosis, 01-12-2012, 09:34 PM
We also have a dedicated server with UK2. Is there any timeline on when it might be done? Like are we talking minutes or wont be back on till tomorrow? thanks for all info you may be able to give us.
Posted by markdrandall, 01-12-2012, 09:35 PM
Midphase say they have no idea how long it will last for and just to be patient and that they are working as fast as they can....
Posted by MyITGuy, 01-12-2012, 09:37 PM
Interesting...I thought I read somewhere that the vps.net nameservers were located in 9 locations and are anycasted?
Posted by Frands, 01-12-2012, 09:38 PM
I'll tell my users that. "Oh, nothing big you know. The entire datacenter went offline, nothing big."
Posted by IGobyTerry, 01-12-2012, 09:39 PM
The nameservers for VPS.NET are hosted in London. We allow our clients to use our anycast setup. Case of giving our client's better than we give ourselves, I guess. But it will be fixed so this doesn't happen in the future.
Posted by markdrandall, 01-12-2012, 09:41 PM
This will make you laugh, call 0800 6122142 option 2 is for the latest system status which says "There are currently no issues to report"
Posted by markdrandall, 01-12-2012, 09:42 PM
Hello, I am sorry for the trouble that this issue has been causing you. It seems that we are currently having problems in our UK datacenter. We do not have a lot of information and are still looking into the issue. We should have the problem sorted shortly. -- Regards, Tanner S. Dedicated Server Engineer
Posted by Thomas, 01-12-2012, 09:45 PM
Hi guys, Sorry for jumping in this a bit late but we've all been working since around ~45 mins ago when we received alerts about the network problems, the network team are working on it right now. Sorry again and further updates to come.
Posted by markdrandall, 01-12-2012, 09:52 PM
Did you edit that post as it said 30 minutes?? Seems to be back now anyway, but it is a serious downtime for a large datacenter, would be nice to know why!
Posted by BELLonline, 01-12-2012, 09:55 PM
Just came back Thanks to the uk2 engineers we can all sleep now haha
Posted by Thomas, 01-12-2012, 09:56 PM
Yes you lose site of time when things go down and you panic a bit (had to double check the clock).... everything should be back now. If not send in a ticket. I am getting a clearer picture now into the problems but it looked to be a routing protocol problem. Let us know if you have any problems still in a ticket! Bit of a further update: OSPF protocols on our core routers failed without warning or reason, we have rectified the immediate issues, and are investigating the root cause to prevent any further reoccurrences Last edited by Thomas; 01-12-2012 at 10:01 PM.
Posted by Psycnosis, 01-12-2012, 09:57 PM
good job uk2 staff
Posted by cutabovehost, 01-12-2012, 10:04 PM
I know that the site http://shamelesshosts.com/ is hosted on one of their servers so I checked it and it was up. I figured it would have to be something localized.
Posted by OpticalGaming, 01-12-2012, 10:11 PM
Our server came back online at 1.47am then offline again at 2.02am (UK). Hopefully won't be long until it's fully fixed!
Posted by Amphibulus, 01-12-2012, 10:16 PM
yeah, down here too, bit long for a complete datacenter downtime.
Posted by IGobyTerry, 01-12-2012, 10:20 PM
An update from our network engineers: OSPF protocols on our core routers failed without warning or reason, we have rectified the immediate issues, and are investigating the root cause to prevent any further reoccurrence
Posted by Frands, 01-12-2012, 10:27 PM
Thomas said that when it was up.. for a short while, why would you repeat it when everything is down again and the "preventing a reoccurrence" clearly failed?
Posted by Amphibulus, 01-12-2012, 10:30 PM
because now they are "preventing a re-reoccurrence"
Posted by QuickWeb-Roel, 01-12-2012, 10:35 PM
yes they are down still
Posted by QuickWeb-Roel, 01-12-2012, 10:42 PM
looks up now!
Posted by Thomas, 01-12-2012, 10:57 PM
It is back now. More info below. Status page being updated from network engineers: http://status.uk2.net/?p=1575
Posted by PeterVerrill, 01-13-2012, 12:30 AM
Down again?
Posted by Sieg, 01-13-2012, 12:35 AM
I guess so, I can't access my server.
Posted by cutabovehost, 01-13-2012, 12:36 AM
cloudkeeper.net and control.vps.net are both down as well for me currently. Odd thing is that the main site www.vps.net is up
Posted by PeterVerrill, 01-13-2012, 12:41 AM
Down here. We receive a total of 4 BGP feeds which went down a little while after each outage started though lots of routes inaccessible until everything stabilised.
Posted by QuickWeb-Roel, 01-13-2012, 01:12 AM
yeah down again!
Posted by ExonHost, 01-13-2012, 01:25 AM
down again.
Posted by IGobyTerry, 01-13-2012, 01:34 AM
Team is putting the finishing touches on the fixes to restore service. Running into a few difficulties, but should have something more substantial in the next couple of minutes.
Posted by Sieg, 01-13-2012, 01:49 AM
Back online, but for how long?
Posted by shadowlmd, 01-13-2012, 01:52 AM
Still offline for me.
Posted by IGobyTerry, 01-13-2012, 01:55 AM
They're bringing the network up vlan by vlan. Once all vlans are online, we expect things to go back to being stable without any significant issues.
Posted by shadowlmd, 01-13-2012, 01:57 AM
I hope so...
Posted by ExonHost, 01-13-2012, 01:59 AM
Back online.
Posted by CrocWeb, 01-13-2012, 02:24 AM
Their website says: Maybe they need to wait till 2013? Last edited by CrocWeb; 01-13-2012 at 02:29 AM.
Posted by cutabovehost, 01-13-2012, 02:37 AM
cloudkeeper.net still down for me
Posted by IGobyTerry, 01-13-2012, 02:51 AM
I have the team checking it out now. Should have it back online in no time.
Posted by cutabovehost, 01-26-2012, 07:21 PM
I wanted to give an update. The people at VPS.net followed up with me till they finally got things settled. So in the end all things were resolved and they gave me all the attention needed to get me rolling.