leaseweb having network issues [merged]

Portal Home > Knowledgebase > Articles Database > leaseweb having network issues [merged]

Posted by luke69, 03-16-2006, 07:00 PM
fyi: Time: Thu 16 Mar 2006 00:00 Expected Downtime: / Unknown Location: TeleCity / Redbus / InterXion / Schuberg Philis Reason: Under investigation / Unknown ******************************* Dear Customer, At the moment we experience some troubles in (part of) our network. We are doing our utmost to solve this issue a.s.a.p. As soon as we have more information available, we will publish it here. Please be patient while we are working on this problem. Regards, LeaseWeb BV
Posted by littlewhitey, 03-16-2006, 07:00 PM
My server with leaseweb (SBP datacentre) went down around 25 minutes ago Anyone else down or know what's happening?
Posted by littlewhitey, 03-16-2006, 07:02 PM
Heh, just posted about this in a seperate thread Hope they get this sorted soon..
Posted by luke69, 03-16-2006, 07:05 PM
its been about an hour so far, first it was one part of the network now it seems to be a different subnet that is affected good thing is that I got that email minutes after they first started having problems, so at least they are on the case
Posted by littlewhitey, 03-16-2006, 07:07 PM
The connection to mine had been flaky for about half an hour, then it just went completely My server is in 83.149.98.* and at their SBP datacentre
Posted by searchguy, 03-16-2006, 07:11 PM
My LW server at 85.17.1.* is also down.
Posted by luke69, 03-16-2006, 07:14 PM
yup, 85.17.x.x and 83.149.x.x networks seem to be affected
Posted by luke69, 03-16-2006, 07:32 PM
http://noc.leaseweb.com/ is back online, they are making progress
Posted by littlewhitey, 03-16-2006, 07:35 PM
Thanks for the update, hope the servers are back soon
Posted by littlewhitey, 03-16-2006, 07:42 PM
Time: Fri 17 Mar 2006 00:00 [22:00h] LeaseWeb just suffered from a large network outage. Telecity2 and Redbus were totally offline. Services are restored at the moment. A problem report will follow as soon as it comes available. [22:05h] It appears there are still some ranges offline. All are within the same VLAN. [22:30h] The problem with the remaining IP-ranges was found and solved. All customers should be online again. If you still experience any problems, please contact us. A full report will follow after evaluation of the problem. [00:30] Most of network connectivity has been restored. Only DSL and several customer vlans have not been restored yet.
Posted by Groovy, 03-16-2006, 07:53 PM
Our servers in the 83.149.x range in SBP are since 23:30 gmt+1 offline, techs are working on it
Posted by luke69, 03-16-2006, 07:59 PM
servers are still offline on the 83.149.73.x and 83.149.103.x subnets, plus 85.17.1.x and 85.17.11.x are suffering from about 40% packet loss
Posted by Groovy, 03-16-2006, 08:00 PM
83.149.123.x too
Posted by littlewhitey, 03-16-2006, 08:52 PM
Over 2 hours down here and counting..pretty disapointed with Leaseweb this time. I've never been down with them for this long before EDIT: Update on NOC site - [01:05] all services have been restored. A report will be communicated with customers at a later time. [01:40] Certain SBP customers are still experiencing problems. This is due to a (very coincedental?) hardware problem. An engineer is replacing the linecard within the next couple of minutes. So now I have to wait for a linecard to be replaced too.. Last edited by littlewhitey; 03-16-2006 at 09:00 PM.
Posted by Host3000, 03-16-2006, 09:08 PM
I have been talking to one of the engineers, and they have had a bad night. A layer 2 loop, causing ospf problems and bgp session drops, spreading to much more than only the LW network itself. For them I hope it is solved soon.
Posted by sambazone, 03-16-2006, 09:21 PM
Is anybody else experiencing network issues with leaseweb right now? Or is it just me?
Posted by Rafal_A, 03-16-2006, 09:22 PM
I see that part of servers from LeaseWeb is down, 2 servers are mine Does anybody have server down or got answer from LW? I wrote e-mail but they didnt answer cause there is night xoxo Rafal
Posted by Rafal_A, 03-16-2006, 09:24 PM
Yes, my servers are down too! Its about 85.*.*.* - I think and a few hours it is down
Posted by EuroVPS, 03-16-2006, 09:36 PM
This is keeping me awake as well.
Posted by DeathWolf, 03-16-2006, 09:38 PM
same here... it went totally poof down 5 mins ago
Posted by Rafal_A, 03-16-2006, 09:44 PM
I got this e-mail one minute ago. // http://noc.leaseweb.com/status.php?i=82
Posted by EuroVPS, 03-16-2006, 09:55 PM
Have you ever noticed that typically these things happen in the middle of the night with seemingly no advance warning. Today was supposed to have been a peaceful, calm evening as most Thursday nights usually are. ...
Posted by DeathWolf, 03-16-2006, 09:58 PM
sadly yes... it's made to stop us from proper sleep
Posted by EuroVPS, 03-16-2006, 10:18 PM
We are up at this time, seems like all service restored
Posted by .::DefCon::., 03-16-2006, 10:44 PM
yes I am experiencing difficulties with my server at Quickred (Leaseweb) . It is also 83.*.*.*
Posted by DeathWolf, 03-16-2006, 11:25 PM
still down in here:/
Posted by DeathWolf, 03-16-2006, 11:42 PM
back it is!
Posted by Chrysalis, 03-16-2006, 11:48 PM
http://noc.leaseweb.com/status.php?i=82
Posted by luke69, 03-17-2006, 01:04 AM
looks like its all back online, 6 hours of network problems with some networks being offline for 5 hours total. Their status page now says: "All services should be restored. Tomorrow we will conduct a full investigation concerning all events. Our sincere apologies for any/all inconvenience caused by aforementioned issues. Regards, LeaseWeb" I really hope they are able to find the issue and put in redundancy so that this will not happen again.
Posted by Groovy, 03-17-2006, 02:41 AM
Hah, its still not over. Already 8hours of downtime...
Posted by slowdive, 03-17-2006, 02:43 AM
Nah... my Quickred server is still out of reach.
Posted by Groovy, 03-17-2006, 02:44 AM
I dont think its the range, its the switches. Cause alot of ip's in my range are up tho
Posted by Groovy, 03-17-2006, 02:54 AM
Seems to be back up, dont wanna look at my mrtg graphs tho
Posted by WirralNet Matt, 03-17-2006, 04:16 AM
My servers are still down. In the 83.149.101.x range. Anyone else in this range still having problems?
Posted by slowdive, 03-17-2006, 04:43 AM
My IP is in the 83.149.74.x range and that is still down.
Posted by Groovy, 03-17-2006, 04:55 AM
email support@leaseweb.com or call them
Posted by WirralNet Matt, 03-17-2006, 05:06 AM
I emailed them earlier on, but i assume they must have a lot of emails to go through so it could take a while lol
Posted by Groovy, 03-17-2006, 05:42 AM
4mins reply on my emails
Posted by slowdive, 03-17-2006, 05:51 AM
I'm with Quickred.nl - just on Leaseweb IP space so I doubt I can email Leaseweb support. No sign of Quickred since their site is down and they only have email support.
Posted by slowdive, 03-17-2006, 06:06 AM
Well tha quickred site is back now. Hopefully it wont be long before my server is back.
Posted by Groovy, 03-17-2006, 06:39 AM
whois their domain and call that #
Posted by sersh, 03-17-2006, 06:51 AM
My server at 83.149.73.185 still down... Emailed 2 times. Waiting for reply to first email 10 hours, to second - 4 hours... Best support ever! And this is with service level 9!
Posted by Vpower, 03-17-2006, 07:00 AM
My was down for 1hour and 35 minutes. Hope they investigate this properly to avoid this happening again. Anyway this is my first server outage with them since I got server from them in august last year. Not to bad..
Posted by slowdive, 03-17-2006, 07:01 AM
Quickred were fast to reply to my email once their site was back. The network is not 100% back yet and they are working with Leaseweb to mop up the last remaining machines.
Posted by Groovy, 03-17-2006, 07:15 AM
Prob. their mailservers where on the same box /network /vlan Anyway, this is my first BIG network outage with leaseweb. Their network have been topnotch so far to me. Their service levels are only for hardware/software errors i think, not for network outages. Are you on volume bw?
Posted by sersh, 03-17-2006, 07:29 AM
Yes. I also had an issue with them when I was on vacation, my server hangs and they didn't do anything. I have returned from trip and see that my server was down 2 days. Have emailed them and they did the reboot... So, sometimes they can even ignore monitoring events...
Posted by Groovy, 03-17-2006, 07:43 AM
Read this: http://www.leaseweb.com/index.php?p=connectivity.type
Posted by slowdive, 03-17-2006, 09:10 AM
My server is finally back. That was too long.
Posted by sersh, 03-17-2006, 10:58 AM
my server still offline...
Posted by EuroVPS/Director, 03-17-2006, 11:47 AM
This type of thing in the middle of the day is generating a variety of problems for us as well as certainly thousands of other clients of Leaseweb colo/cage arrangements.
Posted by n1z, 03-17-2006, 11:50 AM
Ah mr EuroVPS, I understand you use leaseweb and this may be why our site, and yours is down? Hope this don't last long took time off work to work on site
Posted by EuroVPS/Director, 03-17-2006, 11:53 AM
Do you understand what this means
Posted by ViAdcK, 03-17-2006, 11:57 AM
I also got some quickred servers down.
Posted by n1z, 03-17-2006, 11:57 AM
Appears to be working now?
Posted by EuroVPS/Director, 03-17-2006, 11:57 AM
Part of network up for us at this time 1650 CET
Posted by EuroVPS/Director, 03-17-2006, 11:59 AM
All of our address space is currently reachable from US, RU and NL. Let's hope it stays this way permanently this time. This was the first serious network outage in SBP that we have experienced in our time there.
Posted by WirralNet Matt, 03-17-2006, 05:11 PM
Yep, I have had boxes at SBP via LeaseWeb since December 2004, and this is also the first major outage I have expereinced. Indeed over the last 5 months until today, the network uptime had been 100.00%, so I wont be leaving them in a hurry over an "unlucky" chain of events that caused this massive failiure, although I will be interested to see what action is taken as a result of the outage investigation, if any can be, to prevent something similar happening in the future.
Posted by Groovy, 03-18-2006, 06:00 AM
There a maintenance window on monday. they will fix the 2nd core switch then And fyi, attached are leaseweb's network stats from yesterday Attached Images alltraffic.png (33.8 KB, 16 views)