who unplugged the interweb? [Merged]

Portal Home > Knowledgebase > Articles Database > who unplugged the interweb? [Merged]

Posted by TheDedicatedReview, 10-21-2005, 02:09 AM
Is anyone else having issues with LT servers? Even their website appears to be down on our end.
Posted by bahn, 10-21-2005, 02:11 AM
So's sago and pretty much all of L3 it seem's http://www.internetpulse.net/ edit: getting 50% icmp packets returned now.. its trying to come back up but not doing so hot.
Posted by 8inet-Johnathan, 10-21-2005, 02:11 AM
Working fine for me
Posted by dollar, 10-21-2005, 02:11 AM
No problems here
Posted by bahn, 10-21-2005, 02:16 AM
looks like it was out for about ~11 minutes for me. Getting all my packets back and i'm establishing connections again.
Posted by killer15669, 10-21-2005, 02:25 AM
Nothing works for me
Posted by dollar, 10-21-2005, 02:26 AM
Post a trace
Posted by Scottl, 10-21-2005, 02:34 AM
Level3 is routing me to dallas(then dropping) for a cali address...
Posted by RyanD, 10-21-2005, 02:35 AM
http://scoreboard.keynote.com/scoreb...assword=public level3's network is garbage at the moment
Posted by investor, 10-21-2005, 02:36 AM
I can't access my servers at server4you.net :{*** Does anyone know what is going on ?
Posted by sprintserve, 10-21-2005, 02:36 AM
Sagonet seems fine now. I think they have routed around it.
Posted by bithost(NET), 10-21-2005, 02:37 AM
Did not experience any interruptions here, but I come in on Savvis. Bailey
Posted by killer15669, 10-21-2005, 02:38 AM
I am on SAAVIS too and everything is dead for 3 hours now...
Posted by hostable, 10-21-2005, 02:38 AM
Looks like all level3's peering points are down, http://scoreboard.keynote.com/scoreb...nap&Metric=TCP EDIT: Wow more in the red... Verio too :/
Posted by Cirrostratus, 10-21-2005, 02:38 AM
We are not experiencing any network problems at this time. The only maintainence we have had this evening was on our support portal database about 20 mins ago which has been completed. Level3 on the other hand is having some issues. http://scoreboard.keynote.com/scoreb...assword=public Thanks, Jeremy
Posted by RyanD, 10-21-2005, 02:40 AM
*kicks*level3
Posted by crackermun, 10-21-2005, 02:41 AM
I'm coming in through Adelphia and I can't get to Layered. I know it's a L3 issue - still its interesting with BGP that the routes haven't switched to another AS... -Paul
Posted by Kryl, 10-21-2005, 02:42 AM
Anyone find if the internet is being alittle weird this evening? Call center im at right now has been getting a few odd calls from different places. Tried accessing getfirefox.com and a couple of other sites, not resolving at the moment. Heh, internettrafficreport.com didn't want to resolve. Proably our own internal problems. Figured I would check though.
Posted by Cirrostratus, 10-21-2005, 02:45 AM
Now it appears Level3 and Verio are both down. Lovely. http://scoreboard.keynote.com/scoreb...assword=public Jeremy
Posted by getweb, 10-21-2005, 02:46 AM
Nope, I haven't been here in months and I came to get the skinny. We've been watching Keynote's scoreboard for about 45 minutes now... first Level3 started turning all red, almost completely down, then in the last 10 minutes Verio started doing the same. Now Savvis and SBC are starting to see some packet loss too. More and more sites are randomly becoming inaccessible.
Posted by crackermun, 10-21-2005, 02:46 AM
C'mon now - Wilma ain't that close
Posted by RyanD, 10-21-2005, 02:46 AM
oddly enough where we colo our stuff everything has rerouted to cogent heh shouldn't this be the other way around?
Posted by hostable, 10-21-2005, 02:46 AM
Anyone know whats causing this? they've been out for about 2 hours for me now...
Posted by Scottl, 10-21-2005, 02:48 AM
The L3 IP I was pinging looks fine now, routing within cali and pinging normal.
Posted by Hiccups, 10-21-2005, 02:48 AM
Been bad here too. Probably half the sites I've tried to go to wouldn't resolve. WHT wouldn't earlier either. Seems to be OK at the moment but no idea how long that will last. I'm on Roadrunner in North Carolina.
Posted by Cirrostratus, 10-21-2005, 02:48 AM
We are NOT offline. Level3 and Verio on the otherhand are having major issues if you look at the www.internethealthreport.com. If you are unable to view our site it is likely something to do with your local ISP. Many broadband ISP's rely on Level3 bandwidth. Thanks, Jeremy
Posted by getweb, 10-21-2005, 02:51 AM
http://www.digitaldorm.com/keynote.png Keynote's scoreboard as of 1:45am CDT /edit: nevermind... it's even way worser now. Could this be a peering war? Is that even realistic or just conspiracy chatter?
Posted by FHDave, 10-21-2005, 02:51 AM
There is currently problem across several networks. In particular, Level3 seems to be having problem at this moment. Internet Health Report as of 10/21/2005 2:30 AM EST: http://www.fluidhosting.com/images/ihr-102105.JPG
Posted by GetHoste, 10-21-2005, 02:53 AM
yup, level3 and verio are having router problems. i thought it was just me for a sec
Posted by Jeremy, 10-21-2005, 02:59 AM
Is it me or Level3 isn't very responsive. http://scoreboard.keynote.com/scoreb...assword=public Any ideas whats going on?
Posted by DeltaAnime, 10-21-2005, 02:59 AM
they went down with the L3 issues i imagine
Posted by RyanD, 10-21-2005, 03:03 AM
Internet health report was reporting a total outage for all of level(3) and verio. Whats the word? I've been sitting on hold with level3 for the past 30 minutes, seeing 0 traffic accross our lines.
Posted by investor, 10-21-2005, 03:06 AM
yeah, i think so too.
Posted by Kryl, 10-21-2005, 03:07 AM
Much the samething with stuff droping off at Level 3.
Posted by bahn, 10-21-2005, 03:07 AM
I suggest you take a look here: http://www.webhostingtalk.com/showth...hreadid=454190 Enjoy.
Posted by Russ Foster, 10-21-2005, 03:08 AM
I think someone just forgot to pay the bill Rus
Posted by Kryl, 10-21-2005, 03:10 AM
Thanks for the screen shot, just made the NOC persons day.
Posted by amps, 10-21-2005, 03:14 AM
Heard back from level3 (we use them in our mix), they were performing one of their unannounced (at least for us) nationwide software upgrades. Went horribly wrong apparently... good grief.
Posted by RyanD, 10-21-2005, 03:17 AM
This IS NOT a LT issue, which is why I created another thread.
Posted by bummer6666, 10-21-2005, 03:23 AM
First they de-peer XO , than Cogent and now rest of the internet pay up or get de-peered muhahahaha
Posted by pang, 10-21-2005, 03:25 AM
Yes, it looks like Level3 problem... My home connection: Tracing route to control.worria.com [205.234.133.138] over a maximum of 30 hops: 1 19 ms 21 ms 19 ms pcd-hhm9-2-rx.netvigator.com [203.218.114.254] 2 16 ms 16 ms 16 ms h6001-v851.netvigator.com [219.76.97.42] 3 16 ms 16 ms 16 ms 218.102.21.98 4 17 ms 17 ms 17 ms unknown.net.reach.com [134.159.129.113] 5 18 ms 17 ms 17 ms i-4-7.tmh-dist02.net.reach.com [202.84.153.90] 6 17 ms 17 ms 17 ms unknown.net.reach.com [202.84.153.205] 7 171 ms 172 ms 173 ms i-1-3.sjc-core01.net.reach.com [202.84.143.14] 8 230 ms 204 ms 250 ms unknown.net.reach.com [202.84.251.118] 9 168 ms 168 ms 168 ms ge-6-18.car4.SanJose1.Level3.net [4.68.111.109] 10 * * * Request timed out. 11 224 ms 220 ms 221 ms ae-0-0.bbr2.Chicago1.Level3.net [64.159.1.34] 12 ge-6-1.ipcolo2.Chicago1.Level3.net [4.68.101.73] reports: Destination host unreachable. Trace complete. server in a Hong Kong datacenter. traceroute to control.worria.com (205.234.133.138), 64 hops max, 40 byte packets 1 202.64.77.254 (202.64.77.254) 0.661 ms 0.729 ms 0.610 ms 2 202.64.154.53 (202.64.154.53) 195.245 ms * 0.494 ms 3 202.64.26.150 (202.64.26.150) 0.599 ms 0.770 ms 0.611 ms 4 g0-1.wtcr7004.pacific.net.hk (202.64.3.6) 1.609 ms 0.716 ms 0.860 ms 5 434.GE1-0-0.GW7.HKG2.ALTER.NET (202.130.160.165) 1.109 ms 37.031 ms 1.349 ms 6 27.so-3-0-2.XR2.HKG2.ALTER.NET (210.80.2.229) 1.608 ms 1.760 ms 1.732 ms 7 0.so-6-0-0.TR1.HKG2.ALTER.NET (210.80.51.193) 3.116 ms 2.182 ms 2.484 ms 8 0.so-1-0-3.IR1.LAX12.Alter.Net (210.80.50.41) 164.773 ms 164.604 ms 164.542 ms 9 POS3-0.IR1.LAX9.ALTER.NET (137.39.31.226) 172.998 ms 168.450 ms 165.246 ms 10 0.so-5-2-0.TL1.LAX9.ALTER.NET (152.63.0.146) 160.130 ms 160.334 ms 160.014 ms 11 * 0.so-5-0-0.TL1.CHI2.ALTER.NET (152.63.1.145) 214.849 ms 215.221 ms 12 0.so-6-0-0.XL1.CHI13.ALTER.NET (152.63.70.101) 219.479 ms 220.962 ms 219.980 ms 13 POS6-0.GW2.CHI13.ALTER.NET (152.63.71.205) 210.510 ms 211.323 ms * 14 uunet.ord.scnet.net (64.202.110.2) 217.631 ms 218.340 ms 217.216 ms 15 unknown.ord.scnet.net (205.234.205.58) 218.459 ms 218.295 ms 221.806 ms 16 control (205.234.133.138) 228.482 ms 218.370 ms 221.740 ms
Posted by bahn, 10-21-2005, 03:28 AM
Erm, of course it's not an LT issues - it's been stated countless times inside the thread, That it is an L3 issue. I don't really see the purpose in creating another thread about the same "issue" but fair enough if you want to do it that way.
Posted by RyanD, 10-21-2005, 03:30 AM
unfortuneatly people will not read that thread and just see "LT DOWN" and get the wrong impression
Posted by toma1708, 10-21-2005, 03:31 AM
Hi, Does anybody have problems reaching the tranxactglobal site from Europe ? Catalin
Posted by bahn, 10-21-2005, 03:32 AM
Ah, when you put it that way I can see your point. Seem's to be alot of that going around the past week ? Second time for L3 alone. big sigh!
Posted by RyanD, 10-21-2005, 03:34 AM
please see the other threads. Both Level(3) and Verio are experiencing major outages. http://www.internethealthreport.com
Posted by HD Fanatic, 10-21-2005, 03:39 AM
Anyone know what's going on with level3? I can't visit any of my boxes at Sago. my isp uses level3 too.
Posted by kkikki, 10-21-2005, 03:43 AM
21st OCT, is SAGO down AGAIN right now?
Posted by bahn, 10-21-2005, 03:46 AM
My sago boxes are up - All of 'em. I recommend you read: http://www.webhostingtalk.com/showth...hreadid=454212 L3 and Verio bit the dust again
Posted by hostable, 10-21-2005, 03:58 AM
Look's like they are sort'a back online... 4 9 ms 7 ms 9 ms pos6-0-OC-192.tampflerl-rtr4.tampabay.rr.com [65 .32.8.137] 5 9 ms 9 ms 9 ms 4.79.174.1 6 10 ms 11 ms 9 ms 4.68.104.69 7 13 ms 11 ms 12 ms 209.244.26.73 8 40 ms 38 ms 47 ms 64.159.8.101 9 37 ms 32 ms 34 ms 209.244.26.217 10 130 ms 121 ms 123 ms 64.159.8.29 11 124 ms 125 ms 123 ms fe-0-1-0.mc2.Dallas1.Level3.net [209.244.26.254] 12 163 ms 211 ms 166 ms fe-1-1-0.mc2.Atlanta1.Level3.net [64.159.8.65] 13 178 ms 175 ms 175 ms fe-1-1-0.mc2Cincinnati1.Level3.net [64.159.8.85] 14 330 ms 217 ms 256 ms fe-0-1-0.mc1.Cincinnati1.Level3.net [209.244.26. 181] 15 207 ms 196 ms 195 ms fe-1-1-0.mc1.Chicago1.Level3.net [64.159.8.17] 16 106 ms 107 ms 106 ms ge-7-0-9.core2.Chicago1.Level3.net [209.244.25.4 ] 17 * 123 ms * ae-1-51.bbr1.Chicago1.Level3.net [4.68.101.1] 18 * 117 ms * so-6-0-0.edge1.Chicago1.Level3.net [209.244.8.10 ] 19 * 119 ms 123 ms he-level3-oc12.Chicago1.Level3.net [4.68.127.66] 20 144 ms 141 ms 149 ms pos1-0.gsr12416.nyc.he.net [64.71.157.61] 21 144 ms 144 ms * pos3-3.gsr12416.ash.he.net [65.19.129.5] 22 * 189 ms 189 ms pos5-0.gsr12012.dal.he.net [66.160.184.13] 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 * * * Request timed out.
Posted by nsdesign, 10-21-2005, 04:08 AM
Unfortunately yes - and causing us lots of problems!!
Posted by 8inet-Johnathan, 10-21-2005, 04:10 AM
looks like things are all good for verio now level3 is doing a little better Public Stats URL: http://scoreboard.keynote.com/scoreboard/Main.aspx Lgon info: public/public **EDIT**: looks like everything is critical accross the board for level3 & verio now
Posted by pang, 10-21-2005, 04:15 AM
It seems that Level3 is back, though I am not very happy with this nationalwide outage.. I am not sure how Level3 managed to do this.
Posted by 8inet-Johnathan, 10-21-2005, 04:19 AM
level 3 is not back anymore it wasn't at critical for a few moments but its back at it now again
Posted by 8inet-Johnathan, 10-21-2005, 04:41 AM
things seem to be looking a bit better for level3 now..
Posted by nsdesign, 10-21-2005, 04:48 AM
Looks like we're back
Posted by frdalton, 10-21-2005, 05:04 AM
I've been doing server monitoring tonight for about 20 or 30 hosting companies and have had connectivity issues with a few nocs tonight, Fastservers, Anet, Navisite, Rackspace Alot of it seems to be random routing issues affecting one area or another is affected when the rest of the world isn't.
Posted by 8inet-Johnathan, 10-21-2005, 05:05 AM
level3 appears to be doing much better currently no change on verio though, level3 is at about 50% of normal capacity i'd say
Posted by 8inet-Johnathan, 10-21-2005, 05:07 AM
level3 appears to be doing much better currently! they most of solved the problem
Posted by Paul, 10-21-2005, 05:07 AM
Leaving my bgp session with lvlt down till morning
Posted by myusername, 10-21-2005, 05:13 PM
and now it is Cogent across the boards....
Posted by getweb, 10-21-2005, 05:20 PM
Whoa weird... obviously a few threads have been merged into one? I didn't know they could do that, shows how long I've been gone.
Posted by PixelManual, 10-21-2005, 05:36 PM
lol, almost every forum software out there can do this I wonder how this will affect providers...
Posted by Chrysalis, 10-22-2005, 10:13 AM
invalid link specified what I miss?
Posted by layer0, 10-22-2005, 10:49 AM
That thread was merged with this one.