ThePlanet down [merged]

Portal Home > Knowledgebase > Articles Database > ThePlanet down [merged]

Posted by Russ Foster, 03-31-2005, 06:15 AM
Anyone else seeing TMS down? Rus
Posted by sprintserve, 03-31-2005, 06:16 AM
ThePlanet is down. All our servers there is unreachable and blipping bright red on the monitors. Even Orbit is down.
Posted by fashion, 03-31-2005, 06:17 AM
glad it isnt just me... i almost went into a panic when i saw our whole planet network was down...
Posted by neutro, 03-31-2005, 06:17 AM
no wonder
Posted by trustedurl.com, 03-31-2005, 06:18 AM
same here... sigh... 8 ge-2-3-0.r01.asbnva01.us.bb.verio.net (206.223.115.12) 16.644 ms 16.287 ms 16.107 ms 9 p16-0-1-2.r20.asbnva01.us.bb.verio.net (129.250.2.60) 16.393 ms 16.438 ms 16.228 ms 10 p16-0-1-1.r21.dllstx09.us.bb.verio.net (129.250.5.34) 34.456 ms 34.809 ms 34.462 ms 11 ge-1-0-0.a00.dllstx09.us.ra.verio.net (129.250.28.167) 34.678 ms 34.344 ms 34.566 ms 12 ge-1-2.a00.dllstx04.us.ra.verio.net (129.250.28.186) 34.590 ms ge-1-1.a00.dllstx04.us.ra.verio.net (129.250.31.44) 34.850 ms ge-1-2.a00.dllstx04.us.ra.verio.net (129.250.28.186) 34.587 ms 13 ge-9-3.a00.dllstx04.us.ce.verio.net (157.238.228.38) 34.980 ms 34.908 ms 35.065 ms 14 * * *
Posted by herculesnetwork, 03-31-2005, 06:18 AM
anyone know what's going on ?
Posted by Przemek, 03-31-2005, 06:19 AM
My server at ThePlanet is down too.
Posted by A|J, 03-31-2005, 06:19 AM
I cant seem to access my server at ThePlanet.com or www.theplanet.com or orbit.theplanet.com at the moment. Anyone else experiencing the same problem?
Posted by sebastian_d1, 03-31-2005, 06:19 AM
Are they down ? thanks for feedback
Posted by JDMundo, 03-31-2005, 06:19 AM
It looks like all of the Planets network is down. Called them up: "Yeah, we're having some technical difficulties, we're investigating."
Posted by ImLagging, 03-31-2005, 06:19 AM
ThePlanet's DLLSTX2 is down, but DLLSTX4 is up for me.
Posted by ellinas, 03-31-2005, 06:19 AM
Greece + Cyprus... TP - Unreachable
Posted by resellertr, 03-31-2005, 06:20 AM
Same here... Whole The Planet is down...
Posted by neutro, 03-31-2005, 06:20 AM
only one of our servers is alive (dls 4) dls2 out
Posted by Protollix, 03-31-2005, 06:20 AM
yup whole network seems to be down.
Posted by sebastian_d1, 03-31-2005, 06:21 AM
did they give you a time estimate ? i don't have their phone number here ...
Posted by trustedurl.com, 03-31-2005, 06:21 AM
yep... 8 ge-2-3-0.r01.asbnva01.us.bb.verio.net (206.223.115.12) 16.644 ms 16.287 ms 16.107 ms 9 p16-0-1-2.r20.asbnva01.us.bb.verio.net (129.250.2.60) 16.393 ms 16.438 ms 16.228 ms 10 p16-0-1-1.r21.dllstx09.us.bb.verio.net (129.250.5.34) 34.456 ms 34.809 ms 34.462 ms 11 ge-1-0-0.a00.dllstx09.us.ra.verio.net (129.250.28.167) 34.678 ms 34.344 ms 34.566 ms 12 ge-1-2.a00.dllstx04.us.ra.verio.net (129.250.28.186) 34.590 ms ge-1-1.a00.dllstx04.us.ra.verio.net (129.250.31.44) 34.850 ms ge-1-2.a00.dllstx04.us.ra.verio.net (129.250.28.186) 34.587 ms 13 ge-9-3.a00.dllstx04.us.ce.verio.net (157.238.228.38) 34.980 ms 34.908 ms 35.065 ms 14 * * *
Posted by A|J, 03-31-2005, 06:21 AM
wow... i just posted a similar thread exactly at the same time.. crazyness Anyway here's my tracert
Posted by richenou, 03-31-2005, 06:21 AM
yes 25 dedicated server down but not all of the network i have 80 other who are working.
Posted by Fahd, 03-31-2005, 06:21 AM
Yup! Just noticed that myself.
Posted by djwinter, 03-31-2005, 06:22 AM
Same here.
Posted by alpha, 03-31-2005, 06:24 AM
Merged all ThePlanet threads that popped up within the last 5 minutes into this one... Let's read the recent threads before we post, folks
Posted by herculesnetwork, 03-31-2005, 06:25 AM
how can they claim 100% uptime and their main site is even down ??
Posted by trustedurl.com, 03-31-2005, 06:25 AM
I'm on 4, aren't I?... no go either... 13 ge-9-3.a00.dllstx04.us.ce.verio.net (157.238.228.38) 34.980 ms 34.908 ms 35.065 ms 14 * * *
Posted by simscripts, 03-31-2005, 06:25 AM
even www.theplanet.com is down. gogod thing I have a backup server
Posted by sam2579, 03-31-2005, 06:25 AM
Planet site and ALL SERVERS are down at this point. Verified them by calling them and they are aware of it. They dont have any ETA either. Wow who thought of that??? :-)
Posted by EricP, 03-31-2005, 06:26 AM
Down here. Not good.
Posted by richenou, 03-31-2005, 06:26 AM
hi, theplanet down here ?
Posted by Protollix, 03-31-2005, 06:27 AM
we have some servers at DLLSTX4 that appear down as well
Posted by hosttone, 03-31-2005, 06:27 AM
Yes, same with us. We have 3 servers, 2 down and 1 up.
Posted by JDMundo, 03-31-2005, 06:27 AM
Yeah, so much for 100% network uptime. Frustrating.
Posted by TotalChoice - Bill, 03-31-2005, 06:27 AM
All our boxes in The Planet are down as well.
Posted by wheimeng, 03-31-2005, 06:27 AM
That's bad
Posted by trustedurl.com, 03-31-2005, 06:27 AM
power issues maybe? (merely speculating).
Posted by tomf, 03-31-2005, 06:27 AM
how can this happen :/ Also waiting for my server to come up again...
Posted by ImLagging, 03-31-2005, 06:28 AM
Strange. I have 2 servers in DLLSTX2 that are down, and my 2 servers in DLLSTX4 are up.
Posted by barry[CoffeeSprout], 03-31-2005, 06:28 AM
It happens :-)
Posted by EricP, 03-31-2005, 06:28 AM
What could cause such a huge outage?
Posted by uka, 03-31-2005, 06:28 AM
yep. its down
Posted by trustedurl.com, 03-31-2005, 06:29 AM
Right now, I'd speculate a power problem... but hey, I have 0 clue what's going on atm. And what's with the 90 second post delay?
Posted by hosttone, 03-31-2005, 06:29 AM
Theplanet down. Waiting them to fix it.
Posted by Cephren, 03-31-2005, 06:31 AM
yep our servers at the planet down also. (see we do have night tech support too lol) Wow all down...worse of all....its their skeleton twilight shift on duty.... I feel so sorry for them. I guess they are calling in all the higher tier tech guys in right now. If it was a power problem, the backups would have switched on by now..... i just hope the junipers didnt blow out....
Posted by GIGANET, 03-31-2005, 06:32 AM
Is the Planet down ? a whole fleet of our servers are down, and so is theplanet.com and servermatrix.com. I can't even contact them.
Posted by Patryk, 03-31-2005, 06:33 AM
Same here, in Poland .
Posted by alexrobonlin, 03-31-2005, 06:33 AM
Hi Guys, Just wondering if anyone else is experiencing complete outages across theplanet's DSLX 1 datacentre? Alex
Posted by A|J, 03-31-2005, 06:33 AM
Has anyone tried calling them?
Posted by WebAfrica, 03-31-2005, 06:33 AM
The planet is down. I phoned them now... http://216.239.59.104/search?q=cache...en&lr=&strip=1 been about 10 minutes so far.
Posted by AlexAT, 03-31-2005, 06:34 AM
I have several outages during the day and now seems it is completely down (tested from different locations). Anybody can confirm this?
Posted by EricP, 03-31-2005, 06:34 AM
This must be huge. A whole 'Enterprise' datacenter taken offline. Can you imagine what their phones must be like right now?
Posted by webideas, 03-31-2005, 06:34 AM
I just got notification from our monitoring service indicating the servers going down housed at THEPLANET. I can't even access theplanet or SM site..anybody have any news ??
Posted by UH-Matt, 03-31-2005, 06:34 AM
Bah, we host our helpdesk on a box at TP
Posted by moheybee, 03-31-2005, 06:35 AM
Mother of god! We just moved to The Planet a week ago. I can't believe this.
Posted by Cephren, 03-31-2005, 06:35 AM
I call them...no ETA...the guy doesnt even know what is wrong. (or maybe he is not allowed to tell)
Posted by herculesnetwork, 03-31-2005, 06:35 AM
is that what they call 100% uptime
Posted by A|J, 03-31-2005, 06:36 AM
I'm sure they must have some sort of pre-recorded information announcing the outage if thats the case.
Posted by EricP, 03-31-2005, 06:36 AM
This really makes me want to look at some type of DNS failover solution. At least for our main site. Can anyone recommend anything?
Posted by pstation, 03-31-2005, 06:36 AM
DC 2 just seems to be down, my servers in DC 4 are working fine
Posted by trustedurl.com, 03-31-2005, 06:37 AM
Well, since it seems to affect random boxes on DLLSTX1, 2 and 4. I really would think it's a power problem. edit: but I could be wrong. Anyone got any satisfactory answer from the techs?
Posted by websitezone, 03-31-2005, 06:38 AM
I cannot get to ServerMatrix / any of my sites from the UK. Anyone else experiencing this...?
Posted by Cephren, 03-31-2005, 06:38 AM
colo is down.
Posted by immutability, 03-31-2005, 06:38 AM
Hi there, We've got our server hosted with ServerMatrix.com, and we have been unable to access it (ping times out) for about 20 minutes now. But the major problem is that I cannot access their corporate websites and administration interface either! Both www.theplanet.com and www.servermatrix.com will not resolve to an IP address. Is anyone who has got a server with them having these problems? Thanks, Rado
Posted by uka, 03-31-2005, 06:38 AM
3 of our 5 are down. 2 up.
Posted by concorde, 03-31-2005, 06:38 AM
I found that all sites I know(inlude mine) at the planet are down. Unable to access servermatrix.com and theplanet.com What's wrong? Alien attack?
Posted by A|J, 03-31-2005, 06:38 AM
Try www.dnsmadeeasy.com... They were the first ones who notified me on my cell about the outage. Pretty good service and all for $29/year
Posted by WebAfrica, 03-31-2005, 06:38 AM
They sounded pretty stressed when I called them. They still don't know whats going on.
Posted by Homer-HB, 03-31-2005, 06:39 AM
All is good for my boxen in STX4, but the whole of STX2 is down. :\
Posted by net-trend, 03-31-2005, 06:39 AM
Same here...not happy jan!
Posted by Lem0nHead, 03-31-2005, 06:39 AM
it looks like all their network is down probably the cleaning girl pulled some power plug but seriously... that's almost 30 minutes now and i'm getting worried
Posted by ImLagging, 03-31-2005, 06:39 AM
I remember that they had a pretty big DC wide outtage last year around this time. Anyone else think this is a spring time event that happens every year with them?
Posted by trustedurl.com, 03-31-2005, 06:40 AM
cleaning out excess clients?
Posted by A|J, 03-31-2005, 06:41 AM
Yup, the phones were ringing off the hook in the background but apparently, they don't even know what is going on...
Posted by Snowman30, 03-31-2005, 06:41 AM
We have servers across in their DC's and all are down. Im not even able to get thru on the phone to them anyone heard anything official at all?
Posted by ankita, 03-31-2005, 06:41 AM
Even my server is unreachable. nothing's getting connected. Neither Orbit nor the planet site.
Posted by Cephren, 03-31-2005, 06:41 AM
Spring cleaning....... the janitor accidently unplugs the main router... to plug in his vaccum
Posted by WebMate, 03-31-2005, 06:41 AM
I just talked to the support and they said everything should be fine within a few minutes. Not a start of the day, but let's hope it won't last long. ;-)
Posted by trustedurl.com, 03-31-2005, 06:42 AM
What's their toll free # again? I can't believe they have 0 idea what's going on...
Posted by ImLagging, 03-31-2005, 06:43 AM
I've never thought of spring cleaning in this way before.
Posted by pstation, 03-31-2005, 06:43 AM
Same here, i use theplanet for my DNS hosting, and it seems both of the DNS servers are hosting in DC2, which are down. Figured they would've put they in the different datacenters :/
Posted by bidder, 03-31-2005, 06:43 AM
Does this happen a lot with theplanet? I have been with them for little over 2months and they have been great up until now... Both of our servers are down
Posted by eugyx, 03-31-2005, 06:43 AM
Are they going to compensate for the damages caused by this outgage?
Posted by geeks4help, 03-31-2005, 06:43 AM
Anyone having problem with your server at Layered tech?
Posted by EricP, 03-31-2005, 06:43 AM
Well, the frontline techs don't know what's going on at least. You can bet they have engineers working on it. TP has over 200 employees. They said they have power. There were no explosions or fires They said they simply have 0% connectivity.
Posted by bjdea1, 03-31-2005, 06:44 AM
Anyone else noticing www.theplanet.com is not online? As well as www.servermatrix.com and https://orbit.theplanet.com Anyone know whats going on?
Posted by barry[CoffeeSprout], 03-31-2005, 06:44 AM
Somebody jumped the gun on Aprils fool?
Posted by lessthanthree, 03-31-2005, 06:44 AM
i have 2 servers down i *THINK* they are in DC2, not 100% sure on that
Posted by bheka, 03-31-2005, 06:44 AM
What compensation .. dream on .. we'll be luck if we get a discount
Posted by sam2579, 03-31-2005, 06:44 AM
So much for 100% uptime guarantee :-)
Posted by accyroy, 03-31-2005, 06:45 AM
My servermatrix box is up but my planet box is down, i do believe theplanet boxes have a 100% sla....
Posted by trustedurl.com, 03-31-2005, 06:45 AM
hmmm, but why are the outages so spread out? Anyhow, we'll see what happens.... eugyx -> last time there was a major issue they did issue a credit, but of course you'd have to talk to them directly.
Posted by Cephren, 03-31-2005, 06:45 AM
dedicated....probably no compensation... colo is another story.
Posted by zsindhu, 03-31-2005, 06:45 AM
Has anyone called them up yet? WebAfrica, what was their response when you called?
Posted by pstation, 03-31-2005, 06:45 AM
ive been there 2 years, and i cant remember any outage like this
Posted by EricP, 03-31-2005, 06:46 AM
Accounting is going to be just as stressed. You can bet SLA requests are going to beging rolling in ASAP.
Posted by dolay, 03-31-2005, 06:46 AM
d#mn i made huge mistake to cancel only one of our several servers this month, i really have to cancel all of our servers. ThePlanet you are really big looser
Posted by net-trend, 03-31-2005, 06:46 AM
Dedicated clients will get compensation provided they can tell them when the server went down and for how long.
Posted by alexrobonlin, 03-31-2005, 06:47 AM
Just phoned them - phones ringing off the hook. They sounded pretty stressed & lots of voices in the background. No fires, etc... and they're saying that no boxed are wrong - it's "higher up the chain" Probably a power outage to some routers... Either that or their carriers have gone wonky... But I think the problem is more routed with them... 32 mins outage now! :O Alex
Posted by websitezone, 03-31-2005, 06:47 AM
Wow the speed this thread is growing just shows how big SM/TP are!
Posted by sam2579, 03-31-2005, 06:48 AM
Well they can start counting the time their own site went down till the time it came back up. We shouldnt be telling them the time.....
Posted by pstation, 03-31-2005, 06:48 AM
Yeah, and all of SM/TP isnt even down, and plus its 5 in the morning
Posted by A|J, 03-31-2005, 06:48 AM
1800-854-7679
Posted by woxcom, 03-31-2005, 06:48 AM
This is really bad All our servers are down!! SM seems to be down as well!!! Any ideas what's going on there?
Posted by Laws, 03-31-2005, 06:48 AM
<> Moving on.... I have faith that William is on the issue and will get it resolved, my server at DLLSTX4 is currently online. Although all my TMS servers are down... I'm sure it'll be online and an RFO posted shortly... Last edited by anon-e-mouse; 03-31-2005 at 07:33 AM.
Posted by Mix, 03-31-2005, 06:48 AM
Grrrr my phone is ringing off the hook
Posted by trustedurl.com, 03-31-2005, 06:48 AM
not for everyone
Posted by alexrobonlin, 03-31-2005, 06:49 AM
That actually doesn't help!? The Planet are doing everything they can, I'm sure. They don't want these problems, just like we don't remember Alex
Posted by Servstra-Sales, 03-31-2005, 06:49 AM
So do we.
Posted by abang, 03-31-2005, 06:49 AM
2 of our servers are down. Good thing our new server in LayeredTech is still up.
Posted by lessthanthree, 03-31-2005, 06:49 AM
my servermatrix box just came back up my theplanet box is still down my brother's servermatrix box was down, its back up.
Posted by EricP, 03-31-2005, 06:49 AM
You have to keep one think in mind. The Planet is at the TOP of the game. They have ENTERPRISE datacenters and ENTERPRISE Engineers. They have over 200 employees. This is not some small operation here. They are on top of things. These things happen to the best. Nothing is 100%. Things happen and you can bet they have the best of the best working on it right now.
Posted by ImLagging, 03-31-2005, 06:50 AM
In about 2 years being with them, this is only the second time there's been a major network outtage. The first time was last year around this same time for about an hour. Other then that, there has been a small amount (a few minutes at most) of downtime and that has been few and far between. At least with my servers.
Posted by trustedurl.com, 03-31-2005, 06:50 AM
It's been 40 minutes now... I'll be so grumpy today... can't wait to get a coffee! My pager went off 5.10 EST.
Posted by eugyx, 03-31-2005, 06:50 AM
sam2579: agreed 100%, they should give some form of compensation, not talking about losses derived from their problems, but at least credit towards next invoice.
Posted by Lem0nHead, 03-31-2005, 06:51 AM
i guess heads will roll if they have a 10% compensation... think about millions/month they make boss won't like it
Posted by Cephren, 03-31-2005, 06:51 AM
my big boys in colo are still down............... one dedi just came back up..........
Posted by dolay, 03-31-2005, 06:51 AM
this is happening while a high googlebot actvity appearing... possible a quarterly PR update and we are missing this PR update with blank content. Our customers(all of them are webmsters) will kill us Last edited by sprintserve; 03-31-2005 at 09:57 AM.
Posted by alexrobonlin, 03-31-2005, 06:51 AM
No, it doesn't happen a lot Al
Posted by c0bra, 03-31-2005, 06:51 AM
TP do offer a SLA. They will compensate for any downtime.
Posted by bugsoft, 03-31-2005, 06:52 AM
They have no clue what's going on. I've called spoke with several guys. No ETA at the moment. Servers on DLX4 are up thought.
Posted by alexrobonlin, 03-31-2005, 06:53 AM
Indeedy
Posted by JDMundo, 03-31-2005, 06:53 AM
EricP, with all due respect. These things are not supposed to happen with an "enterprise" datacenter with supposedly full redundancy and 100% network uptime guarantee. 40 minutes of downtime so far. Just not acceptable.
Posted by zsindhu, 03-31-2005, 06:53 AM
Right now, the biggest concern should be what's going on and how long will the downtime last. We can worry about compensations later.
Posted by AlexAT, 03-31-2005, 06:53 AM
For all who regret about using TP/SM: don't do it. Their network is much than strong, at least for near 2 years I'm with them. There were only a couple of downtimes, and only 1-2-3 times they were long - more than 30 minutes. It is **very** good result.
Posted by simscripts, 03-31-2005, 06:53 AM
my server is still down, same http://www.theplanet.com/
Posted by trustedurl.com, 03-31-2005, 06:53 AM
Not for everyone
Posted by Nime, 03-31-2005, 06:54 AM
--- Last edited by Nime; 03-31-2005 at 06:58 AM.
Posted by alexrobonlin, 03-31-2005, 06:54 AM
Quite... Don't let's have another "Burst.net" on our hands
Posted by jeroman, 03-31-2005, 06:54 AM
hm... Theplanet 100% uptime gurantee ?
Posted by princo, 03-31-2005, 06:54 AM
It's 1 pm in the Europe our phone doesn't stop ringing
Posted by pstation, 03-31-2005, 06:54 AM
looks like internet in the part the support people are at went down too
Posted by Lem0nHead, 03-31-2005, 06:55 AM
i agree otherwise, what's full redundancy for?
Posted by alexrobonlin, 03-31-2005, 06:56 AM
Yes, I'd support that - tbh, they've been bloody brilliant for the past 15 months we've been using them Alex
Posted by TotalChoice - Bill, 03-31-2005, 06:56 AM
Two rolls of toilet paper in the bathroom, that's redundancy. 45 minutes of downtime is not.
Posted by TopHostSupport, 03-31-2005, 06:56 AM
Dead, Dead, Dead....they are all dead. /me needs coffee, smoke, ugggg 6am. Ticket system *bong bong*....go away it's not me. Live life in the fast lane, become a hoster!
Posted by bheka, 03-31-2005, 06:56 AM
Small downtime, less than a day or two will not affect your google rankings or their updates
Posted by Hostasaurus, 03-31-2005, 06:56 AM
I believe they claim 100% though; they shoudn't claim an unattainable uptime since the other posts show they haven't been able to maintain 100% prior to this event.
Posted by inteltechs, 03-31-2005, 06:56 AM
our servers with layeredtech are up, but all all servers with TP/SM are down.
Posted by EricP, 03-31-2005, 06:57 AM
I somewhat agree. It shouldn't have happened, but we can't really argue that point because we have no idea what really did happen. Now, you can bet that upper management will be working on preventing whatever did happen from happening again. You think this is stressful for us? Imagine what it's like for them. I have full trust in TP that they will fix the issue and ensure it won't happen again. You don't get where TP is at without doing exactly that.
Posted by A|J, 03-31-2005, 06:57 AM
Bill, great analogy...i'll have to agree with you.
Posted by ffeingol, 03-31-2005, 06:57 AM
I can confirm that also. we have 4 servers down, but 1 up. Called and the support guy I talked to had no info and no eta on info. Frank
Posted by Lem0nHead, 03-31-2005, 06:57 AM
now it's more than 44 minutes (0.01%) so even ServerMatrix should get a 10% compensation
Posted by eugyx, 03-31-2005, 06:58 AM
what really is bothering me is the "no idea" answer in regard the problem. I know they have real good people there dealing with their operations, but that answer makes them look like a bunch of kids trying to find a solution just rebooting everything.
Posted by Cephren, 03-31-2005, 06:58 AM
oh boy oh boy oh boy...the sun will come up soon.....in east coast.....
Posted by tamouh, 03-31-2005, 06:58 AM
Things always happen, I'm confident they are working on the problem. I just would have preferred to have a slow connection, or intermittent network downtime than this (so far) 1/2 hour downtime. back to playing AA until they get back up!
Posted by accyroy, 03-31-2005, 06:58 AM
your lucky, its midday in the UK, we are already being hit by calls/tickets!
Posted by suhailc, 03-31-2005, 06:59 AM
One of my servers at Servermatrix is up and running so hasn't affected SM servers, and won't hopefully! Amazing that their site is unavailable considering ThePlanet offers 100% uptime to customers. Here is DNS report for their domain: DNS Report for theplanet.com Generated by www.DNSreport.com at 10:57:15 GMT on 31 Mar 2005. Category Status Test Name Information Parent PASS Missing Direct Parent check OK. Your direct parent zone exists, which is good. Some domains (usually third or fourth level domains, such as example.co.us) do not have a direct parent zone ('co.us' in this example), which is legal but can cause confusion. INFO NS records at parent servers Your NS records at the parent servers are: ns1.theplanet.com. [216.234.234.30] [TTL=172800] [US] ns2.theplanet.com. [12.96.160.115] [TTL=172800] [US] [These were obtained from c.gtld-servers.net] PASS Parent nameservers have your nameservers listed OK. When someone uses DNS to look up your domain, the first step (if it doesn't already know about your domain) is to go to the parent servers. If you aren't listed there, you can't be found. But you are listed there PASS Glue at parent nameservers OK. The parent servers have glue for your nameservers. That means they send out the IP address of your nameservers, as well as their host names. NS FAIL NS A timeout occurred getting the NS records from your nameservers! None of your nameservers responded fast enough. They are probably down or unreachable. I can't continue since your nameservers aren't responding. If you have a Watchguard Firebox, it's due to a bug in their DNS Proxy, which must be disabled..
Posted by concorde, 03-31-2005, 06:59 AM
A few seconds ago my server is up and down again.
Posted by Hostasaurus, 03-31-2005, 06:59 AM
Just an FYI for those who are interested; they're still advertising their networks in BGP via at least Level 3, Global Crossing and UUNet whom they directly peer with; this means their border routers are up and active, so the problem must be more internal than that. I noticed it when I was trying to download Photopost and Vbulletin updates; they both must host there, or at least run their DNS from servers there.
Posted by ultrafex, 03-31-2005, 07:00 AM
Their master network engineer did a poor job. It's really not acceptable to have a single point of failure like that. This situation probably has to do with the fact that DC4 can only be reached through DC2. A poor decision. We can only hope they change that.
Posted by Cephren, 03-31-2005, 07:00 AM
they got a skeletal night shift on duty.... like all enterprise data centers, the majority of the top talent dont need to go on night shifts..... but I am sure all of them are either driving back to work right now or are back.
Posted by neutro, 03-31-2005, 07:00 AM
this why i don't like putting all boxes in one dc.. 2 down 3 up thank god 6.58pm in Asia .. even worse.. phone keep on ringing..
Posted by UH-Matt, 03-31-2005, 07:00 AM
I feel for you guys
Posted by TopHostSupport, 03-31-2005, 07:01 AM
Have you looked out the window? The sun is up.
Posted by Unknown_User, 03-31-2005, 07:01 AM
Yet it is difficult to maintain a large network such as a ThePlanets' and something major that could not be controlled may of happened.
Posted by Jz1977, 03-31-2005, 07:02 AM
Yes, it is down here too. They are located in TX, correct? Wonder what happend to bring down an enitre datacenter. That would have to be a simple as router to a disaster as a fire. However, I seriosly doubt it would be the last. Maybe they forgot to pay their connection bill. lol -Casper
Posted by PierreB, 03-31-2005, 07:02 AM
We have 6 boxes with Theplanet, 3 of them are down while the other 3 are up, seem's to be an isolated part of the network being down. Lets hope they bring that back up asap. thanks Matt
Posted by Jon69, 03-31-2005, 07:02 AM
What a nightmare. All we can do is hope (and whine)
Posted by eniki, 03-31-2005, 07:02 AM
In the UK here too...shame it happened and the day after one of our servers went wild anyway. I can understand how good they normally are and what pressure they must be under now but it's not going to stop me pressing for some kind of compensation.
Posted by Cephren, 03-31-2005, 07:03 AM
its starting to light up, but no sun yet....Toronto.
Posted by Angel78, 03-31-2005, 07:03 AM
13:00 in Europe (austria)...
Posted by galantina, 03-31-2005, 07:03 AM
I noticed it 50 minutes ago and still nothing. It has to be something
Posted by Laws, 03-31-2005, 07:03 AM
Before slamming something you don't understand you might like to get your facts straight. That fact is WRONG and has been for months, the datacenters (2,3,4,5 are all interlinked).
Posted by chrisbond, 03-31-2005, 07:03 AM
Any update from theplanet on what exactly has happened?
Posted by TotalChoice - Bill, 03-31-2005, 07:04 AM
I know, It's a migration by MR. GOLD?
Posted by bheka, 03-31-2005, 07:04 AM
Okay .. enough stressing for me ... let me grab a bite to eat and some coffee .. come back in half an hour and hopefully everything will be up ... sitting here listing to everyone complaining is not making this any better.
Posted by foxgroup, 03-31-2005, 07:04 AM
I was just sending out a mailshot to 20,000 members and it had reached just 5000. Now I have a dead website that people are going to and I cannot even find the telephone number for the planet. Its not my day today.............. Even their website is down at the moment at www.theplanet.com.
Posted by eniki, 03-31-2005, 07:04 AM
I've given up hope on the it'll be back in a few minutes promise
Posted by trustedurl.com, 03-31-2005, 07:05 AM
well, 55 minutes so far...
Posted by WebAfrica, 03-31-2005, 07:05 AM
At least this gives us something to do. It really sucks being at the mercy of others...
Posted by Laws, 03-31-2005, 07:05 AM
Also, thought I'd add that their chief engineer built the Allegiance network and has done a lot of other work for ISPs, it's not as if there are idiots running the show over there....
Posted by Mix, 03-31-2005, 07:06 AM
This is rediculous
Posted by bidder, 03-31-2005, 07:06 AM
does anyone have any updates from theplanet?
Posted by Lem0nHead, 03-31-2005, 07:07 AM
looks like they don't wanna tell
Posted by EricP, 03-31-2005, 07:07 AM
Yes, they are working on it
Posted by whmcsguru, 03-31-2005, 07:07 AM
I hate to say it , but this is incorrect. This downtime doesn't show that, but numerous experiences DO. It's possible, entirely for a datacenter to be out for an hour and a half, especially if fiber is affected, which the provider can do absolutely nothing about. This isn't something the frontline techs would know squat about. You're lucky if your frontline tech knows anything at all about what your problem is with linux, much less networking, routing, etc. You won't get to orbit.theplanet, because at the current time, the dns for orbit.theplanet is nonexistant. NS1 and NS2 . theplanet are screwed up royally right now. Checking the traces, the problem APPEARS to be with verio in texas, at least coming from NAC's NJ datacenter. For the rest of the world? It's a routing problem, which, unfortunately is what they're good at, and means it'll be a while before something gets fixed. Sit back, enjoy your coffee guys, cuz this one's going to take 'em a while to straighten out.
Posted by eniki, 03-31-2005, 07:07 AM
Very good. Lots of people are simliar positions to you dolay. Get over yourself.
Posted by Cephren, 03-31-2005, 07:07 AM
we are about to hit one hour....... Congradulations everyone...... there goes your 99.999% uptime. now its 99.9%
Posted by chrisbond, 03-31-2005, 07:07 AM
From Jose @ TMS: At the moment (3/31/2005 - 05:00am CST) there is a network issue affecting the datacenter entirely. As a result, all of our servers are offline. Engineers are already working to resolve this as soon as possible.
Posted by Hostmax-br, 03-31-2005, 07:08 AM
yeah duds, down all the time... 8 am here (brazil) hope it get back soon... starting to loss my will...
Posted by bjdea1, 03-31-2005, 07:08 AM
hmmm. I'm just looking for info on whats going on. Seems no way to find out anything about this.
Posted by galantina, 03-31-2005, 07:08 AM
Ha anyone got some info from them?
Posted by Lem0nHead, 03-31-2005, 07:09 AM
there's a more than 10 pages thread here: http://www.webhostingtalk.com/showth...postid=2981670 but it isn't very useful to read it just people complaining
Posted by bigd, 03-31-2005, 07:09 AM
The 69.93 class B seems to be down.
Posted by phenx, 03-31-2005, 07:09 AM
We only just updated our DNS and put a gameserver up on a new top of the line $900 Total Control server, and it's now been down for 1 hour. I havent been able to get through via Phone. Our server is in DC5
Posted by JDMundo, 03-31-2005, 07:10 AM
linux-tech.. Argh! You're giving me an ulcer. .. I hope you're wrong, and it'll be back up in no time.
Posted by Hostasaurus, 03-31-2005, 07:10 AM
The problem isn't with their internet providers, they're advertising their routes and the advertisements have been stable, meaning theplanet's border routers are not affected by this.
Posted by NetSender, 03-31-2005, 07:10 AM
I rang up to some guy just knowing that there was a problem, they were aware of it, and they're still working on it. Although I could barely hear him for the millions of phones ringing in the background, hehe
Posted by Vatsis, 03-31-2005, 07:11 AM
THIS IS MY WORST NIGHTMARE. I've ordered 2 new boxes AND 4 OLD SERVERS WILL GET CANCELLED TONIGHT! I've hired two admins to move everything from the old servers to the new ones (with rate 500$/hour), and now they just sit here eating popcorn. Even if thePlanet gets the servers up, there's soon no time for the upgrade... I might well wake up tomorrow seeing all my 4 servers, data gone.
Posted by TotalChoice - Bill, 03-31-2005, 07:11 AM
DNS servers can't work if they can not talk to a workable network. They appear offline, not screwed up.
Posted by Cephren, 03-31-2005, 07:11 AM
yay my shift is over! but yay I guess I can't leave and go to bed.
Posted by coolio, 03-31-2005, 07:12 AM
question for those that have servers that came back up. did the server reboot (power outage) or not (other problem - network) ?
Posted by Jz1977, 03-31-2005, 07:12 AM
here is a quick trace route from my end 01 192.168.123.253 - 500 ms 02 162.40.125.7 cis235.navix.net 32 ms 03 162.40.125.3 h3.125.40.162.ip.alltel.net 47 ms 04 151.213.33.137 h137.33.213.151.ip.alltel.net 47 ms 05 12.125.72.53 - 47 ms 06 12.122.2.213 gar1-p360.kc9mo.ip.att.net 750 ms 07 12.122.2.210 tbr2-p012902.sl9mo.ip.att.net 78 ms 08 12.122.10.90 tbr2-cl6.dlstx.ip.att.net 62 ms 09 12.123.196.97 gar1-p370.dlrtx.ip.att.net 62 ms 10 12.119.136.14 - 63 ms 11 ... ... .. ms 12 ... ... .. ms 13 ... ... .. ms 14 ... ... .. ms 15 ... ... .. ms
Posted by Tommeh, 03-31-2005, 07:12 AM
I've been with Servermatrix now for quiet a while and never really had any probs, this problem is affecting my server because of the DNS problems. Just annoying that most server that are still up are in SM while most are down in TP with the 100% uptime?
Posted by WebAfrica, 03-31-2005, 07:12 AM
Actually my calculations put 99.9% uptime as max 43 minutes downtime per month.. So looks like they on 99.8 heading for 99.7
Posted by chevelleDOD, 03-31-2005, 07:12 AM
this outage has me wondering what the mrtg loads like at the IX's around D-Town/H-Town/Austin since TP is down. I imagine that they account for a noticeable amount of overall traffic coming in and out of the Southwest...
Posted by suhailc, 03-31-2005, 07:12 AM
Are these servers at their new DC?
Posted by alexrobonlin, 03-31-2005, 07:13 AM
58 mins down now... Went down at 11:15am here, is now 12:13pm (UK) Al
Posted by TopHostSupport, 03-31-2005, 07:13 AM
Wow, you should have shopped around. Great techs on this board don't even charge that much.
Posted by seahorse, 03-31-2005, 07:13 AM
initially i thought it was my server. well, now i cant even access orbit. maybe they should have they site hosted out of their datacentre ... alright, enough of my crap. (waiting for my server to be up) Anyway, will there be a compensation. i guess it affects too many people and server...
Posted by Lem0nHead, 03-31-2005, 07:13 AM
$500/hour?
Posted by trustedurl.com, 03-31-2005, 07:14 AM
69 minutes and counting.
Posted by ServerZB, 03-31-2005, 07:14 AM
all down here
Posted by BizB, 03-31-2005, 07:14 AM
my TC servers are ok looks like parts of there network is down
Posted by suhailc, 03-31-2005, 07:15 AM
Are these servers at their new DC?
Posted by HenryJ, 03-31-2005, 07:15 AM
my servers with ips starting with 67.**.**.** are working fine. Servers with Ips starting from 69.*.*.* and 70.*.*.* are down.
Posted by tamouh, 03-31-2005, 07:15 AM
This post growing by the 30 second. by the way, it appears all AA official servers are down as well. Coincidence or maybe they're using ThePlanet ?
Posted by Lem0nHead, 03-31-2005, 07:15 AM
that's worse i wish it was just one single cable they needed to plug
Posted by Cephren, 03-31-2005, 07:15 AM
wow someone pay me $500/hr.... i need the money.
Posted by bidder, 03-31-2005, 07:16 AM
all mail that is sent when the server is down - will it still be received when the server comes back up??
Posted by EricP, 03-31-2005, 07:16 AM
I wasn't going to file an SLA. I think I will now. 33 servers. Most are down.
Posted by ServerZB, 03-31-2005, 07:16 AM
maybe it is a good time to see how they honor their SLA
Posted by sam2579, 03-31-2005, 07:17 AM
Planet is making big headlines now a days, first all iranian server were gone, now all servers down including their own sites, hahahah.
Posted by ampunboss, 03-31-2005, 07:17 AM
its been more than 1 hours + 30 mins my servers are down, but my other servers there are up. I cant access orbit & servermatrix.
Posted by trustedurl.com, 03-31-2005, 07:17 AM
yep, with a delay, but it should be ok.
Posted by Mix, 03-31-2005, 07:17 AM
Tomorrow is billing day too
Posted by inteltechs, 03-31-2005, 07:18 AM
they will honor their SLA though.
Posted by tamouh, 03-31-2005, 07:18 AM
Iranian servers ? were they flagged as terrorists too ?
Posted by MatrixAu, 03-31-2005, 07:18 AM
Strange, I have 3 servers, 2 are down one is up. One's in DLLTX2 (down) one's in DLLTX4 (up) one's in DLLTX5 (down). Lol, And the DLLTX5 one is a ThePlanet Total Control server, just a few days ago, I asked what are the differances compared to servermatrix, one of them he mentioned was "100% network uptime" :p. - Shane.
Posted by Lem0nHead, 03-31-2005, 07:18 AM
they should be put on queue by the server sending that and will be tried again like 4 hours later
Posted by PierreB, 03-31-2005, 07:18 AM
Our 3 out of 6 servers have been down since 4:50am EST so easily, 1hour30minutes down for us.
Posted by Hostasaurus, 03-31-2005, 07:18 AM
In this case, possibly not. If you have DNS that is up and telling the sending mail server what server to deliver your mail to, and that destination server is not available, the sending server will queue it up potentially for up to five days and will keep trying. However, if your DNS servers are the two TP name servers that are now offline, when someone sends a message the sending mail server won't even be able to do a DNS lookup and may bounce the message immediately. That's why it's critical to have geographically redundant DNS; even if everything is offline, that one remote DNS server can keep the email from bouncing.
Posted by BrentOfHG, 03-31-2005, 07:18 AM
Does anyone have any official news yet? I would suspect some type of power problem, only because we had two servers reboot magically at the same time earlier.
Posted by chevelleDOD, 03-31-2005, 07:18 AM
my 67.* box is down
Posted by BlackMail, 03-31-2005, 07:19 AM
I'm with the planet since over a year (servers are down too here) so what will happen anyway if it comes to their 99,9% uptime promise? I would not care normally but I run a leading news site and 1 hour and more means refunding top 500 companies who put a press release in for today (that's around 100 x 25 $ or more losses as we lose Google Ad income too) - just great...
Posted by ServerZB, 03-31-2005, 07:19 AM
dh , are you guys at theplanete now?
Posted by paulstest, 03-31-2005, 07:19 AM
What I would like to see is an official statement from ThePlanet / ServerMatrix... I don't entirely know where they'd post it tho ;-) Paul @ MoDaCo.com
Posted by eniki, 03-31-2005, 07:19 AM
Does anyone have copies of both the TP and SM SLA agreements so that we can help people understand whether they are covered or not.
Posted by bjdea1, 03-31-2005, 07:19 AM
No I think the new Datacenter is ok - it seems to be their original (first) Datacenter.
Posted by woolly, 03-31-2005, 07:19 AM
The Planet totally sucks. It isn't the first time something like this happened either. Their 100% uptime guarantee is a joke. They refuse to compensate for downtime when requested in the orbit.
Posted by Erwin, 03-31-2005, 07:19 AM
I have servers down too...
Posted by Cephren, 03-31-2005, 07:20 AM
no news. They still havent figured it out.
Posted by bheka, 03-31-2005, 07:20 AM
Nope .. they aren't saying a word
Posted by whmcsguru, 03-31-2005, 07:20 AM
Actually, I hate to say it, but at least PART of the problem IS with verio in texas. That's where it's dying from New Jersey at least. Don't trust me? Trust a traceroute. As soon as it should be leaving verio, it's dying. Granted, from a few other spots, it IS hitting the borders, but from NJ it isn't even hitting the borders, it's dying right before them apparently. Agreed. Moving your servers will take at minimum 3 hours, getting data from network a to network B, and that's assuming they're SMALL servers and CPanel can be used . Sounds like you got horribly ripped there, and that'll cost you MORE than this downtime would have So do I, for your sake, but experience shows that I'm not, and the current responses of "We have no clue what's going on" show absolutely no reason for that hope
Posted by trustedurl.com, 03-31-2005, 07:21 AM
nah, we have 1 windows box at the planet... everything else is in our racks @ GC. That's why I'm not too worried, but it's time for us to move the windows accounts I'd say.
Posted by cbtrussell, 03-31-2005, 07:21 AM
Back up for me.
Posted by moheybee, 03-31-2005, 07:21 AM
My 70.84.*.* TC servers are back up!
Posted by neutro, 03-31-2005, 07:21 AM
back! now for me
Posted by Hostasaurus, 03-31-2005, 07:21 AM
If you lose $2500/hour when your host has a problem, I'd recommend going with a much more redundant setup such as two hosts (or one that can manage servers in two data centers for you), synchronized content and a third party DNS service that can do the automated/immediate failover. That wouldn't cost more than what you'd lose in an hour due to downtime so it should be money well spent to have that level of insurance against outages.
Posted by phenx, 03-31-2005, 07:21 AM
My server is back online now
Posted by richenou, 03-31-2005, 07:21 AM
Yes i have : 20 servers down : and 80 other is good depends of the network affected network in my box : 69.93.... 69.18... 69.56.. 69.41..
Posted by TotalChoice - Bill, 03-31-2005, 07:22 AM
One of ours just came back up.
Posted by Cephren, 03-31-2005, 07:22 AM
100% Network Guarantee: The Planet offers a 100% uptime guarantee on all network related services (i.e. bandwidth, routers, switches, and cabling,). To qualify for the 100% uptime guarantee, customer equipment must utilize redundant input network drops running Hot Swap Router Protocol (HSRP). Customer generated outages created by failed equipment, customer misconfigurations, exploited servers, or traffic in excess of the maximum allowed by contract are not subject to the SLA. Service credits are based directly on all equipment and/or services affected by a network outage. Services or hardware not related to the network outage do not qualify for a service credit. From Master SLA of The Planet. 99.90% Network Guarantee: The Planet offers a 99.90% uptime guarantee on all network related services (i.e. bandwidth, routers, switches, and cabling,). The 99.90% uptime guarantee is applicable to all network services delivered to customers via a single independent network drop. Customer generated outages created by failed equipment, exploited servers, misconfigurations or traffic in excess of the maximum allowed by contract are not subject to the SLA. Service credits are based directly on all equipment and/or services affected by a network outage. Services or hardware not related to the network outage will not qualify for a service credit.
Posted by foxgroup, 03-31-2005, 07:22 AM
How long does it have to be down for me to get a refund for this months hosting with the planet? Like everyone here this is costing my company a lot of money!
Posted by tamouh, 03-31-2005, 07:22 AM
I'm back up now 69.*
Posted by EricP, 03-31-2005, 07:22 AM
My main server just came back online
Posted by Fahd, 03-31-2005, 07:22 AM
My server just came back up!
Posted by ultrafex, 03-31-2005, 07:22 AM
Server is back. And you were right, routing goes through DC3 to DC2, hmmm, that's a big change...Still internally routed, still bad. Last edited by ultrafex; 03-31-2005 at 07:28 AM.
Posted by renski, 03-31-2005, 07:22 AM
**** hit the fan there for a while back now though..
Posted by suhailc, 03-31-2005, 07:23 AM
20 servers? 33 servers? You guys should b uild your own DCs!
Posted by woxcom, 03-31-2005, 07:23 AM
we're back up
Posted by alexdio, 03-31-2005, 07:23 AM
Wow! My sites are online now!
Posted by jedrek, 03-31-2005, 07:23 AM
my 69.18.* is up, the 67.56.* is still down.
Posted by HighLineHost, 03-31-2005, 07:23 AM
You dont see this too often, all of theplanet's sites are down. Intresting... Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp. C:\Documents and Settings\Greg Agri>cd\ C:\>tracert theplanet.com Tracing route to 212.67-18-142.reverse.theplanet.com [] over a maximum of 30 hops: 1 1 ms <1 ms <1 ms 192.168.102.1 2 15 ms 20 ms 28 ms 10.184.128.1 3 10 ms 7 ms 8 ms 12.244.16.145 4 11 ms 7 ms 10 ms 12.244.69.153 5 14 ms 14 ms 15 ms 12.127.32.85 6 17 ms 15 ms 17 ms tbr2-p013701.sffca.ip.att.net [12.123.13.177] 7 26 ms 23 ms 27 ms tbr1-cl3.la2ca.ip.att.net [12.122.10.26] 8 77 ms 55 ms 55 ms tbr1-cl2.dlstx.ip.att.net [12.122.10.49] 9 55 ms 52 ms 59 ms gar1-p360.dlrtx.ip.att.net [12.123.196.101] 10 60 ms 55 ms 55 ms 12.119.136.14 11 54 ms 63 ms 56 ms 70.85.127.29 12 * * * Request timed out. 13 * * * Request timed out. 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 * * * Request timed out. 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 Last edited by HighLineHost; 03-31-2005 at 07:29 AM.
Posted by TotalChoice - Bill, 03-31-2005, 07:23 AM
05:22:26 up 57 days is the server that just came back up.
Posted by COH_Henrik, 03-31-2005, 07:23 AM
I have still one server that is down with the Planet...
Posted by andrewgmol, 03-31-2005, 07:24 AM
Still down for me.
Posted by Lem0nHead, 03-31-2005, 07:24 AM
doesn't this traceroute shows the problem is AFTER verio texas?
Posted by RI-VVM, 03-31-2005, 07:24 AM
69.56.* is still down
Posted by Laws, 03-31-2005, 07:24 AM
You'll get a partial % credit but not a full refund.
Posted by inteltechs, 03-31-2005, 07:24 AM
our servers are all up.
Posted by AlexAT, 03-31-2005, 07:24 AM
>> They refuse to compensate for downtime when requested in the orbit. It is not true.
Posted by GoLions, 03-31-2005, 07:25 AM
Just spoke to them on the phone.... they said it looks like they've isolated the problem, and are now in the process of fixing it. The tech said he could see green lights coming back on.... which is a good thing. No ETA, but he figured it wouldn't be long. As I write this, all my servers have come back to life
Posted by MatrixAu, 03-31-2005, 07:25 AM
Back up here.
Posted by Bill-zilonhost, 03-31-2005, 07:25 AM
They're back thank God.
Posted by Iggy, 03-31-2005, 07:25 AM
69.93.* up 64.5.* down
Posted by JDMundo, 03-31-2005, 07:25 AM
My 69.56 box is up.
Posted by dolay, 03-31-2005, 07:25 AM
when i mentioned that more politely someone said me IDIOT and some of them get angry to me... I am guessing your situation now how you will recieve alot of replies... The King is Naked,
Posted by Arvand, 03-31-2005, 07:25 AM
All 7 down....
Posted by c0bra, 03-31-2005, 07:25 AM
It's a 20% refund claim right for this outage that lasted just over an hour? Any mathmaticians on here?
Posted by BlackMail, 03-31-2005, 07:25 AM
Yep that's what I am going to do now. Don't get me wrong I really love their service and I really don't care much to get any money from them - I just believe that a downtime like this should not happen - or what I at least expect is somebody like "YourHost" posting a note to a forum like this one here - even if it is only a "we don't know yet"
Posted by trustedurl.com, 03-31-2005, 07:25 AM
back up 69.56.x.x. server was not rebooted it seems. 1 hour 23 minutes of downtime (at least).
Posted by Gually, 03-31-2005, 07:25 AM
69.93.* still down..
Posted by Lem0nHead, 03-31-2005, 07:25 AM
69.56.* is down for me yet
Posted by Hostasaurus, 03-31-2005, 07:26 AM
Traceroutes typically show the hop IP as the outbound interface from the router the packet is passing through. So the trace you posted showing the last hop as being Verio doesn't necessarily mean there is anything wrong with their border router. What it does possibly mean is that the link from that specific border router to their internal network is down. If there were a problem with that router, they would be unable to advertise their routes to Verio in BGP, which they are currently doing, so that border router should be okay.
Posted by v0ltair3, 03-31-2005, 07:26 AM
I'm at 69.93.228.* and I'm still down
Posted by NetSender, 03-31-2005, 07:26 AM
3 Boxes still down
Posted by ImLagging, 03-31-2005, 07:26 AM
Both of my DLLSTX2 servers are now up and I checked on one and it wasn't rebooted or lost power.
Posted by ServerZB, 03-31-2005, 07:26 AM
I buy servers from TMS, I don;t want to see that TMS will push responsibility to planete and plnate will push it to verio/att....
Posted by Cephren, 03-31-2005, 07:27 AM
ummm just wondering how many of you guys are COLO clients.... our colo servers are still down. dedicated ones up though.
Posted by rsilviu2005, 03-31-2005, 07:27 AM
One of my servers is up again.
Posted by bidder, 03-31-2005, 07:27 AM
Id like to know the answer to that too.. My server is still down...
Posted by Mix, 03-31-2005, 07:27 AM
Still down
Posted by Laws, 03-31-2005, 07:27 AM
their internal staff network went down for a time too..
Posted by Gually, 03-31-2005, 07:27 AM
69.93.146.* stil down
Posted by fashion, 03-31-2005, 07:27 AM
one of mine just came back up... the rest are still flashing red...
Posted by dkitchen, 03-31-2005, 07:28 AM
Their network is back it appears, probably just waiting for some BGP routes to propogate if their border routers were down. Dan
Posted by web_host_guy_007, 03-31-2005, 07:28 AM
Almost all the servers at planet are down. There can be a possibility of a power failure.
Posted by ankita, 03-31-2005, 07:28 AM
The servers are up.
Posted by chrisbond, 03-31-2005, 07:28 AM
216.66.19.* down 216.66.18.* down
Posted by andrewgmol, 03-31-2005, 07:28 AM
I'm on 69.93.x.x too. Down still.
Posted by djdiz-e, 03-31-2005, 07:28 AM
i just moved from velocity.net ..99% uptime blah more like 99% downtime lol im haven alot of trouble the last 2 months
Posted by Bill-zilonhost, 03-31-2005, 07:28 AM
My server is up but the planet website is still down.
Posted by NetSender, 03-31-2005, 07:28 AM
Also still cant get to http://www.theplanet.com
Posted by Maxo, 03-31-2005, 07:28 AM
Are you their customer? Did anyone try to call them up?
Posted by bidder, 03-31-2005, 07:29 AM
http://www.theplanet.com << still down
Posted by Hostasaurus, 03-31-2005, 07:29 AM
They haven't had any BGP flaps on at least Level 3 and Global Crossing, I think the problem was internal.
Posted by smartins, 03-31-2005, 07:29 AM
My 69.56 box just went up.
Posted by dee_at_candl, 03-31-2005, 07:29 AM
Up here .. thank God!
Posted by tamouh, 03-31-2005, 07:29 AM
colo is up for us dedicated is up 70.85.* 69.56.*
Posted by Tommeh, 03-31-2005, 07:30 AM
Wow this is good, my DNS is all working again fine, but theplanet's site is still down They put the customers first !
Posted by navmonkey, 03-31-2005, 07:30 AM
Anyone here using the Total Balance package? Are their load balancers reachable? Thank you.
Posted by HighLineHost, 03-31-2005, 07:30 AM
My Friend has his shoutcast server over there..
Posted by alexrobonlin, 03-31-2005, 07:30 AM
Been down for 1hour 15 mins for me so far (12:30pm here - uk) What levels of comp. are we talking? Al
Posted by Bill-zilonhost, 03-31-2005, 07:30 AM
Is anone still down? (besides theplanet)
Posted by bidder, 03-31-2005, 07:31 AM
yes im down.......STILL
Posted by dxer, 03-31-2005, 07:31 AM
69.93.186.xxx still down
Posted by donovanh, 03-31-2005, 07:31 AM
Still down here: 70.85.*
Posted by Mix, 03-31-2005, 07:31 AM
deuce.made2own.com is still down
Posted by Lem0nHead, 03-31-2005, 07:31 AM
my 69.56 is still down
Posted by simscripts, 03-31-2005, 07:31 AM
69.93.118.* down
Posted by RI-VVM, 03-31-2005, 07:31 AM
69.56.132.* is down yet 13 ffm-b3-pos1-1-0.telia.net (213.248.64.66) 104.193 ms 86.624 ms 86.057 ms 14 global-01985-ffm-b3.telia.net (213.248.77.182) 90.032 ms 127.163 ms 96.039 ms 15 so0-0-0-2488m.ar1.dal2.gblx.net (67.17.73.30) 213.984 ms 211.652 ms 213.654 ms 16 The-Planet.ge-0-2-1.ar1.DAL2.gblx.net (64.213.176.150) 222.038 ms 222.422 ms 223.049 ms 17 dist-vlan31.dsr3-2.dllstx3.theplanet.com (70.85.127.30) 219.329 ms dist-vlan32.dsr3-1.dllstx3.theplanet.com (70.85.127.61) 223.608 ms dist-vlan31.dsr3-2.dllstx3.theplanet.com (70.85.127.30) 222.873 ms 18 dist-vlan21.dsr1-1.dllstx2.theplanet.com (70.85.127.67) 226.286 ms dist-vlan22.dsr1-2.dllstx2.theplanet.com (70.85.127.76) 226.892 ms dist-vlan21.dsr1-1.dllstx2.theplanet.com (70.85.127.67) 227.060 ms 19 car2-4-v1.dllstx2.theplanet.com (12.96.160.20) 237.683 ms 227.484 ms 235.437 ms 20 * * *
Posted by zsindhu, 03-31-2005, 07:31 AM
That's good news. Our's are still down, though
Posted by jedrek, 03-31-2005, 07:31 AM
still down on 67.18.*
Posted by ffeingol, 03-31-2005, 07:31 AM
we've had 1 server come back, 3 still down
Posted by Gually, 03-31-2005, 07:31 AM
Bill-zilonhost, i'm still down.. 69.93.146.x
Posted by gpattabhi, 03-31-2005, 07:31 AM
Up 69.93.*
Posted by richenou, 03-31-2005, 07:32 AM
yes not all the network is repared !
Posted by BrentOfHG, 03-31-2005, 07:32 AM
I was told by someone in management that he was told a couple cars on a few routers failed. (blade servers) This is what he was told but may not be accurate.
Posted by bareck, 03-31-2005, 07:32 AM
67.19.* up
Posted by Bill-zilonhost, 03-31-2005, 07:33 AM
I am still up. theplanet.com still down.
Posted by alexrobonlin, 03-31-2005, 07:33 AM
I'm still down, yes
Posted by Przemek, 03-31-2005, 07:33 AM
my 69.93.170.* is now up.
Posted by BONNER, 03-31-2005, 07:33 AM
Yep I did. They say the problem is only with their DNS. I can confirm it as my system is accessible by ip address, but not by name. Not funny anyway.
Posted by kamurj, 03-31-2005, 07:34 AM
i am a servermatrix customer, called them 5 min ago and was told that there is a network issue and that they are working on it.
Posted by Iggy, 03-31-2005, 07:34 AM
My colos are still down at the Planet. Leased at ServerMatrix are up. Go figure.
Posted by TotalChoice - Bill, 03-31-2005, 07:34 AM
Our 69.93.215.* Are all still down.
Posted by Lem0nHead, 03-31-2005, 07:35 AM
those servers that are back weren't they rebooted?
Posted by djdiz-e, 03-31-2005, 07:36 AM
70.85.68.* is still down
Posted by andrewgmol, 03-31-2005, 07:36 AM
69.93.178.x still down
Posted by navmonkey, 03-31-2005, 07:36 AM
I thought the Internet is nuke-proof.
Posted by forumtalk, 03-31-2005, 07:36 AM
This is going very bad
Posted by TotalChoice - Bill, 03-31-2005, 07:36 AM
No
Posted by jedrek, 03-31-2005, 07:36 AM
no reboot over here.
Posted by TopHostSupport, 03-31-2005, 07:36 AM
Nope. 2 up 1 still down.
Posted by alexrobonlin, 03-31-2005, 07:36 AM
Just spoken to them again - they said they're waiting for a report. I mentioned that people here are saying that their servers are up, and he's saying that there are intermittant bursts of connectivity, but he didn't say if anything was staying solidly up... He said a report will go into their forums asap, as soon as it's all up. I didn't mention compensation, but I got the feeling he was expecting me to ask... Al
Posted by coolio, 03-31-2005, 07:36 AM
2 of 3 back up. maybe something to know ... their 20mbit-maxtreme-servers were also down (atleast mine - back up now) ... which are on a separated network (with only cogent gigbit lines) ... kinda weird, but should mean it's certainly no verio problem.
Posted by HighLineHost, 03-31-2005, 07:37 AM
Well, anyways, its not too often when you see something like this happen to one of the largest datacenters
Posted by ankita, 03-31-2005, 07:37 AM
My 70.85.* is up donovanh. I am sure yours would be up soon.
Posted by alexdio, 03-31-2005, 07:37 AM
I wonder, how many servers are on ThePlanet?
Posted by wheimeng, 03-31-2005, 07:37 AM
Looks good
Posted by donovanh, 03-31-2005, 07:37 AM
Thanks ankita, fingers crossed.
Posted by chevelleDOD, 03-31-2005, 07:37 AM
ATT flapped routes on me right when this started I'm pretty sure it was just the nature of the beast cause the last i saw was a route loop on gar.att. and then silence
Posted by AKWS, 03-31-2005, 07:37 AM
Noticed gpattabhi said 69.93.* was up yet 69.93.160.* still down
Posted by ImLagging, 03-31-2005, 07:37 AM
I only checked one of the 2 that were down, and no, it was not rebooted.
Posted by misnomer, 03-31-2005, 07:38 AM
One of my servers is still down, and the internal RHN proxy servers and DNS servers are still down as well.
Posted by ankita, 03-31-2005, 07:38 AM
alexdio.. cant you make out.. look at the number of postings made in last one hour.
Posted by EricP, 03-31-2005, 07:39 AM
70.85.20.* is down for us
Posted by Gually, 03-31-2005, 07:39 AM
69.93.146.x still down. near the 1:30. somebody from Catalonia?
Posted by Laws, 03-31-2005, 07:39 AM
Not quite true... If they need to they can switch you over to premium bandwidth straight away so it's not completely separate to the main network...
Posted by inteltechs, 03-31-2005, 07:39 AM
http://servermatrix.com http://theplanet.com 67.18.101.* are still down
Posted by alexrobonlin, 03-31-2005, 07:40 AM
69.93* still down too... *
Posted by zsindhu, 03-31-2005, 07:41 AM
Why would they need to reboot them? The problem is with the network, not the servers.
Posted by donovanh, 03-31-2005, 07:41 AM
70.85.56.* back up
Posted by hosttone, 03-31-2005, 07:41 AM
theplanet - down. servermatirx - down.
Posted by lessthanthree, 03-31-2005, 07:41 AM
my 67.19. is up, 67.18. is down, somone i knows 69.93. is up
Posted by Symen_4ab, 03-31-2005, 07:41 AM
69.93.36.x server back online for me !
Posted by whmcsguru, 03-31-2005, 07:42 AM
Actually, it shows that the last hop IS verio, but that's it. It's entirely possible that the routes affected were between verio and TP, or that somewhere along the way (one of the routers there) was what's causing everything to go to crap. It's entirely possible they're blocking traceroutes to this ip, which would be the cause of the delay, but from every other point I tried the trace, the network was at least reached. This is incorrect Mail will still be attempted for 5 days. The only time it will NOT be attempted for 5 days is if the mail is explicitly rejected by the receiving server. If, say, the server is down, or can't be located, then it'll be filed and queued out later. No, not necessarily. Given the prices of most companies, it's not profitable to build a dc until you have a minimum of 1000 servers (and that's a low guess). Think of it this way, seriously. You pay 150-200 for those 33, 20 servers . That averages about 3-5000 per month. Admittedly, yes it's a good chunk of change, but the profit margin on those is maybe (maybe) $50/server. Now, you take those and put them into a datacenter, and here's what you've got to worry about (literally) -- Power -- IP Addresses -- Multiple routes to multiple providers -- Employees -- Security -- Building the servers Initially, your datacenter investment is going to be (minimally) $100k, and it'll cost (again, minimally) 10k/month. Hence, it's not really beneficial UNTIL you get close to that 1000 server range to do this.
Posted by alexrobonlin, 03-31-2005, 07:42 AM
They've got a good chunk of the net at the planet, but I doubt they'll have quite as big-a chunk after this. Shame, as their Telementary newsletter came out this morning too!! Been an hour and a half now Alex
Posted by Lem0nHead, 03-31-2005, 07:42 AM
now that's more than 88 minutes, or 0.2% down
Posted by ellinas, 03-31-2005, 07:43 AM
69.93. Down 70.84 Up ( Had no problems at all )
Posted by The Napster, 03-31-2005, 07:43 AM
http://www.webhostingtalk.com/showth...hreadid=390336 All there servers are down i believe
Posted by zsindhu, 03-31-2005, 07:43 AM
Two of ours are still down and it's been over an hour now I do hope this gets resolved for us, soon.
Posted by ALGORYTHM, 03-31-2005, 07:43 AM
67.18.** down great way to celebrate my one year anniversery Tracing route to 67.18.** over a maximum of 30 hops 1 16 ms 19 ms 17 ms loopback1.ar1.gs1.systems.pipex.net [62.241.161.247] 2 16 ms 17 ms 19 ms ge-1-0-0.cr1.gs1.systems.pipex.net [62.241.161.94] 3 14 ms 21 ms 15 ms 999.ge6-2.mpr1.lhr1.uk.above.net [217.79.160.12] 4 16 ms 17 ms 17 ms so-4-1-0.cr1.lhr3.uk.above.net [208.184.231.174] 5 88 ms 91 ms 89 ms so-7-0-0.cr1.dca2.us.above.net [64.125.31.186] 6 88 ms 89 ms 89 ms so-0-0-0.cr2.dca2.us.above.net [64.125.29.122] 7 113 ms 116 ms 116 ms so-2-3-0.cr2.dfw2.us.above.net [64.125.29.13] 8 122 ms 126 ms 126 ms 216.200.6.237.theplanet.com [216.200.6.237] 9 123 ms 122 ms 120 ms 70.85.127.30 10 * * * Request timed out. 11 * * * Request timed out.
Posted by bheka, 03-31-2005, 07:44 AM
I still can't get to 67.19
Posted by ePlanetDesign, 03-31-2005, 07:45 AM
70.84 Down
Posted by Lem0nHead, 03-31-2005, 07:45 AM
that was just to confirm if it wasn't a power problem
Posted by devioustrap, 03-31-2005, 07:45 AM
67.18.35.* is still down. This going over an hour and a half of downtime.
Posted by Cephren, 03-31-2005, 07:45 AM
im logging more than 88 mins for sure............ for the servers that came backup....77 mins downtime... confirmed with logs by activity of couple sites. still got server down right now...
Posted by HighLineHost, 03-31-2005, 07:46 AM
Ah didnt see that post sorry for the extra post on their mayjor problem
Posted by ankita, 03-31-2005, 07:46 AM
I know that's difficult lem0nhead. But i am sure they are fixing it up and soon your servers would be up too.
Posted by Tommeh, 03-31-2005, 07:46 AM
Mines 67.18.* and it's been up all this time... no network outages at all...
Posted by chatcircuit, 03-31-2005, 07:47 AM
ArteryPlanet.Net All client servers still down.
Posted by bugsoft, 03-31-2005, 07:47 AM
Just called again. "We think we have isolated the problem and the network will be up within an hour"
Posted by hosttone, 03-31-2005, 07:47 AM
We have 3 server with theplanet, 2 up and 1 down.
Posted by ampunboss, 03-31-2005, 07:47 AM
www.servermatrix.com & TP should backup their secondary DNS at EV1
Posted by Lem0nHead, 03-31-2005, 07:47 AM
here's the ServerMatrix SLA: so we should get the 20% compensation
Posted by cem, 03-31-2005, 07:47 AM
Can someone nearby go there and take a look whats going on?
Posted by Iggy, 03-31-2005, 07:47 AM
You bastard!
Posted by ffeingol, 03-31-2005, 07:48 AM
I just called again and level 1 tech's still have no idea what the problem is. I'm not going to try and tell them how to run their business, but can't one level one walk over, find out what the problem/eta is and relay that to all the other level ones? Frank
Posted by The Napster, 03-31-2005, 07:48 AM
hehe well most of the people i know hosted at the planet have all gone crazy, would be intresting to see how long there down for!
Posted by Hostasaurus, 03-31-2005, 07:48 AM
AT&T's route server at route-server.ip.att.net doesn't show any flaps for TP's 69.56.128.0/17 prefix. They may have different border routers to AT&T advertising different prefixes though.
Posted by woolly, 03-31-2005, 07:49 AM
69.93 is still down for me.
Posted by zsindhu, 03-31-2005, 07:49 AM
ah, ok
Posted by kneadingu, 03-31-2005, 07:49 AM
100% correct.
Posted by chevelleDOD, 03-31-2005, 07:49 AM
:bs:
Posted by MatrixAu, 03-31-2005, 07:50 AM
All mine are back up.. . My ThePlanet Total Control server in dllstx5: Tracing route to 70.84.217.98 with TTL of 32: 1 23ms 21ms 19ms 10.128.68.1 2 31ms 23ms 13ms 61.9.207.131 3 12ms 27ms 19ms 61.9.193.3 4 34ms 35ms 11ms GigabitEthernet3-5.pic2.Sydney.telstra.net [139.130.0.4 9] 5 8ms 20ms 16ms GigabitEthernet10-2.ken-core4.Sydney.telstra.net [203.5 0.12.126] 6 14ms 22ms 21ms 10GigabitEthernet3-0.pad-core4.Sydney.telstra.net [203. 50.6.86] 7 34ms 24ms 17ms GigabitEthernet2-2.syd-core01.Sydney.net.reach.com [203 .50.13.38] 8 167ms 186ms 172ms i-6-1.wil-core02.net.reach.com [202.84.249.201] 9 192ms 225ms 202ms i-2-0.dal-core01.net.reach.com [202.84.143.66] 10 226ms 202ms 203ms level3.Dal-core01.net.reach.com [134.159.63.142] 11 211ms 200ms 200ms ge-1-1-55.car3.Dallas1.Level3.net [4.68.122.134] 12 210ms 205ms 200ms 4.78.220.10 13 195ms 210ms 198ms 70.85.127.29 14 213ms 212ms 212ms 70.85.127.106 15 230ms 192ms 195ms po2.tp-car3.dllstx5.theplanet.com [70.84.160.165] 16 201ms 233ms 208ms 98.70-84-217.reverse.theplanet.com [70.84.217.98] Traceroute complete. My ServerMatrix server in dllstx2: Tracing route to 69.93.231.18 with TTL of 32: 1 11ms 13ms 8ms 10.128.68.1 2 10ms 11ms 9ms 61.9.207.131 3 40ms 28ms 12ms 61.9.193.3 4 11ms 11ms 11ms GigabitEthernet3-5.pic2.Sydney.telstra.net [139.130.0.4 9] 5 9ms 13ms 11ms GigabitEthernet11-1.ken-core4.Sydney.telstra.net [203.5 0.12.117] 6 22ms 10ms 14ms 10GigabitEthernet3-0.pad-core4.Sydney.telstra.net [203. 50.6.86] 7 29ms 11ms 8ms GigabitEthernet2-2.syd-core01.Sydney.net.reach.com [203 .50.13.38] 8 165ms 179ms 179ms i-12-3.wil-core02.net.reach.com [202.84.249.205] 9 233ms 190ms 207ms i-2-0.dal-core01.net.reach.com [202.84.143.66] 10 198ms 197ms 222ms level3.Dal-core01.net.reach.com [134.159.63.142] 11 190ms 191ms 200ms ge-1-2-54.car3.Dallas1.Level3.net [4.68.122.102] 12 205ms 198ms 211ms 4.78.220.10 13 259ms 195ms 200ms dist-vlan31.dsr3-1.dllstx3.theplanet.com [70.85.127.29] 14 199ms 210ms 206ms dist-vlan21.dsr1-1.dllstx2.theplanet.com [70.85.127.67] 15 197ms 199ms 200ms car2-2-v2.dllstx2.theplanet.com [12.96.160.46] 16 194ms 194ms 207ms 69.93.231.18 Traceroute complete. And my ServerMatrix server in dllstx4: Tracing route to 2.70-84-245.reverse.theplanet.com [70.84.245.2] with TTL of 32: 1 16ms 7ms 32ms 10.128.68.1 2 21ms 10ms 18ms 61.9.207.131 3 10ms 17ms 13ms 61.9.193.3 4 45ms 26ms 9ms GigabitEthernet3-5.pic2.Sydney.telstra.net [139.130.0.4 9] 5 13ms 10ms 13ms GigabitEthernet11-1.ken-core4.Sydney.telstra.net [203.5 0.12.117] 6 10ms 59ms 9ms 10GigabitEthernet3-0.pad-core4.Sydney.telstra.net [203. 50.6.86] 7 8ms 48ms 12ms GigabitEthernet2-2.syd-core01.Sydney.net.reach.com [203 .50.13.38] 8 177ms 183ms 166ms i-12-3.wil-core02.net.reach.com [202.84.249.205] 9 225ms 188ms 219ms i-2-0.dal-core01.net.reach.com [202.84.143.66] 10 203ms 213ms 213ms level3.Dal-core01.net.reach.com [134.159.63.142] 11 215ms 207ms 196ms ge-1-2-52.car3.Dallas1.Level3.net [4.68.122.38] 12 199ms 197ms 199ms 4.78.220.10 13 190ms 201ms 216ms dist-vlan31.dsr3-1.dllstx3.theplanet.com [70.85.127.29] 14 190ms 189ms 197ms dist-vlan41.dsr2-1.dllstx4.theplanet.com [70.85.127.83] 15 189ms 195ms 195ms 67.19.255.142 16 193ms 193ms 235ms 2.70-84-245.reverse.theplanet.com [70.84.245.2] Traceroute complete.
Posted by Erwin, 03-31-2005, 07:50 AM
Quite an amazing outage for so many websites...
Posted by Tommeh, 03-31-2005, 07:50 AM
Lmao! what is the next set in your ip? 67.18.???
Posted by Gually, 03-31-2005, 07:50 AM
69.93.146.x still down..........................................
Posted by Nameslave, 03-31-2005, 07:50 AM
Just moved out of ThePlanet exactly 10 days ago (on March 21).
Posted by Hostasaurus, 03-31-2005, 07:50 AM
Not all mail server software will queue mail for what it views as a domain that doesn't exist in DNS.
Posted by kmike, 03-31-2005, 07:51 AM
Weird, two of our colo-ed servers came up, and they were rebooted. Or rather, power-downed abruptly, then powered up after 1.5 hours. The rest of servers didn't come up yet.
Posted by HighLineHost, 03-31-2005, 07:51 AM
I know its odd tring to get to a website and it doesnt work, its like hey another planet customer hehe ... Oh well... they are still a good company, just intrested to see what happened as it could happen to any of us right?
Posted by Erwin, 03-31-2005, 07:52 AM
Hi kmike... are you from vB.com?
Posted by AKWS, 03-31-2005, 07:52 AM
Lem0nHead, Even with 20% compensation its still a a bad exchange for downtime. At this time I am not really concerned with compensation, I would rather pay a extra 20% if they got my servers back up in the next 2 mins. However being it night time in the USA was a bit on the lucky side, which I have to feel sorry for all the Euro/UK admins.
Posted by NNNils, 03-31-2005, 07:53 AM
I notice our 64.5.* servers have been rebooted.
Posted by The Napster, 03-31-2005, 07:53 AM
Like said it would be good to see how long there down for,
Posted by Laws, 03-31-2005, 07:53 AM
I reckon it was a power outage somewhere and when the CAR routers came back up the hardware crashed out or something...
Posted by ServerZB, 03-31-2005, 07:53 AM
you were sleeping I guess
Posted by vidahost, 03-31-2005, 07:53 AM
It's back for me.
Posted by UH-Matt, 03-31-2005, 07:54 AM
69.93.* still out.
Posted by hosttone, 03-31-2005, 07:54 AM
64.5.44.* down.
Posted by Lem0nHead, 03-31-2005, 07:54 AM
I know but since they will fell a 20% less money this month (think about 20% of millions), i'm sure they will do everything possible so that it doesn't happen again so I think everyone that had they servers down for more than 44 minutes should ask for compensation... even if it's little, even if they don't need that
Posted by M7I, 03-31-2005, 07:55 AM
And here as well.
Posted by alexrobonlin, 03-31-2005, 07:55 AM
Ok. Limbering up to 1:45hrs downtime now... 44.64 mins = 0.1% of a 31 day month. So we're entitled to at least a 20% discount... Alex
Posted by Mix, 03-31-2005, 07:55 AM
The only reason I am with The Planet is for their uptime and Cisco Flood Guard. This is really leaving a bad taste in my mouth.
Posted by Tommeh, 03-31-2005, 07:55 AM
Nope, been here since the beginning of the 'crisis' :p, im in the UK so the time wasnt an inconveince for me :p
Posted by Snowman30, 03-31-2005, 07:55 AM
we have a TCS on 69.93 and a SM ded on 69.93 and they are both out. our other servershowever are all back up thank god
Posted by Iggy, 03-31-2005, 07:55 AM
64.5.48* is stil down for me. What's your next octet?
Posted by DJ_fan, 03-31-2005, 07:55 AM
Interesting about theplanet down. Must not be all there DNS servers, my server is still accessible? All 35 of my domains are up at this time, but Host Gator's sites are down also. I'm not even Irish.. LOL
Posted by kneadingu, 03-31-2005, 07:56 AM
lol we're just the opposite well at TP anyway lol 2 up and 1 down.
Posted by kmike, 03-31-2005, 07:56 AM
Erwin: hi there, yes, that's me... bad reason to meet, eh?
Posted by eric1234, 03-31-2005, 07:57 AM
Are they going to compensate for this downtime? i dont think its 99.99% upload time already..
Posted by HighLineHost, 03-31-2005, 07:58 AM
hehe , its not a DNS problem, they are having a problem with one of their main routers, so that means about 90% or so of their customers are down
Posted by bheka, 03-31-2005, 07:58 AM
67.19.*.* back up for sure now .. still getting intermittent timeouts though. I'm sure it will be 100% fine in a few mins Off to do some damage control
Posted by inc-designs.co.uk, 03-31-2005, 07:58 AM
our server is back up
Posted by NNNils, 03-31-2005, 07:59 AM
64.5.32.* 2 up 1 down 64.5.43.* 1 up
Posted by djdiz-e, 03-31-2005, 07:59 AM
70.85.68.* still down
Posted by xade, 03-31-2005, 08:00 AM
I recommend something. Anything. Get a server on the other side of the world and use it for secondary MX and secondary DNS. You should probably use it to host your service status pages, too. alex
Posted by alexrobonlin, 03-31-2005, 08:00 AM
How does one go about setting-up something like that?
Posted by ALGORYTHM, 03-31-2005, 08:01 AM
yay we're back up
Posted by v0ltair3, 03-31-2005, 08:01 AM
My servers at 69.93.228.* are finnaly up but there is more to it... Both servers had a unexpected shutdown at the time out the connection lost but with a 3 minutes diference between them and they just booted now. That indicates that they manualy powerdown the servers. It was not a OS shutdown. Anyone else had this kind of situation ?
Posted by ResellerPlanet, 03-31-2005, 08:01 AM
70.85 server is down. All my other servers are @ BURST and I was about to move next month :p But I still trust ThePlanet. I don't get one thing however: So far I thought that TP didn't have one central router but several routers connected to several ISP's... So there's no central-point-of-failure, but a whole DC is down. Power issue? They have Diezel and UPS, that shouldn't be possible... I don't get what the cause could be... Maybe all routers connected to the ISP's are connected on the same power source that appears to have a cable break or something lol.
Posted by The Napster, 03-31-2005, 08:01 AM
One of my MSN contacts says there server is back up again, so i think there sorting it out!
Posted by forumtalk, 03-31-2005, 08:01 AM
Hmmm when they going to fix this ? now I know clients will kill us
Posted by Iggy, 03-31-2005, 08:02 AM
4AM PST (on the nose) 64.5.48* came back up. It was rebooted. Odd.
Posted by kmike, 03-31-2005, 08:03 AM
And down go our colo servers again... now it's not funny. Erwin: weren't vbulletin.com/org hosted at theplanet, too? They do not even resolve now. Nameservers point to theplanet.
Posted by BlackMail, 03-31-2005, 08:03 AM
Guess what - 2 of our servers just went down again
Posted by MatrixAu, 03-31-2005, 08:03 AM
Back down here... .
Posted by AKWS, 03-31-2005, 08:03 AM
20% will be a big chunk from them.. As I write this, my server just popped online.. I guess the read my post and will be billing me +20% this month 69.93.160.* was online but down again, of course as they stated everything is "intermitant"
Posted by DJ_Waffles, 03-31-2005, 08:03 AM
70.85 was back up for me for about 15-20 minutes, now it's down again.
Posted by silentnight, 03-31-2005, 08:04 AM
Even my site backs up, but seems their shared sql server is still down. IP starts with 12. That's really a nightmare!!!!!
Posted by NNNils, 03-31-2005, 08:04 AM
It's going on/off all the time here.
Posted by moodler, 03-31-2005, 08:04 AM
Three out of four servers came up for a while, but now they're all gone again. Major **** is going on ... can't wait to hear what the cause was.
Posted by moheybee, 03-31-2005, 08:04 AM
Everything just went down again. This is ricockulous. 70.84.*.*
Posted by eric1234, 03-31-2005, 08:04 AM
After this incident, it really tells me that rackspace are more more stable... what is happening to theplanet? Is anyone able to tell me if we will be compensate for this downtime? alot of my customers call and complain already....
Posted by ResellerPlanet, 03-31-2005, 08:04 AM
So this can mean 2 things: - There was a power outage or - A power outage isn't the cause, but they had to shut down the power for some reasons. I hope there's not a fire or something... -->
Posted by chevelleDOD, 03-31-2005, 08:04 AM
i dont know bout y'all but mine was touch and go and it just went... I'd be suprised if it were a power issue though....
Posted by gpattabhi, 03-31-2005, 08:04 AM
69.93.* down again
Posted by alexrobonlin, 03-31-2005, 08:04 AM
LOL! Probably dolly the cleaner tripping over the "power" cable for the whole shebang Al
Posted by OSW - Chris, 03-31-2005, 08:04 AM
69.93.92.xx dead again
Posted by ivStream, 03-31-2005, 08:04 AM
Yeah, same here. Servers down again.
Posted by Yellow@home, 03-31-2005, 08:04 AM
It seems they are down Again!!!!!
Posted by sashbox, 03-31-2005, 08:04 AM
Just died again... 70.85.x
Posted by v0ltair3, 03-31-2005, 08:04 AM
Servers went down again
Posted by Erwin, 03-31-2005, 08:04 AM
My servers came online just now... but after 5 minutes, offline again...
Posted by Cephren, 03-31-2005, 08:05 AM
wow this is real wierd... down again lets start logging time again
Posted by phenx, 03-31-2005, 08:05 AM
And.. Everything just died again.
Posted by Angel78, 03-31-2005, 08:05 AM
down agai n...argggggggg
Posted by ServerZB, 03-31-2005, 08:05 AM
down AGAIN!!!!!!!!!!
Posted by Snowman30, 03-31-2005, 08:05 AM
we just lost our 69.93 TCS server which had only just coime back up and we stillhave a ded on the 69.93 range out too
Posted by bheka, 03-31-2005, 08:05 AM
mine also just went offline ... damn damn .. already sent an all okay email
Posted by PierreB, 03-31-2005, 08:05 AM
down for us as well
Posted by Valve Ben, 03-31-2005, 08:06 AM
64.5.48.* is down again
Posted by Hostasaurus, 03-31-2005, 08:06 AM
It sounds like they lost power which took down some servers and internal routers. Their internet routers, which are connected to many different ISP's, all stayed online, well at least the few that I checked.
Posted by coight, 03-31-2005, 08:06 AM
Piece of crap, 9 servers down.
Posted by Iggy, 03-31-2005, 08:06 AM
I take that back. Just lost all servers again. Hrrm. What. IS. Going. On?
Posted by AKWS, 03-31-2005, 08:06 AM
ResellerPlanet; They are super redundant, no way to have down time. Unless that night clerk tripped over that plugin again and accidentally unplugged the DC. Or it could be those level 1 techs playing with us... You know April fools is even more funny when they do it 2 week early.
Posted by TS-Dave, 03-31-2005, 08:06 AM
I would be very surprised if anyone received a dime of credit or refund from the planet.
Posted by silentnight, 03-31-2005, 08:06 AM
Oh! My site went down again? I think SM encountering a big problem.
Posted by mikkom, 03-31-2005, 08:06 AM
down again.. this is REALLY annoying, I think I have to change host because of this .. we have business critical stuff (thank god) on other server but our name servers are still on theplanet so I just pray this is temporary Last edited by mikkom; 03-31-2005 at 08:19 AM.
Posted by abstracthost, 03-31-2005, 08:07 AM
offline also
Posted by chatcircuit, 03-31-2005, 08:07 AM
was up 2 minutes and down again. Dont sounds like a power problem, the uptime inforrmation wasnt showing a down.
Posted by chevelleDOD, 03-31-2005, 08:08 AM
and now my routing is all jacked up .... i got thrown off to qwest and then back to att which now wants to bounce me up to IL ..... i'd bet bgp has us on iggy
Posted by almico, 03-31-2005, 08:08 AM
I'm getting this for a PING: From 144.232.8.67 icmp_seq=15 Time to live exceeded From 144.232.8.67 icmp_seq=16 Time to live exceeded From 144.232.20.17 icmp_seq=17 Time to live exceeded ... A routing problem?
Posted by kmike, 03-31-2005, 08:08 AM
Same here, two of several servers in cabinet which came up were abruptly powered down at the time of connection crash. The rest still didn't even came up.
Posted by PierreB, 03-31-2005, 08:08 AM
our server with IP starting as 70.85 are up, weird.
Posted by Hostmax-br, 03-31-2005, 08:08 AM
nah... I believe it was a matter of fiber or something like that. Couldn´t be just power source, they got to have UPS and other sources.
Posted by GIGANET, 03-31-2005, 08:08 AM
damn - i just informed my customers that we're back in biz, now down. so unprofessional
Posted by andrewgmol, 03-31-2005, 08:08 AM
69.93.x.x still down
Posted by Valve Ben, 03-31-2005, 08:08 AM
bleh, 70.85.79.* is down too... My phones are red hot! :-(
Posted by vidahost, 03-31-2005, 08:08 AM
Was up, back down now - 69.93.*.* Server had also been rebooted, didn't get to check the logs before it died again though.
Posted by hosttone, 03-31-2005, 08:08 AM
down again. just now one server up and now, that server down again.
Posted by woxcom, 03-31-2005, 08:08 AM
we are down again!!! This is really - argh!!
Posted by Russ Foster, 03-31-2005, 08:08 AM
I saw 119 days uptime on the server there but yup gone again Rus
Posted by ResellerPlanet, 03-31-2005, 08:09 AM
lol first of all, a dc like TP is not also redundant, but also SECURE. There's no way a small accident from the nigh clerk or cleaning lady caused a whole DC downtime, that is just not possible. An april fools joke is offcourse also out of the question :p
Posted by Gentle Giant, 03-31-2005, 08:09 AM
Down again - and even more serious - the batteries in my cordless mouse have died - servers I can cope with, but how is one supposed to work with a dead mouse!!!!
Posted by Linux7, 03-31-2005, 08:09 AM
Down over 2 hours straight now at ServerMatrix. My ZoneEdit backup e-mail service is working just fine though as I can see Postini delivering it to the ZoneEdit server. So won't miss any e-mails but this is a hell of a way to find out it's working ok! I moved all my domains of this server yesterday save one, so got VERY lucky but I feel for the rest of you guys! Last edited by Linux7; 03-31-2005 at 08:16 AM.
Posted by bheka, 03-31-2005, 08:10 AM
if its power to whatever .. why aren't the generators kicking in .. this is getting frustrating now ... really really frustrating
Posted by [x10]Corey, 03-31-2005, 08:10 AM
69.93.194.* 70.85.87.* 69.93.137.* All down for me.
Posted by HighLineHost, 03-31-2005, 08:10 AM
lol what a day hehe..... You know they may be moving servers down the street to their new datacenter or soemthing and just keeping everyone in the dark as its really odd that they are down.
Posted by pwbc, 03-31-2005, 08:10 AM
My 69.93.x.x box went down about 2 hours ago, came back up ~30 minutes ago, and is now down again (hadn't been rebooted). I've been with these guys for 16 months now, and I've never seen downtime as bad as this. I'm just thankful that my server's not a commercial venture - I feel very sorry for those of you having to deal with support calls/tickets from customers right now. It's quite disconcerting that so much of TP's to-date impressive network was brought down in one go. :\ I know a techie there, and I can only feel sympathy for their staff at the moment - I know how I'd feel having to cope with a problem like this, and can only imagine the deluge of calls they must be getting... I hope it all stabilises soon.
Posted by EricP, 03-31-2005, 08:11 AM
Down here - again.
Posted by ResellerPlanet, 03-31-2005, 08:11 AM
Yeah I know... I don't know how power things work in datacenters, but if there's a problem at the point where the power is divided (and thus also the part that controls when power generators have to get in action)...
Posted by AKWS, 03-31-2005, 08:12 AM
GIGANET; When your server comes back, then email everyone that your working on the problem. But it may still be down intermittently as techs are still working on the problem and testing everything to locate the fault.
Posted by TrevorN, 03-31-2005, 08:12 AM
just imagine how the planets and servermatrix's are haha
Posted by navmonkey, 03-31-2005, 08:12 AM
Is the planet on fire? Or under attack?
Posted by HuskyNOC, 03-31-2005, 08:12 AM
yap ... all my customer server on planet also down
Posted by universal2001, 03-31-2005, 08:12 AM
hrm it seems to be down again, not good.
Posted by Gint, 03-31-2005, 08:12 AM
haha You mean you don't have a redundant power supply for you mouse? I have mine hooked up to gas generators ready to kick in if the batteries die...
Posted by John D, 03-31-2005, 08:13 AM
Glad this was down to ThePlanet... Thought I had something wrong on my cPanel servers.... Any word from TP on this?
Posted by camron978, 03-31-2005, 08:13 AM
this downtime is hurting my business
Posted by ResellerPlanet, 03-31-2005, 08:13 AM
Then atleast their phone network is still up :p
Posted by MatrixAu, 03-31-2005, 08:13 AM
Where's the datacentre cam!! . Or is that down too.. .
Posted by c0bra, 03-31-2005, 08:14 AM
Down again here too.
Posted by Hostmax-br, 03-31-2005, 08:14 AM
Does TP gave any announcement away?
Posted by pstation, 03-31-2005, 08:14 AM
yeah, good thing they arent using VoIP
Posted by sam2579, 03-31-2005, 08:14 AM
mine came back and then down again.
Posted by rvrsl, 03-31-2005, 08:14 AM
Using all your experience with generators, maybe you should do some contract work at the TP datacenter then so you can ensure this doesn't happen again.
Posted by Erwin, 03-31-2005, 08:14 AM
This is going to be one of the longest threads here I think...
Posted by accyroy, 03-31-2005, 08:14 AM
Oh cheers for being glad that thousands are down instead of just you hehe....
Posted by EricP, 03-31-2005, 08:14 AM
My god. This is insane.
Posted by Cephren, 03-31-2005, 08:14 AM
imagine those poor bastards looking at porn sites...... and then going through galleries... "the page cannot be displayed"
Posted by JDMundo, 03-31-2005, 08:15 AM
Argh, man, this is beyond frustrating. Server down again.
Posted by andrewgmol, 03-31-2005, 08:15 AM
Oh, the humanity.
Posted by nothingg, 03-31-2005, 08:15 AM
this is so Fu???'ing annoying!!!
Posted by ResellerPlanet, 03-31-2005, 08:15 AM
Well some people called and so far they say they're working on it. I can't find any other way they could give announcements... This thread feels more like IRC than a forum lol
Posted by ancilenet, 03-31-2005, 08:15 AM
Gone Again. 70.85.* 1 209.190.4.1 (209.190.4.1) 1.066 ms 1.067 ms 0.911 ms 2 letmeout.fastpc.com (206.222.25.9) 0.884 ms 0.675 ms 0.549 ms 3 twt-border.ee.net (209.51.192.22) 1.098 ms 1.365 ms 1.297 ms 4 66.237.112.17.ptr.us.xo.net (66.237.112.17) 1.776 ms 5.715 ms 1.443 ms 5 p4-3-0.MAR2.Columbus-OH.us.xo.net (207.88.84.141) 2.919 ms 2.331 ms 2.169 ms 6 p5-1-0-2.RAR2.Washington-DC.us.xo.net (65.106.3.177) 14.305 ms 14.538 ms 14.167 ms 7 p1-0.IR1.Ashburn-VA.us.xo.net (65.106.3.138) 16.253 ms 15.054 ms 14.543 ms 8 206.111.0.22.ptr.us.xo.net (206.111.0.22) 15.454 ms 15.056 ms 14.930 ms 9 so1-0-0-2488m.ar1.dal2.gblx.net (67.17.73.34) 49.381 ms 48.878 ms 90.343 ms 10 The-Planet.ge-0-2-1.ar1.DAL2.gblx.net (64.213.176.150) 44.699 ms 42.915 ms 46.508 ms 11 * * * 12 * * * 13 * * *
Posted by gama3ana, 03-31-2005, 08:15 AM
ugh, ThePlanet of Apes...my server down again... 64.5.* 65.56.* 69.93.*
Posted by wheimeng, 03-31-2005, 08:16 AM
Down again, how great, any updates?
Posted by Mix, 03-31-2005, 08:16 AM

Posted by Gjuro, 03-31-2005, 08:16 AM
They are UP !!!
Posted by gpattabhi, 03-31-2005, 08:16 AM
69.93.* up
Posted by djdiz-e, 03-31-2005, 08:16 AM
our grandchildren will hear about this say .. "The Day the planet fell of the face of the planet" lol
Posted by Cephren, 03-31-2005, 08:16 AM
yay weve hit the 2 hour mark
Posted by Erwin, 03-31-2005, 08:16 AM
Good call. LOL!
Posted by TrevorN, 03-31-2005, 08:16 AM
if this was a power issue it may not have been a generator. it could be a ups that got overloaded which cascaded to the others (like the northeastern black outs)
Posted by Hostmax-br, 03-31-2005, 08:16 AM
SOMEBODY PLEASE CALL TP... somebody has to got something to say about that... Damn...
Posted by Hostasaurus, 03-31-2005, 08:16 AM
Yep, auto transfer switches and power distribution units do sometimes fail but not very often. Does anyone remember the cause of that fire in the Alabanza facility a year or so ago? I think that was caused by some power equipment failing which caused the local fire marshall to order the power off for the building.
Posted by DJ_Waffles, 03-31-2005, 08:16 AM
seems it's back up
Posted by HighLineHost, 03-31-2005, 08:17 AM
Something tells me after this mayjor downtime more people will be on the market for dedicated server with non-planet business lol
Posted by Iggy, 03-31-2005, 08:17 AM
69.93.72 just popped back into existence 4:17AM PST 64.5* still down
Posted by rvrsl, 03-31-2005, 08:17 AM
Maybe it'll be a new WHT active member record soon!
Posted by ancilenet, 03-31-2005, 08:17 AM
70.84.228.* is still up tho - anyone else in that range?
Posted by UH-Matt, 03-31-2005, 08:17 AM
If its power related then they could learn a thing or two from EV1.
Posted by LayZ, 03-31-2005, 08:17 AM
Must be nearly 2 hrs now? 69.93.181.* I have been with them 18 months and this is the first time anything like this has happened but still totally unacceptable.
Posted by moheybee, 03-31-2005, 08:17 AM
70.84.*.* is back up!
Posted by dee_at_candl, 03-31-2005, 08:18 AM
LOL
Posted by jarekb, 03-31-2005, 08:18 AM
Simple question Where is redundancy?
Posted by andrewgmol, 03-31-2005, 08:18 AM
Not here...
Posted by MatrixAu, 03-31-2005, 08:18 AM
http://69.41.224.20/popup.html < Datacenter cam, no action there... .
Posted by Mix, 03-31-2005, 08:18 AM
I am still down
Posted by v0ltair3, 03-31-2005, 08:18 AM
It cant be a power problem. 2 of my servers that are down again have been shutdown with a 3 min diference. That indicates that someone press the button.
Posted by Gually, 03-31-2005, 08:18 AM
69.93.146.x still down
Posted by Erwin, 03-31-2005, 08:18 AM
Still down too...
Posted by chatcircuit, 03-31-2005, 08:18 AM
ArteryPlanet.Net - All servers were up, now all down again.
Posted by TrevorN, 03-31-2005, 08:19 AM
the webcam is working fo rme... lol
Posted by PierreB, 03-31-2005, 08:19 AM
4 out of 6 are up, missing 2 to be up
Posted by HighLineHost, 03-31-2005, 08:19 AM
Maybe their monkey was in the back room un-pluggin cat5 cables
Posted by djdiz-e, 03-31-2005, 08:19 AM
im 70.85.68.* ..still down
Posted by zsindhu, 03-31-2005, 08:19 AM
... as it is ours and almost everyone else's
Posted by Cephren, 03-31-2005, 08:20 AM
if your server has been down since the begining of this thing and is still down....you have been down over 110 minutes
Posted by forumtalk, 03-31-2005, 08:20 AM
Where is planet guy in WHT , at least they must tell us what going on then we can hide
Posted by Mix, 03-31-2005, 08:20 AM
I am late for my fulltime job
Posted by dee_at_candl, 03-31-2005, 08:20 AM
69.93.188.* is up.
Posted by Patryk, 03-31-2005, 08:20 AM
It's up now.
Posted by MatrixAu, 03-31-2005, 08:20 AM
Working for me too. Pretty boring though.
Posted by AKWS, 03-31-2005, 08:21 AM
We better get someone in the vicinity with a pair of eyes on the DC then, as their just might be a stream of L3 techs lugging servers down the street. Or their security guard is "busy" and they are lugging them strait home!
Posted by djdiz-e, 03-31-2005, 08:21 AM
maybe they looped it ..lol
Posted by TrevorN, 03-31-2005, 08:21 AM
yeah, i was hoping for people running around doing stuff haha
Posted by richenou, 03-31-2005, 08:21 AM
not all is working now already server down !
Posted by moheybee, 03-31-2005, 08:21 AM
My servers have not lost power this entire time. 70.84.*.*
Posted by Snowman30, 03-31-2005, 08:21 AM
our servers are dropping off again one by one. surely someone from the DC can make a statement here about whats happening...
Posted by silentnight, 03-31-2005, 08:22 AM
69.93.97.* backs up, I hope their shared sql sever will be up very soon, since my site is using their shared database server.
Posted by HighLineHost, 03-31-2005, 08:22 AM
hehe thanks great, tho I think maybe someone hit a light switch and now they cant figure it out
Posted by MatrixAu, 03-31-2005, 08:23 AM
haha yeah. .
Posted by Hostmax-br, 03-31-2005, 08:23 AM
not up here yet...
Posted by TrevorN, 03-31-2005, 08:23 AM
not if they are having a hard time accessing the internet
Posted by patrick24601, 03-31-2005, 08:23 AM
If they were moving anything I am 110% sure they would have broadcasted this info.
Posted by NNNils, 03-31-2005, 08:23 AM
They rebooted the servers again
Posted by AKWS, 03-31-2005, 08:23 AM
I moved from Alabanza a few years ago. However the fire could explain for their mega slow servers between 2.00am and 1.59am every day.
Posted by nothingg, 03-31-2005, 08:23 AM
i wonder if they will give us an SLA credit
Posted by linesiders, 03-31-2005, 08:23 AM
As I wait for my server to come back up I just wanted to make sure that I posted on this thread (insert smiley face that I can't because my server is down)...
Posted by [x10]Corey, 03-31-2005, 08:24 AM
Are we responsible for keeping track of the downtime for an SLA credit or can we just request one and they credit us the due amount?
Posted by Hoojx, 03-31-2005, 08:25 AM
My servers located at 67.18.* are just back online.
Posted by mikkom, 03-31-2005, 08:25 AM
our server has been up for some minutes now (was down, up and down again). My god this is frustrating, at least I will move our other dns away from theplanet. Any suggestions for plain dns service?
Posted by eric1234, 03-31-2005, 08:25 AM
Can somebody from theplanet please explain what is happening, I had so many customers calling me to complain.... Now i understand that "cheap thing are not good, good thing are not cheap"...
Posted by nothingg, 03-31-2005, 08:26 AM
i want to know that too
Posted by mikkom, 03-31-2005, 08:26 AM
My servers were never rebooted, uptime is still there
Posted by ViAdcK, 03-31-2005, 08:26 AM
69.93.219.* still down !!!!
Posted by Gually, 03-31-2005, 08:26 AM
I think they haven't time for answer forums...
Posted by c0bra, 03-31-2005, 08:27 AM
You are responsible.
Posted by PierreB, 03-31-2005, 08:27 AM
call them to know what's happening. I guess they themself don't know yet.
Posted by zsindhu, 03-31-2005, 08:27 AM
I sure hope not.
Posted by chrisbond, 03-31-2005, 08:27 AM
TMS' he.net feed is still down in theplanets DC too. You'd really think by now they'd have some answers
Posted by wheimeng, 03-31-2005, 08:27 AM
Could it be power outage?
Posted by AKWS, 03-31-2005, 08:28 AM
BTW no fires and all seems quiet..... http://69.41.224.20/popup.html
Posted by Yellow@home, 03-31-2005, 08:28 AM
Here also no reboot. Range 67.18*
Posted by Lem0nHead, 03-31-2005, 08:28 AM
that's not a problem the problem is probably in the "start"... not in the "middle of the servers" so you shouldn't see techs running from on side to other there
Posted by Gint, 03-31-2005, 08:28 AM
haha. I think they may need it right about now... Of course that assumes that power is the issue. The scary thing is, I haven't heard anything definate on what the issue is.
Posted by v0ltair3, 03-31-2005, 08:28 AM
1 server is up again but have been power down. The other one at 69.93.228.* is still power down.
Posted by Hostmax-br, 03-31-2005, 08:28 AM
I don´t think so... I don´t think its power
Posted by Cephren, 03-31-2005, 08:28 AM
ive logged 1 hour and 62 mins downtime from the server logs for the last server that came back up....
Posted by jeroman, 03-31-2005, 08:28 AM
Hm..most of the servers is up now but still one to go and I can't reach SM or theplanet sites yet.
Posted by bjdea1, 03-31-2005, 08:29 AM
Well at least we are all in this together - so its not like customers can blame us really - plenty of proof its The Planet
Posted by mikkom, 03-31-2005, 08:29 AM
my range is 69.56.*
Posted by neXolt, 03-31-2005, 08:29 AM
Holy, im lucky, my servers are still online, i hope your guys servers get back online soon
Posted by PierreB, 03-31-2005, 08:29 AM
Wouldn't say it's a power outtage,, server are coming back online and still have the time they have been up.
Posted by djdiz-e, 03-31-2005, 08:29 AM
maybe that is a video loop ..lol
Posted by AKWS, 03-31-2005, 08:29 AM
nothingg: Keep track of your own downtime and then notify them about it later. They WONT keep track of your downtime for you.
Posted by andrewgmol, 03-31-2005, 08:29 AM
2 hours and 2 minutes?
Posted by jedrek, 03-31-2005, 08:30 AM
67.18.79.* still down. :/
Posted by EricP, 03-31-2005, 08:30 AM
...and down again!
Posted by [x10]Corey, 03-31-2005, 08:30 AM
I'm no linux master, what logs should I be checking to see the amount of downtime occured so I can request SLA?
Posted by BlackMail, 03-31-2005, 08:30 AM
And down again (3rd time) - This is slowly becoming annoying
Posted by gpattabhi, 03-31-2005, 08:30 AM
69.93.* down again
Posted by WebAfrica, 03-31-2005, 08:30 AM
loooks like they back down again!
Posted by PierreB, 03-31-2005, 08:31 AM
2hour 40 minutes here not counting the time the servers were backup. Still have 1 server that did not at all come back.
Posted by v0ltair3, 03-31-2005, 08:31 AM
IT IS NOT A POWER OUTAGE. I have 2 servers side by side. 1 is up and the other is POWER DOWN for the second time. The first time they went down with a 3 min diference. That means manual power down.
Posted by bjdea1, 03-31-2005, 08:31 AM
67.18.* - not woking for us - all out other servers are up and fine.
Posted by Hostmax-br, 03-31-2005, 08:31 AM
yeah... so I guess... 2 hr and 2 min...
Posted by Iggy, 03-31-2005, 08:31 AM
And down again. Mistakes happen, understandably. This yo-yo action is not though.
Posted by pstation, 03-31-2005, 08:31 AM
mine went back down again ^*D$&YHPNO^T&@#$N&@#TO(#O$@#$@#*$@#M)$*()@#(E@<@#()#@$@ (HM#@H((PN@#$OHM @#OMPI$@#JKO$:@K#L$L:K@#$@#M$L#$@#$ ARGH@#@!!
Posted by Cephren, 03-31-2005, 08:31 AM
down again...............................
Posted by mikkom, 03-31-2005, 08:31 AM
OH MY ****ING GOD My servers are down again (3rd time), **** **** **** ****
Posted by Patryk, 03-31-2005, 08:31 AM
UGH! 70.85.x down again..
Posted by Mix, 03-31-2005, 08:32 AM
Mine cam again for 2 minutes now down again. What gives?
Posted by Gentle Giant, 03-31-2005, 08:32 AM
Good news - I have new batteries in my mouse
Posted by awardle2005, 03-31-2005, 08:32 AM
My servers are back up now on 69.93 range they seem to be up and down though How do we go about claiming for downtime and loss of earnings? Also those PC's in the datacenter look a bit dated. Aaron
Posted by Gually, 03-31-2005, 08:32 AM
we will bat the record of posts / hour in web hosting talk, isn't it?
Posted by HighLineHost, 03-31-2005, 08:32 AM
Ok i need to take my-self off of this post, I have like 200 emails in the past 10min unsubscribe from this thread , ah much better
Posted by ServerZB, 03-31-2005, 08:32 AM
backup for 10 inutes and then down again.... third down already
Posted by Lem0nHead, 03-31-2005, 08:32 AM
it looks like they're plugging and unplugging cables
Posted by forumtalk, 03-31-2005, 08:33 AM
I wonder what TP going to tell us & what we going to tell our clients
Posted by MatrixAu, 03-31-2005, 08:33 AM
One of my servers has been totally uneffected by all this... lol. And it's a ServerMatrix server in DLLSTX4.
Posted by Hostmax-br, 03-31-2005, 08:33 AM
65.51.*.* not working properly that sux...
Posted by TrevorN, 03-31-2005, 08:33 AM
yay!
Posted by richenou, 03-31-2005, 08:33 AM
downnnnnnnnnnnnnnnnnnnnnnnnn !
Posted by AKWS, 03-31-2005, 08:33 AM
jdhf: I think in this rare occasion everyones servers will be down about the same length of time. Thus you can wait till everything is backup and running and request SLA for the same amount of time that everyone else claims. I don't think they will be that picky this time about logs.
Posted by v0ltair3, 03-31-2005, 08:33 AM
Ok, now the 1th server is down again
Posted by galantina, 03-31-2005, 08:34 AM
Has anyone got any relevant info from them?
Posted by BoardHoster, 03-31-2005, 08:34 AM
Somehow I don't see compensation as a primary issue right now.... Tomorrow Yes, but let it get fixed first, deal with $$$ later.
Posted by ServerZB, 03-31-2005, 08:34 AM
seems a good time to have a planet gathering here
Posted by HighLineHost, 03-31-2005, 08:34 AM
that took a while , did you have to run to the store
Posted by Lem0nHead, 03-31-2005, 08:34 AM
more than 2:20h probably the biggest global outage of enterprise datacenters?
Posted by Erwin, 03-31-2005, 08:35 AM
Servers came back online for another 5 minutes... they all went down again...
Posted by TrevorN, 03-31-2005, 08:35 AM
no, i'm not the one who needed batteries... haha
Posted by silentnight, 03-31-2005, 08:35 AM
down again!! YOU MAKE ME CRAZY!
Posted by beachtrader, 03-31-2005, 08:35 AM
I called just now and there is no eta. The tech I spoke with just got in and said they are not being told what has happened yet. He said he thinks they may have isolated the issue though.
Posted by [x10]Corey, 03-31-2005, 08:35 AM
AKWS: Thanks... so can anyone recommend a new datacenter while we're waiting?
Posted by Tom O'Connor, 03-31-2005, 08:35 AM
Must be a handy day at the office then Matt. This is business hours for us here in Ireland the phones have been ringing off the hook since 11 am.
Posted by AKWS, 03-31-2005, 08:35 AM
awardle2005 & Everyone else; Wait untill they have fixed everything and all servers are back online. Then once that happens, they will post news on their forums about where to claim SLA downtime compensation.
Posted by v0ltair3, 03-31-2005, 08:35 AM
Yep, that is it but the big problem is that they are unplugging POWER cables not Network cables. That could lead to big data corruption or complet OS failure.
Posted by EricP, 03-31-2005, 08:35 AM
I would love to hear the explanation. We have logged over 2 hours of down time.
Posted by pwbc, 03-31-2005, 08:36 AM
My server's down again. I echo Iggy's sentiments about the yo-yo effect. At least I got the chance to check my Apache logs to see how long I was down for the first time (70 minutes). I hope TP pulls it all together soon, for their sake as much as ours. LOL. Thanks for lightening my mood.
Posted by 1980, 03-31-2005, 08:36 AM
Stumbled on this thread, nice to see i'm not alone in this one! Anyone have an idea of the amount of sites hosted at theplanet affected by this outage? Must be huge. I'm actually hosted at intersq.com, but they use TP servers. Anyway, must be hell at the tech and helpdesk departement there now, nice to find some comfort at least here! And for the record, still offline here aswell
Posted by coight, 03-31-2005, 08:36 AM
Checks phone, upto 22 messages received.
Posted by TS-Dave, 03-31-2005, 08:36 AM
We've been hearing garbage like this for over 2 hours now.
Posted by Hostmax-br, 03-31-2005, 08:36 AM
I got talk with my lawyer...
Posted by mikkom, 03-31-2005, 08:37 AM
how in the ****ing hell they can claim 100% uptime? I'm pulling my hair here, my god I'm happy that our business critical stuff is on another server here in finland. This would have been a total disaster for our business (we asp business applications)
Posted by coolio, 03-31-2005, 08:37 AM
they're gonna pull a dedicatednow.com/unitedcolo.com on us?
Posted by Erwin, 03-31-2005, 08:37 AM
Are you sure they are unplugging power? My servers have uptime of 48 days when I last checked when it was intermittently online...
Posted by dolay, 03-31-2005, 08:37 AM
No i remember 6hours of downtime at VlueWeb at early 2004
Posted by Yellow@home, 03-31-2005, 08:37 AM
Have they hired staff from managed.com or so????? This is not good!!!!!!!!!!!!!!!!!!!
Posted by chrisbond, 03-31-2005, 08:37 AM
Just got through to someone and he wouldnt tell me anything just said "we have no ETA", didnt know what the the problem was or nothing. Even asked do you have a temp page for announcements - answer was no.
Posted by whmcsguru, 03-31-2005, 08:37 AM
You won't be able to tell from the "linux" logs, that's where outsourcing uptime checks comes in, from a completely and totally offsite location. From the sounds of things, it IS a power issue. Either that or a LOT of equipment going to crap @ once (highly unlikely). Just because you're not getting all of your servers offline doesn't mean that it ISN'T. Your servers aren't guaranteed to be next to each other, and in most cases aren't. Sounds like they've got some faulty circuitry there causing these to come back up, getting overloaded, then causing the servers to be shutdown again.
Posted by BlackMail, 03-31-2005, 08:37 AM
Again, I think it is very poor that nobody from their staff finds a way to post any message to a known board like WHT. I mean it's not like their staff doesn't hang out here to promote products. Even if their connectivity is gone - some servers are UP and somebody COULD just post a note here. If it comes to sales they post follow up's too so why not at such a critical time?
Posted by ffeingol, 03-31-2005, 08:37 AM
You must not have delt with SM/TP and SLA credits They are very picky. Frank
Posted by craig1972, 03-31-2005, 08:37 AM
Can someone tell me why resellerworkz.com is down? I thought they use savvis networks? And savvis.com is up!
Posted by Sam Bauers, 03-31-2005, 08:38 AM
I think we should be more creative with our suggestions as to what is wrong... I think that there are several monkies on the loose in the planets datacentres. That would explain a lot I think. None of this pie-in-the-sky conjecture about power outages and router failure.
Posted by Erwin, 03-31-2005, 08:39 AM
Like someone said... ThePlanet of the Apes... LOL!
Posted by BoardHoster, 03-31-2005, 08:39 AM
Why? Is your lawyer a computer tech? Can he fix the servers? .
Posted by coight, 03-31-2005, 08:39 AM
This is what happens when you spread staff accross three datacenters.
Posted by AKWS, 03-31-2005, 08:39 AM
jdhf; There is plenty of other DC out there, but I would move based on past performace and wait untill we found out what went wrong. If some stupid a$$ gas workers were digging up fiber outside, then I have no problem staying with TP. So its really best to wait and see what wrong. (Yes, I know it wasn't fiber loss outside. That was just an EXAMPLE)
Posted by TrevorN, 03-31-2005, 08:39 AM
it could be that they can't access the internet to do that.
Posted by UH-Matt, 03-31-2005, 08:39 AM
Hey, TP can all stay down all they like ... no tickets to tend to is always good
Posted by sbdquinten, 03-31-2005, 08:40 AM
Come on, keep up the postingspeed! 2.3 posts per minute... ow!
Posted by chevelleDOD, 03-31-2005, 08:40 AM
nah CIHost holds that title over 72 hours if I remember correctly
Posted by mikkom, 03-31-2005, 08:40 AM
my servers have been down-up-down-up-down (and still down) but uptime shos that they were never booted. This is not a power issue.
Posted by kmike, 03-31-2005, 08:40 AM
I'm absolutely sure, servers which were up second time were rebooted again. Probably power issue after all...
Posted by sashbox, 03-31-2005, 08:40 AM
They won't give any specifics for 'legal' reasons.
Posted by Sam Bauers, 03-31-2005, 08:40 AM
What we need here are some monkey wranglers...
Posted by Bill-zilonhost, 03-31-2005, 08:40 AM
I am back down, again.
Posted by Hostmax-br, 03-31-2005, 08:40 AM
Actually he is, but my deal will be that I ALREADY lost some CRITICAL stuff... 2 hrs is more than enough... Somebody has to pay for that...
Posted by rvrsl, 03-31-2005, 08:40 AM
As much as I hate to play conspiracy theory, doesn't anyone seem to think its a bit odd that such a high end datacenter can have such a massive outage? Some users are reporting having multiple servers and them being rebooted 3.. 4 minutes apart from each other which would indicate manual reboots. More than a bunch of people here are reporting reboots on their servers, while some are not. Why would they be manually rebooting servers during an outage? There doesn't seem to be any sort of a repeatable pattern to which part of the TP network the downtime is affected, which DC is affected. Combine that with the fact that they are not posting here, or have made a statement at all, that they simple "Don't know whats happening". Has anyone considered a possible federal law enforcement raid to check on servers? Maybe looking for something? Seems a lot like the whole Foonet raid. Seems very odd. Last edited by rvrsl; 03-31-2005 at 08:44 AM.
Posted by ServerZB, 03-31-2005, 08:41 AM
everyone is clubbing here lol
Posted by nothingg, 03-31-2005, 08:41 AM
is it true that is no uptime guarantee for the matriXtreme servers ???
Posted by SSHocker, 03-31-2005, 08:41 AM
They do use savvis but their own stuff is TP
Posted by BlackMail, 03-31-2005, 08:41 AM
TrevorN - they have multiple datacenters and a few are up. I believe it would be possible that at least one of their employees posts at least a note - even if it is a "i dont know but we are aware".
Posted by Lem0nHead, 03-31-2005, 08:42 AM
the worst scenario I can imagine is power having problems (like high or low voltage), leading to hardware problems but that's very unlike to happen... i'd say it still looks like a network problem and we could make bets here i bet they will be back in 20 minutes
Posted by AKWS, 03-31-2005, 08:42 AM
Erwin; It seems after they servers back online, they are rebooting them as well. As I assume they are troubleshooting and trying to back track any problems.
Posted by coight, 03-31-2005, 08:42 AM
When we checked our main box it had 66 days uptime. Then it went down again. 34 messages now
Posted by BoardHoster, 03-31-2005, 08:42 AM
ffeingol>>Doreo; I see phpBB.com is still up. .
Posted by whmcsguru, 03-31-2005, 08:42 AM
check out http://www.fastservers.net . Great prices, server auctions for old stuff (when it's needed), absolutely fantastic support and excellent network. Just because YOUR servers haven't been shutdown doesn't mean it's not a power issue. All it takes is a circuit to go causing the router your servers are on to go, causing your server to be down. The only other explaination for the mass of "server shutdowns" is hardware, and, while hardware can be faulty, it's not likely that this much hardware was this faulty all at once.
Posted by DCM, 03-31-2005, 08:42 AM
Do any of you think before you post?
Posted by The3bl, 03-31-2005, 08:43 AM
Geez you guys can dream some stuff up can't you? When my servers came online none of them had lost power or been rebooted. So I doubt it is a power issue. Anything is a guess until they come online and explain it. All anyone can do at this point is share the pain and wait. I am sure they are working on it as hard as they can.
Posted by Iggy, 03-31-2005, 08:43 AM
HA! Thanks for that! "YOU MANIACS! YOU BLEW IT UP! DAMN YOU! GOD DAMN YOU ALL TO HELL!"
Posted by Erwin, 03-31-2005, 08:43 AM
@ AKWS and kmike They seem to be rebooting some servers but not others like mine (thank God).
Posted by coolio, 03-31-2005, 08:43 AM
Burst will probably stay in the history books for this record
Posted by Von Vonder, 03-31-2005, 08:44 AM
No, but I heard thinking is overrated. My servers in the oldest datacenter were not rebooted.
Posted by TrevorN, 03-31-2005, 08:44 AM
people have been noting that they have servers down in different dc's so it could still be that.
Posted by Yellow@home, 03-31-2005, 08:44 AM
Up again for the mean time!!
Posted by mikkom, 03-31-2005, 08:44 AM
up again. How long this time? (69.56.*) By the way,. checked uptime. No reboots or power outage.
Posted by wheimeng, 03-31-2005, 08:44 AM
LOL, I can never catch up the speed, after finished a page, there goes another.
Posted by BoardHoster, 03-31-2005, 08:44 AM
Yikes, I've been promoted! Somebody said something wayyyy back there about post/day on this board; What about posts/minute? .
Posted by gpattabhi, 03-31-2005, 08:44 AM
69.93.* up again
Posted by intek, 03-31-2005, 08:45 AM
Hello Has anyone any idea when they will be back up and running? We have 2 servers over there.
Posted by WebAfrica, 03-31-2005, 08:45 AM
Did webhostingtalks DNS servers just go down too? I could check this forum for about 5 minutes...
Posted by Erwin, 03-31-2005, 08:45 AM
Yup, up again. uptime 48 days - so mine hasn't been rebooted. I ain't doing anything on it.
Posted by chrisbond, 03-31-2005, 08:46 AM
Anyone on HE.NET feeds up?
Posted by silentnight, 03-31-2005, 08:46 AM
I think that could be a network problem, I remember TP has blocked all customers from Iran, is this possible that Iranian hackers are attacking TP?
Posted by AKWS, 03-31-2005, 08:46 AM
ffeingol; I have not dealt with them SLA credits before, but usually its only a few servers affected and they want to make sure it was their fault. However in this large scale situation, that was just a guess.. However they might come online and say "Outtage? We didn't have any problems. Lets see your logs".. You never know
Posted by nothingg, 03-31-2005, 08:46 AM
is it true that there is not uptime guarantee for matriXtreme servers ?????
Posted by andrewgmol, 03-31-2005, 08:46 AM
Not here...
Posted by jedrek, 03-31-2005, 08:46 AM
69.56.* up and down 67.18.* still down
Posted by v0ltair3, 03-31-2005, 08:46 AM
Well, first good resume of the situation I've seen. What I dont understand is why do they power down the servers one by one ?
Posted by Yellow@home, 03-31-2005, 08:46 AM
Some explaining from TP wont hurt ;-)
Posted by ServerZB, 03-31-2005, 08:46 AM
just back up ****for the time being****
Posted by NetSender, 03-31-2005, 08:46 AM
Mines never been back up I'm on TMS, HE.net Has anyone on HE.net hosted in TP been back up? I'm on 216.66.21.* and 216.66.19.* ranges.
Posted by ffeingol, 03-31-2005, 08:46 AM
phpbb.com is not at TP. It's a coloc at Server Central. But you can feel bad for me I'm in a hotel room with the laptop connected to a cell phone for connectivity. OTOH, I'm getting 200 kbps . Frank
Posted by ImLagging, 03-31-2005, 08:47 AM
This doesn't have anything to do with server logs or even what OS you're using. You need a way of pinging your server from an outside location, either a server in another datacenter or a 3rd party monitoring company such as Alertra or WebSitePulse. They'll ping your server and notify you of any downtime as well as let you know what the total downtime was.
Posted by BlackMail, 03-31-2005, 08:47 AM
Just tried to download my raw access logs to prove the downtime.... CPanel.net Unactivated License File oh well
Posted by Pheaton, 03-31-2005, 08:48 AM
So whats happening here? Can anyone give me a brief summary? I dont think I can read through 46 pages.
Posted by chrisbond, 03-31-2005, 08:48 AM
NetSender, TMS HE's feeds seem to have been down through the outage ive got servers on both those segments and nothing yet. Im sure jose has gone down there to fix there probs
Posted by sashbox, 03-31-2005, 08:48 AM
Doubtful... that would be a setup for a class-action. They'd be smart to just bite-it and be generous (credit everyone across the board)
Posted by navmonkey, 03-31-2005, 08:48 AM
Anyone try to use their DRAC card to access their server?
Posted by BoardHoster, 03-31-2005, 08:48 AM
"I'm getting 200 kbps" Wouldn't a pencil be faster?
Posted by NNNils, 03-31-2005, 08:48 AM
If I was them I would not dare to ask for a proof of downtime...
Posted by Darktwist, 03-31-2005, 08:48 AM
I run official Olympic site and it down!!!!
Posted by phiber_9, 03-31-2005, 08:49 AM
How about this theory: A "0-day" cisco exploit has shut down all of their routers. Except Junipers at the border. This would explain why some part of the network is reachable.
Posted by TrevorN, 03-31-2005, 08:49 AM
no one really knows and theplanet has no eta though they think they may have found the problem.
Posted by vidahost, 03-31-2005, 08:49 AM
Doesn't explain *power* loss though
Posted by fire100, 03-31-2005, 08:49 AM
All you would have to do, is point them to this thread for proof.
Posted by rvrsl, 03-31-2005, 08:50 AM
Not sure. Could be installing hardware sniffing devices and requires a cold reboot or shutdown to do so. Could be a bunch of different stuff. But I can tell you that this doesn't seem like a regular outage. People are reporting their servers manually being rebooted, but only some. Nothing is a common pattern to what we would normally see on a regular outage.
Posted by EricP, 03-31-2005, 08:50 AM
Back up...but for how long?
Posted by Sam Bauers, 03-31-2005, 08:50 AM
"I'm up" "I'm down" "I'm up" SHUT UP!!!! I don't care if you are up or down! I only care if I am up or down, and I know that already... Telling everyone your subnet and whether your server is working is not going to help anyone. Nobody here is going to diagnose the problem via your up/down posts. ... and you all wonder why they don't post an announcement here... damn...
Posted by Mix, 03-31-2005, 08:51 AM
Whats the phone number to the planet? One of my servers came back up but I am unable to SSH or load anything.
Posted by PierreB, 03-31-2005, 08:51 AM
Why does everyone think it is a power outtage? Cause I don't think it is one.
Posted by phiber_9, 03-31-2005, 08:51 AM
I don't think it's a power loss. Only large scale router vulnerability could explain this situation, as far as I'm concerned.
Posted by chrisbond, 03-31-2005, 08:51 AM
They've probably got parts for a router on route from somewhere and not telling us =)
Posted by Von Vonder, 03-31-2005, 08:52 AM
All these people wondering about SLA credits. I could easily imagine that they have some type of insurance to cover massive SLA claims. If that is the case they should end up being rather open about offering refunds.
Posted by TrevorN, 03-31-2005, 08:52 AM
the dc webcam still doesn't show anything interesting...
Posted by rvrsl, 03-31-2005, 08:52 AM
Because some people are reporting their servers have been rebooted.
Posted by WebAfrica, 03-31-2005, 08:52 AM
Webhostingtalk DNS was definately down just now...
Posted by Hostmax-br, 03-31-2005, 08:53 AM
yeah yeah... I think it were the martians...
Posted by whmcsguru, 03-31-2005, 08:53 AM
Dream? Not too likely. Now if you can't see the reality of what I posted, I'll be nice and post it again. Firstly, just because YOUR servers had "lost power" doesn't mean that it's not a power issue again. Your servers aren't the only thing affected by power. Routers, gateways, etc, alllll are affected by power. All it takes is for the racks that one of those are on to go nuts, lose power and you're affected. Secondly, there's really no other rational explaination for the massive shutdowns/reboots going on other than a power issue, aside from "hardware failure", and, again, the odds of that happening in 20+ servers, all at close to the same time, not really too good. Like I said, just because your server isn't powered down, yet is unreachable doesn't mean it's NOT a power issue, it just means that your router/gateway/etc isn't available, and that could quite easily be a power issue. If you've got another (rational) explaination of the massive shutdowns, then I'd love to hear it, but right now, power is the most reasonable and unreaching explaination. Far from a dream.
Posted by AKWS, 03-31-2005, 08:53 AM
Yes, it seems they are manually rebooting servers. http://69.93.160.202/~pc20com/uptime.php I assumed it was manual, as improper shutdown should take MUCH longer to get up and going again.
Posted by TrevorN, 03-31-2005, 08:53 AM
thats not good...
Posted by FargoNorthDakota, 03-31-2005, 08:54 AM
Arrgggg I need to get my stuff up - seems like this is not your ordinary outage....
Posted by remote-admin, 03-31-2005, 08:54 AM
hi, same down damn. server come up, several minutes speed was excellent like always, than come packet losses and again down.
Posted by Iggy, 03-31-2005, 08:54 AM
No reboot on leased machines. Colo was rebooted when it came back up and now down again. Panic in the datacenter, maybe? Seems unlikely that the FEDs would touch a few thousand machines at a time. That would, I think, be a difficult warrant to get a judge to sign off on and this would be a spectacularly dumb way to go about it (not that the FEDs aren't ham-handed anyways but this is pretty severe)
Posted by freeflight2, 03-31-2005, 08:54 AM
back up again! 04:53:38 up 80 days, 3:22, 1 user, load average: 0.44, 0.47, 0.33 no power outage on none of our 12 servers
Posted by TRIBOLIS, 03-31-2005, 08:55 AM
Server is up! Everything is fine.
Posted by DCM, 03-31-2005, 08:55 AM
Planet/SM Down Terrorist have taken out the power grid and the night janitor watching the NOC is playing PS2 on the 65" HS widescreen and not telling people the problem when they call. Lawyers are in a chartered plane heading to Dallas to seize the assets of the company for the class action lawsuit. Everyone is pulling their boxes and moving to another DC as it must be better. Oh, some guy is paying an engineer $500 an hour to eat popcorn. My resume is in the mail.
Posted by Mix, 03-31-2005, 08:55 AM
Anyone got the phone number? I need to reboot one of the servers that came back up.
Posted by remote-admin, 03-31-2005, 08:56 AM
looks like ups again
Posted by rvrsl, 03-31-2005, 08:56 AM
Now, I can't possibly think of any scenario for what has been occuring for the last two hours combined with manual reboots. Why would some servers slowly but surely be manually rebooted during network connectivity loss?
Posted by Lem0nHead, 03-31-2005, 08:57 AM
THAT'S ALMOST 3 HOURS NOW!
Posted by PierreB, 03-31-2005, 08:57 AM
+1800 854 7679
Posted by 25GigPak, 03-31-2005, 08:57 AM
1-800-854-7679
Posted by dolay, 03-31-2005, 08:57 AM
there is a 2nd scene for web hosting services here after TP fixed everythings... what we gonna say to the customers? I also bet TP will not refund anything. Maybe %10 if it last for whole day like that... this is the first time that i ever seen a big wide outgate on TP for 2 years but i know the hardware crashs that last more then a day on our servers and never get a refund nor an apologize.
Posted by phiber_9, 03-31-2005, 08:57 AM
and you think a reboot can't be done remotely? They have hardware for this install by default. You just don't have the ability to use it. And if the above is wrong, why would a reboot mean that the power has been lost? Maybe they are rebooting them manually. And maybe they shut down the power on purpose.... (another theory -- OSPF)
Posted by kris1351, 03-31-2005, 08:57 AM
214-782-7800
Posted by TRIBOLIS, 03-31-2005, 08:57 AM
So fast to reply here, in 1 sec.
Posted by AKWS, 03-31-2005, 08:58 AM
PierreB: Exactly. I don't see why it would be a power problem. If it was power, it would be down or fixed by now. As for that theory that breakers are pushing the laod to the next breaker and so on. Within 10seconds the whole datacenter would be black and dead! So the chance its a power problem, is very rare. Could people post actual usefull info. If you say your server is up then give us your IP. If your server was DOWN and now up but was NOT rebooted, then show us your uptime. Only with proper info could we really guess what is going on.
Posted by Hostmax-br, 03-31-2005, 08:58 AM
anybody got a response from TP?
Posted by Lem0nHead, 03-31-2005, 08:58 AM
there is no power... this is just the matrix
Posted by dee_at_candl, 03-31-2005, 08:58 AM
LOL Excellent!
Posted by kmike, 03-31-2005, 08:59 AM
For all non-believers that at least some power problems were there: One server: 06:58:09 up 9 min, 1 user, load average: 0.00, 0.02, 0.01 Another server: 06:58:42 up 9 min, 1 user, load average: 0.00, 0.07, 0.06 One more server: 06:59:10 up 10 min, 1 user, load average: 0.05, 0.03, 0.00 Mind you, this is 3rd time I was able to log in, meaning there were at least 3 reboots.
Posted by ImLagging, 03-31-2005, 08:59 AM
LOL! Why would anyone do that? :p
Posted by Bill-zilonhost, 03-31-2005, 08:59 AM
(800) 854-7679
Posted by rvrsl, 03-31-2005, 08:59 AM
You asked, "Why do people think its a power outage?" and I simply stated why people are reporting that. I didn't say I agree'ed or disagreed. I'm aware of how a reboot can be done. I just told you why people are stating power outage.
Posted by kneadingu, 03-31-2005, 09:00 AM
no they did not reboot.
Posted by Iggy, 03-31-2005, 09:00 AM
69.93.72.* Leased machine - up - no reboot 64.5.48.* Colo - down currently - was able to briefly login and find it was rebooted then down again.
Posted by PierreB, 03-31-2005, 09:01 AM
5 out of 6 server are online, 1 has never come back online yet. So 3Hours, 11mins. the best thing is just to wait and see.
Posted by eugyx, 03-31-2005, 09:02 AM
2 servers back, IPs 69.56 and 69.93, uptime 152 and 123 days, no signs of power problems.
Posted by Iggy, 03-31-2005, 09:02 AM
Good morning East Coast customers! arg.
Posted by DigiWire, 03-31-2005, 09:03 AM
Would they give a compensation if it was something that happened which was out of their control?
Posted by Lem0nHead, 03-31-2005, 09:03 AM
my 69.56 is still down and never came back
Posted by sbdquinten, 03-31-2005, 09:03 AM
3 posts per minute!
Posted by Mix, 03-31-2005, 09:04 AM
My chances of getting a reboot right now are slim to none. When I called you can hear the phones ringing off the hook.
Posted by Yellow@home, 03-31-2005, 09:04 AM
Problems can happen but the worst of it all is the Info we get from TP (not) They should have a info noc page when somthing like this happens. Maby they can rent a dedi with ev1 for this
Posted by WebMate, 03-31-2005, 09:04 AM
I've just called TP and they say it's not a power issue.
Posted by Mix, 03-31-2005, 09:05 AM
Page 50 heh
Posted by Hostmax-br, 03-31-2005, 09:05 AM
TraceRoute to 216.185.111.40 [www.theplanet.com] Hop (ms) (ms) (ms) IP Address Host name 1 0 0 0 66.98.244.1 gphou-66-98-244-1.ev1.net 2 0 0 0 66.98.241.4 gphou-66-98-241-4.ev1.net 3 1 1 1 129.250.10.105 ge-0-1-0-50.r00.hstntx01.us.bb.verio.net 4 6 6 6 129.250.2.50 p16-0-1-3.r20.dllstx09.us.bb.verio.net 5 7 7 7 129.250.31.40 ge-0-0-0.a00.dllstx09.us.ra.verio.net 6 6 7 7 129.250.28.186 ge-1-2.a00.dllstx04.us.ra.verio.net 7 6 6 6 157.238.228.38 ge-9-3.a00.dllstx04.us.ce.verio.net 8 5 5 6 70.85.127.29 dist-vlan31.dsr3-1.dllstx3.theplanet.com 9 6 6 6 70.85.127.67 dist-vlan21.dsr1-1.dllstx2.theplanet.com 10 6 6 6 12.96.160.11 car1-1-v1.dllstx2.theplanet.com 11 Timed out Timed out Timed out -
Posted by zsindhu, 03-31-2005, 09:05 AM
ouch!
Posted by fluxhost, 03-31-2005, 09:06 AM
If it was a power issue, they couldn't answer the phone
Posted by whmcsguru, 03-31-2005, 09:06 AM
In many cases, you'll find some disclaimers in their SLA like "We're not responsible for router issues, etc". Not sure, as I haven't read through TP's SLA in a while (and can't, because their site's not accessible), but most likely they have something like this in there, yes.
Posted by ServerZB, 03-31-2005, 09:06 AM
best place for forum newbies to accumulate post numbers
Posted by BoardHoster, 03-31-2005, 09:06 AM
A couple of others beat me to it, but how do you know if they are doing hard reboots [turning power off] or software reboots? .
Posted by Lem0nHead, 03-31-2005, 09:07 AM
they should they don't say anything about "if it out of our control" in their SLA
Posted by princo, 03-31-2005, 09:07 AM
My server is on and off couple of times. So far 3 reboots, since these are power cycles, I'm thinking of shuting down the server completly while power is restored normally. What do you think?
Posted by Oleanhst, 03-31-2005, 09:07 AM
You got that right
Posted by TrevorN, 03-31-2005, 09:07 AM
what i think is really weird is through out this whole thing the webcam has been up and running... haha but nothing interesting is happening on it.
Posted by eadz, 03-31-2005, 09:07 AM
So did you ask what kind of issue it is?
Posted by 1980, 03-31-2005, 09:07 AM
or for people who forgot their password and their email address is hosted on a TP server :p
Posted by Rituparna, 03-31-2005, 09:07 AM
======================================== I've just called TP and they say it's not a power issue. ========================================= Have you got any further info. Douglas Windows Expert
Posted by BlackMail, 03-31-2005, 09:08 AM
^^^^ best example to do the same
Posted by Gentle Giant, 03-31-2005, 09:08 AM
Just been told on the phone that the sla might not come into affect as the servers remained up!!!
Posted by ALGORYTHM, 03-31-2005, 09:08 AM
it's a valid point, the servers are most likely unaffected.
Posted by Lem0nHead, 03-31-2005, 09:09 AM
phone works with no power the phone line already have it at least i've used phone (of course not wireless) with no power
Posted by MFrag, 03-31-2005, 09:09 AM
My servers are running and I can get to them by IP, but it does appear their DNS servers have been having trouble since yesterday 3/30. They came back up or resolved and went out at 02:00CST so it appears from my mailqueue logs, which show it unable to resolve my SMTP mail host since then. It is still down, and it is now 8:00am. DNS outage that long! I can't even get to their portal to put in a ticket! My other server is at EV1. I have never had a problem like this from EV1. But then again my Dual 2.8 P4 at Theplanet is $200/mo, and my 2.4Ghs Dual at EV1 are $250/mo, they now go for $299/mo. So it appears you get what you pay for. They just went through adding a data center, and that did not go very cleanly either. I filed several tickets during an 8 day period and asked for a refund. They were regrooming fiber, yet their cust-svc techs had no idea when I said I had pings going from 70ms to 450ms for 8 days. All in all they are good, but you should at least not ignore the customer and act like there is no problem as they did with the poor execution of the data facility add. At a Network Engineer, with 18 years designing networks for Wall Street, and a Ph.D in EE, I can say if my guys executed that poorly, they would be unemployed. If it were my plan, and it was executed perfectly, I should be fired. A local DNS outage this long.... someone does not know what they were doing, or they were making major changes to their DNS (hum... yes that follows directly the addition of a new data center....), and they muffed it up. I don't know how they pay in Texas, but it would seem they don't have the best people on the job there IMHO. I hope once they get over this major bump (Which like I said, it would be nice if I received a notification as I do from EV1. EV1 if they have a notification from a carrier that their loops may be impacted even though my server fails over to an alternate loop, will take a hit, they will send out a notice. I never received a notification from Theplanet. I don't even think they will even ack they had a problem. It is like the CIA. Hope they resolve their problem and come clean. That would be a very nice change. (Admitting they had a problem). As for 100% uptime, I keep track (Hey, I am a network engineer), without this outage, best was 99.992 for one month. No one can say 100%, or 5, 9's... 99.999, but they came close one month... MFrag
Posted by whmcsguru, 03-31-2005, 09:09 AM
Again, not necessarily. JUST because it's a power issue doesn't mean that ALL circuits are gone, but it does mean that certain circuits are. Ever seen a blown circuit? The rest of your house (or building, etc) will be up and with power just fine, but the area affected by that blown circuit will not until said circuit is replaced or reset. I wouldn't put any faith in their tech support saying it's "not a power issue" at all. Too many servers have been rebooted (or shutdown or whatever) multiple times for this to be a mere coincidence.
Posted by Mix, 03-31-2005, 09:09 AM
Oh yes it will!
Posted by PierreB, 03-31-2005, 09:09 AM
that's about 15 people said on this thread, I wonder if people bother reading.
Posted by WebMate, 03-31-2005, 09:09 AM
Well, their phone system could be on different circuites (should be), but that still doesn't explain the short uptimes people have reported.
Posted by Cahl, 03-31-2005, 09:09 AM
"This is not and has not been a power issue. It's a routing issue, Mr Cahl, and I assure you that our engineers are working on restoring full connectivity as soon as possible." R'gds Dean
Posted by Yellow@home, 03-31-2005, 09:09 AM
LOL!!!!!!!! Thieves
Posted by v0ltair3, 03-31-2005, 09:10 AM
Because in win2k3 you get a unxpected shutdown logged. Normal soft shutdowns are logged as normal and require a reason. Hard Shutdowns are logged as unxpected and when you log back to the server you get a pupup requesting a reason for the hard shutdown
Posted by BlackMail, 03-31-2005, 09:10 AM
Luckily there is a thing called raw logs which prove more than enough if you run sites with thousands of uniques per minute. Also it would be very bad for them to fight about SLA as they would lose a lot of customers. We only have 11 servers - but there are more people than only me
Posted by eadz, 03-31-2005, 09:10 AM

Posted by jaume, 03-31-2005, 09:10 AM
But is a fact that many servers are being rebooted. I could login to 5 and 2 of them were rebooted.
Posted by zsindhu, 03-31-2005, 09:10 AM
are you serious? anyway, right now, I don't give a hoot about the SLA, just as long as they make my phone stop ringing very VERY soon!
Posted by coolio, 03-31-2005, 09:11 AM
no reboots here: 2 up (were down several times earlier): 15:09:34 up 75 days, 5:01, 1 user, load average: 0.00, 0.00, 0.00 15:10:02 up 18 days, 4:14, 1 user, load average: 0.74, 0.50, 0.35 1down
Posted by Lem0nHead, 03-31-2005, 09:11 AM
OH MY SERVER IS BACK AND HAS NO DATA AT ALL!! (just putting more action here)
Posted by kris1351, 03-31-2005, 09:11 AM
They have a 100% guaranteed SLA on network coverage. This has been intermittent downtime on the network and even some reboots of servers. It will be covered, you will just have to put in your documentation. They only require proof of the times it went down and when it was back up. SM/TP has always been fair with SLA compensation.
Posted by eugyx, 03-31-2005, 09:12 AM
I hope that doesn't reflect the way they will deal with this problem. From my point of view, I am not paying for a server up, but for a service that includes the server and the connectivity, so, they should be responsible for both (of course, if i mess with the server, is my problem)
Posted by Von Vonder, 03-31-2005, 09:12 AM
That is not correct. The SLA covers connectivity and internal routing.
Posted by Karl Austin, 03-31-2005, 09:12 AM
The reboots could be because they are looking for an item of equipment that is cuasing circuits to blow - They turn off some kit, reset the circuit, if it blows, turn the other servers back on, turn off the next batch of kits, rinse, repeat.
Posted by coolio, 03-31-2005, 09:13 AM
or like Burst does/did ... on a aol.com userspage. LOL
Posted by phiber_9, 03-31-2005, 09:13 AM
which brings us closer to my scenario....
Posted by AKWS, 03-31-2005, 09:13 AM
I think we all should ban together and demand more cams. Then we could at least see them sitting on their a$$ eating donuts. No really... They do need more than 1 cam crammed in the corner of the DC. I don't know about you, but being able to just see my server makes me feel much better.
Posted by bheka, 03-31-2005, 09:13 AM
Up and down like a yoyo ... this is very very bad. I took a server at the planet to avoid exactly this .. paying a bit extra and this is really not on.
Posted by Lem0nHead, 03-31-2005, 09:13 AM
nice, nice 3 hours down no official communicate or anything luckly i've backup of my customers mails on other server, so I sent them a communicate
Posted by WebMate, 03-31-2005, 09:13 AM
A sticky post from TP would be a wise thing to do. I did ask them, but I could hear phones ringing in the background, so it's understandable.
Posted by Gually, 03-31-2005, 09:13 AM

Posted by universal2001, 03-31-2005, 09:13 AM
It's just an early April fools joke.
Posted by cyberhost, 03-31-2005, 09:13 AM
This is very strange. Some people report multiple reboots, others no reboots, TP claims its not a power problem. Maybe a single fault is causing chaotic behaviour. livejournal went down weeks ago after somebody pressed a button to cut power in case of fire. it. It took them days to recover when they discovered their RAID cards weren't doing synchronous writes & their database was corrupted.
Posted by cjhhiv, 03-31-2005, 09:15 AM
THEY'RE BACK!!!!!!!!!!!!!!!!!!!!!
Posted by zsindhu, 03-31-2005, 09:15 AM
LOL!
Posted by sieb, 03-31-2005, 09:15 AM
67.18.*.* still down
Posted by Lem0nHead, 03-31-2005, 09:15 AM
but that's something specific to one server even if they have some power problem, most servers will be back when everything is fine but a few others may have problems
Posted by Gually, 03-31-2005, 09:16 AM
ja ja ja
Posted by andrewgmol, 03-31-2005, 09:16 AM
NO THEY'RE NOT!!
Posted by rvrsl, 03-31-2005, 09:16 AM
Or maybe it's a federal law enforcement raid/serving and they're doing a massive checking of servers, looking for something specific. Some servers require reboot for local user login to root.
Posted by NetSender, 03-31-2005, 09:16 AM
Not back. Grrrrrr
Posted by PierreB, 03-31-2005, 09:17 AM
I confirm, 67.18 still down for me as well 3h26mins.
Posted by cjhhiv, 03-31-2005, 09:17 AM
$#&%^*&^(%^#(*&#$@#$ Okay, I saw their site for 10 seconds. Now they are not back :-(
Posted by v0ltair3, 03-31-2005, 09:17 AM
69.93.228 are still down Please everyone, put your network ID when you refer to your machines to see if we can get an ideia of what networks are having more problems.
Posted by MFrag, 03-31-2005, 09:17 AM
I verified a fellow server owner of a "totalcontrol" server can reach his server, as I can. Others with Totalcontrol servers that can get to them?
Posted by Yellow@home, 03-31-2005, 09:17 AM
What did the lost then????? A dime in a server?????
Posted by coolio, 03-31-2005, 09:17 AM
The network graphs in orbit will show enough I think
Posted by Lem0nHead, 03-31-2005, 09:18 AM
good luck for them if they're looking for something on 20,000 servers usually if some federal law enforcement happens, they already know the IP of the server that may have problems... or at least a range... they wouldn't need to take down all the network... including theplanet and SM sites
Posted by BoardHoster, 03-31-2005, 09:18 AM
The PBX's and Key systems I have een involved with are usually on battery back-ups and will run for several hours with no power anywhere in the building. I did one in a US Courthouse that would run nearly 24 hours with no AC power. .
Posted by jaume, 03-31-2005, 09:18 AM
some of my servers have been rebooted: admin pts/0 62-43-7-13.user Thu Mar 31 14:17 still logged in reboot system boot 2.4.21-27.0.2.EL Thu Mar 31 14:03 (00:14) admin pts/0 62-43-7-13.user Thu Mar 31 13:27 - 13:28 (00:00) reboot system boot 2.4.21-27.0.2.EL Thu Mar 31 13:24 (00:52) reboot system boot 2.4.21-27.0.2.EL Thu Mar 31 13:00 (01:16)
Posted by AKWS, 03-31-2005, 09:18 AM
Most business phones are run through PBX's which require power.
Posted by Gually, 03-31-2005, 09:18 AM
v0ltair3: 69.93.146.x is still down and hasn't been up since the frist time
Posted by net, 03-31-2005, 09:18 AM
From LT: So don't relax yet :-(
Posted by remote-admin, 03-31-2005, 09:18 AM
LOL maybe TP think that today 1st april and cutoff power in planet ? LOL
Posted by dkloke, 03-31-2005, 09:18 AM
yeah .. and they have the webcam looping an old clip to cover it up.. i keed. i think.
Posted by PierreB, 03-31-2005, 09:18 AM
Our totalcontrol servers are up, never gone down.
Posted by nothingg, 03-31-2005, 09:18 AM
any ideas where do i get those raw logs ?
Posted by Whocares, 03-31-2005, 09:19 AM
Interesting my all servers are up
Posted by Lem0nHead, 03-31-2005, 09:19 AM
oh i hope they are not doing that one by one... that would take 2 days
Posted by pstation, 03-31-2005, 09:20 AM
I can verify that, my TC server has had no downtime at all during this. The problem dosent seem to exist in DC4
Posted by rvrsl, 03-31-2005, 09:20 AM
Not in the Foonet raid. They couldn't figure out which server so they just closed the whole DC and took 350 servers. While TP is obviously larger, my scenario would be probable for the behaviour of this "outage".
Posted by richyc, 03-31-2005, 09:20 AM
If it's a "Federal Law sweep", I doubt they'll shut down so many servers simulatenously, shut down the SM/TP websites (and orbit) and still allow staff to answer phones... MatrixExtreme 70.84.33.* still down SM 69.93.83.* back up TP 70.85.104.* backup All our other servers at SM ok
Posted by MFrag, 03-31-2005, 09:22 AM
Yes, mine as well, but I have no DNS outbound. So if you look at your outbound email log, bet it is not resolving..... MFrag
Posted by 25GigPak, 03-31-2005, 09:22 AM
Do you have the URL to the web cam?
Posted by zsindhu, 03-31-2005, 09:22 AM
Same here. TotalControl servers never went down.
Posted by accyroy, 03-31-2005, 09:23 AM
http://69.41.224.20/popup.html
Posted by TrevorN, 03-31-2005, 09:23 AM
http://69.41.224.20/popup.html still working too lol
Posted by misnomer, 03-31-2005, 09:23 AM
Install a local dns caching server - my mx backup at DC4 has been unaffected by this outage, while my mx backup at DC2 has been down the whole time. I also have a TC server which has been up and down a few times (network availability wise, not reboots).
Posted by eadz, 03-31-2005, 09:23 AM
LT?
Posted by EndOffice LLC, 03-31-2005, 09:24 AM
Seems like there was some sort of power issue. I just logged into my server and look at this: atom ttyp0 x.x.x.x. Thu Mar 31 08:24 still logged in reboot ~ Thu Mar 31 08:10 reboot ~ Thu Mar 31 07:57 tresuiri ttyp0 x.x.x.x Thu Mar 31 07:31 - crash (00:25) reboot ~ Thu Mar 31 07:29 reboot ~ Thu Mar 31 07:03 That SUCKS! I don't want my box to be flapping up and down.
Posted by [x10]Corey, 03-31-2005, 09:24 AM
My total control went down for a bit at 70.85.87.* . 2 other servermatrix were up and down, and one servermatrix hasn't been up at all. 69.93.194.*
Posted by tamouh, 03-31-2005, 09:25 AM
I noticed couple of things: 1) Our servers were NOT powered off 2) Some locations were still able to access our server, and others were not, so definitely routing problem 3) The only downtime solid from all locations I found was between 7 am - 7:38 am
Posted by SHTech, 03-31-2005, 09:25 AM
We have servers at 67.18.82.* and 67.18.89.* which never came up. Others randomly up & down, but the time they up there is no reboots on them. Very frustrating...
Posted by MFrag, 03-31-2005, 09:25 AM
Gents, they just added a new datacenter. Timing is interesting isn't it? Even for my total control server there, it was more bumpy then they let on. I can deal with the outage, they never fessed up to the issues while adding the data center. They can't glaze over this one.... When I see things like this during major additions, the flag goes up in the "Human error" in poor engineering category.... Likely designed by all CCIE's from Mumbai...
Posted by PierreB, 03-31-2005, 09:25 AM
LT = layeredtech
Posted by forumtalk, 03-31-2005, 09:25 AM
69.93.160.* completely 2 hours down
Posted by eugyx, 03-31-2005, 09:25 AM
LT? [/B][/QUOTE] LT = Layered Technologies.. Planet reseller
Posted by Iggy, 03-31-2005, 09:26 AM
64.5.48.* up - rebooted 3 times 3:58PST, 4:26, 5:10. What is up with that? Leased machines at ServerMatrix not rebooted at all.
Posted by SHTech, 03-31-2005, 09:28 AM
Servers which are up at least, but up & down during the initial problem: 69.93.* 69.56.* I wonder why there is still no, at least short, news posted from TP?!
Posted by cyberhost, 03-31-2005, 09:28 AM
Could be a very expensive border router thats gone gaga (I learned that term in my CISCO exams). It could be difficult to replace a BGP router quickly.
Posted by tinnitus, 03-31-2005, 09:28 AM
my server at 69.93.32.* is down for 3+ hours. server at 69.56.211.* is now reachable. orbit.theplanet.com, www.servermatrix.com etc. all unreachable from here.
Posted by hosttone, 03-31-2005, 09:29 AM
our servers 3 hours down.
Posted by v0ltair3, 03-31-2005, 09:29 AM
My servers are TotalControl P2800sr5x and are down. 69.93.228.*
Posted by Yellow@home, 03-31-2005, 09:30 AM
Or the dont know what to post Or the dont have a place to post it. Pick one!
Posted by bugsoft, 03-31-2005, 09:31 AM
69.93.* down 67.18.* down 64.5.* down
Posted by elmister, 03-31-2005, 09:31 AM
69.56.147 3h 20 min down, never came up 69.56 down, came up about an hour ago, wasn't rebooted, went down again 67.19.226 had no problems 70.84.50 had no problems 67.19.213 matrixextreme, down
Posted by Ramprage, 03-31-2005, 09:32 AM
This doesn't look good I can't even connect to ORBIT! http://orbit.theplanet.com/ --< DEAD My server has been offline for the last 8 hours. Any word from server matrix about what is happening?
Posted by Mani-Web, 03-31-2005, 09:32 AM
I beleive they prefer them to stay unreachable! They may not want to answer handreds of questions about this outage right now... Too bad
Posted by coolio, 03-31-2005, 09:32 AM
I'll create them a free hosting account on my working TP server ;p
Posted by Yellow@home, 03-31-2005, 09:33 AM
LOL
Posted by M7I, 03-31-2005, 09:33 AM
One of our servers that was down the entire period just came back up and has been rebooted. 69.93
Posted by HighLineHost, 03-31-2005, 09:33 AM
Wow are they still down?
Posted by blueface, 03-31-2005, 09:33 AM
I can't access theplanet.com from here. How long has it been down?
Posted by MFrag, 03-31-2005, 09:34 AM
Total control in DC4 reboot system boot 2.6.9-5.0.3.ELsm Mon Mar 28 10:20 (2+21:12) reboot system boot 2.6.9-5.ELsmp Sun Mar 27 14:52 (00:09) wtmp begins Sun Mar 27 14:52:03 2005 [root@trv01 ]# uptime 07:32:57 up 2 days, 21:12, 2 users, load average: 0.15, 0.14, 0.10
Posted by jmweb, 03-31-2005, 09:34 AM
Don't think theres anything official however they will be releasing a statement once their PR guys get in. Remember its only 7:33 there
Posted by Tom Pyles, 03-31-2005, 09:34 AM
Our servers there are still down.
Posted by snJohn, 03-31-2005, 09:35 AM
about three hours
Posted by FargoNorthDakota, 03-31-2005, 09:35 AM
and counting..
Posted by HighLineHost, 03-31-2005, 09:35 AM
this is some amazing downtime.... I would have never though a place like ThePlanet would ever have a problem like this.. So happy non of our servers are there..... no offence to the people who do
Posted by v0ltair3, 03-31-2005, 09:35 AM
Are there any other cams besides http://69.41.224.20/popup.html My servers are not desktop boxes. Mine are rack mount.
Posted by InsideHosts.com, 03-31-2005, 09:36 AM
3 hours 30 minutes.
Posted by fire100, 03-31-2005, 09:36 AM
Since shortly after 5 a.m. EST.
Posted by djdiz-e, 03-31-2005, 09:36 AM
i just moved to TP from velocity <--sucks and now i get this nice welcome to TP
Posted by MFrag, 03-31-2005, 09:37 AM
My outage of local DNS started at 02:00CST... so I think that would be more likely the time they "started" to show signs of trouble?
Posted by AKWS, 03-31-2005, 09:38 AM
I don't think they have other cameras. Also yrou rack servers are most likely at a different Data center I would guess.
Posted by kris1351, 03-31-2005, 09:38 AM
I called when I saw the first pages and the tech said they are having technical difficulties. When I asked for an official start time he said 4:10am CST so not sure where you are getting the 8 hours thing from. You sound like one of my customers.
Posted by SHTech, 03-31-2005, 09:38 AM
theplanet.com resolve to 216.185.111.40 However it is not pingable. ns1.theplanet.com. 165825 IN A 216.234.234.30 -> DEAD ns2.theplanet.com. 165825 IN A 12.96.160.115 -> PINGABLE
Posted by elmister, 03-31-2005, 09:38 AM
Thread was started at 10:15 AM, your message was at 1:35PM, so 'at least' 3h 20 min, asume 10-15 minutes before the thread was started. This is a very long downtime
Posted by v0ltair3, 03-31-2005, 09:38 AM
3 1/2 hours now for servers on 69.93.228.* Have been hard rebooted at least twice.
Posted by kris1351, 03-31-2005, 09:39 AM
4:16am CST was the first page I received from our off-site monitoring. Going on 4 hours of up and down here, but only on one server thankfully. The others have been running fine.
Posted by Ramprage, 03-31-2005, 09:40 AM
That makes me feel all warm and fuzzy inside. So their network remains down because they're still in bed heh. I know they're working on it but 8 hours of downtime. I hope they have a good explanation - like a snowstorm inside their DC!
Posted by PierreB, 03-31-2005, 09:40 AM
3.50CSt was the first time I noticed the problem, we are nearly at 4 hours.
Posted by Mach3, 03-31-2005, 09:40 AM
My server seems to be down too, im just hoping this isnt a virus or anything...judging by the webcam new or old you can see someone's head and a computer screen in the corner, so atleast someone is working on it. I would expect people ot be running around like crazy on it though...
Posted by snJohn, 03-31-2005, 09:41 AM
it's not just mine then
Posted by HighLineHost, 03-31-2005, 09:41 AM
Guys, It really sounds like they may in fact be moving things around, but the odd part is I would have though they would of sent out some kind of email to everyone about this. Who knows what the real problem is , but for sure, whatever admin was on they should fire them, because 3 hours is way to long for a datacenter to go down... BILL - PLUG THAT CAT5 BACK IN!!
Posted by phenx, 03-31-2005, 09:41 AM
Orbit is back up. Though, i dont think it liked everyone trying to login at once
Posted by Valve Ben, 03-31-2005, 09:42 AM
They used to have another cam at: http://dcwebcam01-dllstx2.theplanet.com/popup.html It's not responding though...
Posted by jeroman, 03-31-2005, 09:43 AM
I can't access Orbit !
Posted by NetSender, 03-31-2005, 09:43 AM
Actually yes it is working Valve Ben, you can see some black boxes, and a lot of bright light, hmmm.
Posted by Lem0nHead, 03-31-2005, 09:45 AM
i get ping response from orbit, but can't login
Posted by jedrek, 03-31-2005, 09:46 AM
my both my LT boxes have been up the whole time.
Posted by lessthanthree, 03-31-2005, 09:46 AM
both my servers are up right now, have been for awhile now one had 30 minutes down time other had 149 minutes neither had been rebooted.
Posted by coolio, 03-31-2005, 09:46 AM
Let's hope many maxitreme people will leave now ... that'll lower the load on the cogent lines
Posted by Ramprage, 03-31-2005, 09:46 AM
You know what makes me laugh about this entire situation. The fact that they promote having a REDUNDANT network. Well that sure is helping everyone right now....... I'm starting to think they're just using a bunch of cable modems together with one provider. Or maybe the tech is a Hommer and spilt coffee on the main control center and fried the network He forgot to push the big "Press the any key if the network is down" - The Tech is frantic, "WHERE IS THE ANY KEY?!" My server status monitor starter alerting me of server goiong on/off since midnight EST. It's now 8:30 AM... so 8 hours of network issues from the start of this.. it doesn't mean my servers have been completely off for 8 hours but that's when this whole thing seemed to start.
Posted by TrevorN, 03-31-2005, 09:47 AM
they added 2 gigs a week ago or so, so that isn't an issue anymore
Posted by NetSender, 03-31-2005, 09:47 AM
I just saw some one fly through the picture of that webcam, hehe.
Posted by Von Vonder, 03-31-2005, 09:48 AM
I just had a server not cleanly rebooted in dllstx4. Some minor mysql corruption is the easy way to know.
Posted by Legal Eagle, 03-31-2005, 09:48 AM
At approximately 4:15AM CST, a pair of redundant Powerware 500KVA UPS units failed creating a power failure in section B of our DLLSTX2 datacenter. Emergency teams were deployed within minutes and power was restored within minutes but intermmittment power outages did occur until 6:45AM CST. Powerware, JT Packard, and electricians are currently onsite with over 100 Planet technicians working to resolve the issue. We do not anticipate any further outages . A formal RFO will be released once the team debriefs. We apologize for all issues that has caused.
Posted by Karl Austin, 03-31-2005, 09:49 AM
I love this thread, so many people insulting TP and their staff, yet I bet half that are doing it wouldn't know BGP from CAT5 nevermind running a large scale network. Problems happen, sure it's crap when they do, but not matter what anyone does, you will never ever have 100% uptime indefinately.
Posted by zsindhu, 03-31-2005, 09:49 AM
That's not funny, mate.
Posted by Lem0nHead, 03-31-2005, 09:49 AM
thanks for the official information we were needing that but my server is still down... some ETA?
Posted by chrisbond, 03-31-2005, 09:50 AM
hehe - The Support Site is not currently available. Please try again later.
Posted by Darkanoid, 03-31-2005, 09:50 AM
Strange my server is still down
Posted by ALGORYTHM, 03-31-2005, 09:50 AM
yay thanks for the info LE
Posted by Erwin, 03-31-2005, 09:51 AM
Thanks for the info. Hope everything will be okay now.
Posted by galantina, 03-31-2005, 09:51 AM
>>Strange my server is still down whay is that strange?
Posted by paulstest, 03-31-2005, 09:51 AM
Yeah, my server is still down too... so that kinda flies in the face of that statement, can resolve but not reach... P
Posted by TS-Dave, 03-31-2005, 09:51 AM
If it was a power outage; Why are the servers still reporting 50+ days of uptime? Is this the power grid that all of the routers and switches sat on?
Posted by coight, 03-31-2005, 09:51 AM
2 of ours are still down. 24 hour period: Device Outages Uptime Downtime Other Uptime% EART 1 20:32:44 03:27:16 00:00:00 85.606% URAN 1 20:25:22 03:34:38 00:00:00 85.095%
Posted by Tom Pyles, 03-31-2005, 09:51 AM
I see A LOT of people moving around the datacenter. Don't think I've ever seen it that busy. Lots of suits too.
Posted by Von Vonder, 03-31-2005, 09:52 AM
How could a power outage cause downtime in the other datacenters and reboots of servers in the other datacenters?
Posted by UH-Matt, 03-31-2005, 09:52 AM
Dont you think you should be saying that once the problem is FIXED? Still a lot / most people down.
Posted by richyc, 03-31-2005, 09:52 AM
Our Matrixtreme server is still down and Orbit is still unavailable (so is http://www.servermatrix.com/ )...
Posted by zsindhu, 03-31-2005, 09:52 AM
Thank you for the post, Legal Eagle. Any estimate on when the servers would be back online?
Posted by Ramprage, 03-31-2005, 09:53 AM
Wow- 100 technicians, that's alot of caffeine and donuts. Appreciate the update, it will help my customers from having heart attacks
Posted by v0ltair3, 03-31-2005, 09:53 AM
Where did you see that ? Is that another webcam ?
Posted by bugsoft, 03-31-2005, 09:53 AM
10 servers still down!
Posted by Mach3, 03-31-2005, 09:53 AM
so is this saying the power cut off and too many servers tried to boot at the same time causing network porblems or something? What about backup power?
Posted by M7I, 03-31-2005, 09:53 AM
We had 4 servers total down. 3 were up and down a few times. One was down the entire time until about 20 minute ago. It is now up and was rebooted. The other 3 were never rebooted. 69.93.216 is the ip range of the one that was booted.
Posted by photodesign, 03-31-2005, 09:53 AM
Strange, my server too is still down.
Posted by UH-Matt, 03-31-2005, 09:54 AM
I think that explanation is terrible. You have obviously not planned your power setup very well. The failure of redundant units has taken down thousands of servers for many hours? Jeez go have a look at EV1 and learn something.
Posted by SHTech, 03-31-2005, 09:54 AM
Nice to hear semi-official explanation... It may bring us a point as they should solve issues soon... Unfortunately, 67.18.* is still offline. Hope that they not burned them! LOL!
Posted by dkloke, 03-31-2005, 09:54 AM
thanks!! where is that? theplanet.com is still down (from here). i'd like to pass the link to some clients.. ty
Posted by TrevorN, 03-31-2005, 09:54 AM
is it just me or are some servers missing in the webcam?
Posted by Ramprage, 03-31-2005, 09:55 AM
What webcams are you watching, share any webcam URLS please.
Posted by Legal Eagle, 03-31-2005, 09:55 AM
Technicians are currenlty on the floor putting consoles on all boxes. Power is up and everything is normal (fixed).
Posted by kaosweaver, 03-31-2005, 09:55 AM
69.93.186.* Down (still)
Posted by chrisbond, 03-31-2005, 09:55 AM
Tom Pyles - what webcam you using?
Posted by silentnight, 03-31-2005, 09:55 AM
Hi, TP staff, TP's Shared SQL Server is still down, can you help??? IP: 12.96.161.*
Posted by Tom Pyles, 03-31-2005, 09:55 AM
http://dcwebcam01-dllstx2.theplanet.com/popup.html Although the activity has died down, perhaps making their way to another part of the building. EDITED: Now I'm only see a person here and there. When I first posted, there was a flurry of people hustling about.
Posted by djdiz-e, 03-31-2005, 09:55 AM
witch cam you guys watching i see nothing on this one
Posted by v0ltair3, 03-31-2005, 09:56 AM
69.93.228.* are still down, almost 4 hours and have been hard rebooted.
Posted by NetSender, 03-31-2005, 09:56 AM
http://www.geocities.com/netsendercouk/tpc.jpg Look he is doing something about it! Well its a blur of speed so I can only guess he's nicking it or something. hehe!
Posted by daveborg, 03-31-2005, 09:56 AM
My nagios still has a LOT of red
Posted by sirius, 03-31-2005, 09:56 AM
Very strange... I did not lose a single box during this entire outage. My MatrixExtreme boxes in D2 never lost power. I assume I was in the "lucky" area of the data center. Sirius
Posted by bugsoft, 03-31-2005, 09:57 AM
http://dcwebcam01-dllstx2.theplanet.com/popup.html is down for me What's the IP's ?
Posted by AKWS, 03-31-2005, 09:58 AM
Exactly! I have been through many hosts in the past that were much worse. Its a big business running a data center.
Posted by Darkanoid, 03-31-2005, 09:58 AM
69.93.181.* still down
Posted by djdiz-e, 03-31-2005, 09:58 AM
down for me to
Posted by NetSender, 03-31-2005, 09:58 AM
Pinging dcwebcam01.dllstx2.theplanet.com [64.5.44.170] with 32 bytes of data: Reply from 64.5.44.170: bytes=32 time=135ms TTL=44 Reply from 64.5.44.170: bytes=32 time=134ms TTL=44
Posted by Laws, 03-31-2005, 09:58 AM
How can people be seeing webcams and other can't, surely there IS still a network issue somewhere?
Posted by Oleanhst, 03-31-2005, 09:59 AM
209.120.***.** still down
Posted by hosttone, 03-31-2005, 09:59 AM
our server still down.
Posted by photodesign, 03-31-2005, 09:59 AM
3 Servers still down. Not fixed and normal at all.
Posted by bugsoft, 03-31-2005, 09:59 AM
http://64.5.44.170/popup.html
Posted by SHTech, 03-31-2005, 09:59 AM
One of our nagios monitors downed this time.... 67.18.* sub. Another at EV1 & DNOW brings us the red all over the place for 67.18.* servers... Customers are calling, mailing.... Oh my God!
Posted by Nick Jones, 03-31-2005, 09:59 AM
LOL, exceeded bandwidth
Posted by jbrw, 03-31-2005, 10:00 AM
I wouldn't say normal - i'm seeing very high packet loss. Pinging the Level3 box one hop away from TP results in 0% packet loss. ...j
Posted by TrevorN, 03-31-2005, 10:00 AM
i can't unless i use a ip instead of a url http://64.5.44.170/popup.html http://69.41.224.20/popup.html
Posted by UH-Matt, 03-31-2005, 10:00 AM
I would not say things are fixed at all. 69.93.* still down as are your own sites.
Posted by BoardHoster, 03-31-2005, 10:01 AM
Not too many webcams can take hundreds of concurrent users..... .
Posted by MatrixAu, 03-31-2005, 10:01 AM
Lol, that was fast, upload the image to http://putfile.com/ (it's at ThePlanet but seems to be up).
Posted by jeroman, 03-31-2005, 10:01 AM
This is like a live chat :-) Just the girls missing it seems
Posted by Mani-Web, 03-31-2005, 10:01 AM
Thank you very much Legal Eagle for all the updates.
Posted by peterh, 03-31-2005, 10:01 AM
67.18.49.* down as well
Posted by PierreB, 03-31-2005, 10:02 AM
that's when you have to show your real skills in communication, talk to your customer, tell them what's up
Posted by coight, 03-31-2005, 10:02 AM
Correct
Posted by chrisbond, 03-31-2005, 10:02 AM
HE.NET links still seem down too - maybe they might have a look at that soon.
Posted by cyberhost, 03-31-2005, 10:02 AM
It very emarrassing to say power was restored 75 mins ago and www.theplanet.com is still offline. By the way its April fools day here in Australia.
Posted by silentnight, 03-31-2005, 10:02 AM
It Is NOT normal the Shared SQL Server still could not be connected!!!!!!!!!!!!!!!!
Posted by DCM, 03-31-2005, 10:02 AM
69.56*.* down
Posted by sirius, 03-31-2005, 10:03 AM
Yeah, I'd agree... because I'd want them working on their servers before they get mine back up. The ignorance in this thread is PROFOUND and SCARY! Sirius
Posted by alexrobonlin, 03-31-2005, 10:03 AM
Which is why it's pretty bloody important to get it right! When it's down, people get pissed off... Pretty simple really. Al
Posted by paulstest, 03-31-2005, 10:03 AM
What was that file? Mail me - Paul at MoDaCo dot com and i'll mirror it. P
Posted by sightz, 03-31-2005, 10:03 AM
69.93.232.* still down But it has been responding to pings for over an hour now - weird! WHY would it do that?
Posted by MatrixAu, 03-31-2005, 10:03 AM
EAST coast of Australia it is... It is here too though, I'm in Sydney. .
Posted by PageUp2, 03-31-2005, 10:04 AM
My server is now up.
Posted by andrewgmol, 03-31-2005, 10:04 AM
69.93.17x.x still down. No pings. No nothing.
Posted by Darkanoid, 03-31-2005, 10:05 AM
IP?
Posted by Lem0nHead, 03-31-2005, 10:05 AM
which mask (IP)?
Posted by SHTech, 03-31-2005, 10:05 AM
We have created a page for them, posting news related to our servers Hope to end (successfully) this page soon We do reply everyone and guess they will understood us.
Posted by binary1, 03-31-2005, 10:05 AM
Well, you win some, you lose some. Yesterday, I caused a problem (oops, long story) with a server hosted at SM. So I had about 1 hr total downtime on that server. The Planet helped me out via phone and got the issue resolved, and got me out of hot water. LUCKY for me, that server is still accessible today and apparently isn't affected by today's Planet outage. That server is in dllstx2. I have a 2nd server at the Planet that's also up. It's in dllstx3. Yesterday, I also ordered 3 more servers, and they were provisioned overnight but I haven't been able to log in to Orbit to get the root passwords, so I don't know if they are reachable. So I guess I lucked out. My uptime with Planet was so good that when I fubared my server yesterday, I had several people call me who were actually confused about what was going on -- they had NEVER had to deal with such an outage! That's quite a testimonial.... OK, I'll shut up.. Hang in there people.
Posted by paulstest, 03-31-2005, 10:06 AM
69.56.146.x still dead... P
Posted by Yellow@home, 03-31-2005, 10:06 AM
Was it a power outage Or......... Some crazy staff member with mental problems. http://www.surfnetusa.com/pinelli/pr...ages/page7.jpg
Posted by AKWS, 03-31-2005, 10:07 AM
Alternate URL: http://www.pc20.com/tmp/
Posted by Mach3, 03-31-2005, 10:07 AM
I wonder if pinging and doing all this is really the smart idea, I hope we arne't adding on to the problem trying to connect every 5 seconds..
Posted by snJohn, 03-31-2005, 10:07 AM
link doesn't work
Posted by zsindhu, 03-31-2005, 10:07 AM
"Sorry, this site is temporarily unavailable! The web site you are trying to access has exceeded its allocated data transfer. Visit our help area for more information." Woah! That was fast
Posted by shockx5, 03-31-2005, 10:07 AM
Anyone have any news on 70.85.49.*
Posted by photodesign, 03-31-2005, 10:08 AM
Still down...
Posted by hosttone, 03-31-2005, 10:08 AM
our server, 64.5.44.* still down.
Posted by Lem0nHead, 03-31-2005, 10:08 AM
anyone is getting to access orbit?
Posted by bugsoft, 03-31-2005, 10:08 AM
69.93.186.* down
Posted by dkloke, 03-31-2005, 10:09 AM
70.85.91 is up for about the last hour
Posted by NNNils, 03-31-2005, 10:09 AM
just some went down here again 64.5.32.* 1 up 2 down (was 2 up 1 down) 64.5.43.* 1 down (was 1 up)
Posted by NetSender, 03-31-2005, 10:09 AM
I've got onto Orbit, but All I got a page saying that it was temporarily closed
Posted by Weppel, 03-31-2005, 10:09 AM
69.56.214.x still down (about 4 hours now) 69.41.241.x and 69.93.56.x are both up again
Posted by Karl Austin, 03-31-2005, 10:09 AM
Yes, simple in theory, not so simple in practice. If a pair of items in an N+1 configuration both fail at the same time, then there isn't a lot that can be done, other than spending more money and going for N+2. It's rare for two items of equipment in an N+1 configuratino to fail at the same time, but it does happen, it's not 100% foolproof, the same as it isn't with N+2, N+3, N+100 etc. The question is, where do you stop with N+X - because someone has to pay the bills, and they get passed on to the end user, so you have to work out what your target market will pay.
Posted by paulstest, 03-31-2005, 10:10 AM
They really shoudd host Orbit outside their network... P
Posted by igorkl, 03-31-2005, 10:10 AM
PING 70.85.9.x (70.85.9.x) 56(84) bytes of data. 64 bytes from 70.85.9.x: icmp_seq=0 ttl=52 time=144 ms 64 bytes from 70.85.9.x: icmp_seq=1 ttl=52 time=142 ms I host DNS at theplanet's dns servers. Told me mama to keep dns in independent networks.
Posted by Mach3, 03-31-2005, 10:10 AM
when i go to orbit I ether timeout or i get an: The Support Site is not currently available. Please try again later
Posted by v0ltair3, 03-31-2005, 10:11 AM
I can access the login page but can't login. It stay on the form post and eventualy gets timeout.
Posted by BenS2, 03-31-2005, 10:11 AM
Why would SM/TP release a statement saying all is fixed when CLEARLY its not. There are loads of us that still can't see our servers, and the SM/TP servers arn't available either. Whats the point of releasing such a statement? All it does is make you wonder if even they know the status of their own network! Also, if it was as simple as a power failure, and the power has now been restored then why are some servers still down, and why are there apparent network issues (High packet loss and ping times are getting slower and slower for me!). Ben
Posted by TheRose, 03-31-2005, 10:11 AM
Not everything Seems many servers are still out. Why? When? Thanks LE
Posted by meldra, 03-31-2005, 10:11 AM
Then, if they make it circle the entire network, it would be true to it's name
Posted by djdiz-e, 03-31-2005, 10:11 AM
what happening on the cam ..i cant seem to connect to it
Posted by WebMate, 03-31-2005, 10:12 AM
I'll second that. No mate, things aint all fixed.
Posted by cloak, 03-31-2005, 10:12 AM
67.18.56.* still seems to be down. The rest of my servers are up.
Posted by paulstest, 03-31-2005, 10:12 AM
He looks stressed! http://www.MoDaCo.com/index.php?showtopic=204149 lol... we've all had days like this at work right? ;-) P
Posted by jeroman, 03-31-2005, 10:12 AM
Offcourse! Getting hundreds/thousands of tickets asking the same thing is not so good. But they should have a "status/info" page somewhere else...
Posted by almico, 03-31-2005, 10:12 AM
I just logged into Orbit, but my 67.18.* server is down
Posted by Mach3, 03-31-2005, 10:12 AM
they said the power is fixed. nothing about all the servers being up.
Posted by Lem0nHead, 03-31-2005, 10:13 AM
4 HOURS AND COUNTING wow that's almost 1% downtime...
Posted by chrisbond, 03-31-2005, 10:13 AM
From watching the webcam its quite clear there trying to isolate problems as the big U2Share cabinet seems to be where a few engineers are.
Posted by SHTech, 03-31-2005, 10:13 AM
At least they have stabilized the situation... Servers not popping up & down... Hope that they are working on the rest, not to relax imagine that "The Planet is the best"... ;(
Posted by Hostmax-br, 03-31-2005, 10:14 AM
Indeed... now it´s not 100%....
Posted by cyberhost, 03-31-2005, 10:14 AM
I blame The Alamo (its the only thing we Australians know about Texas, apart from GWB ;-)
Posted by Lem0nHead, 03-31-2005, 10:14 AM
my bad it's 0.5% not 1%
Posted by cjhhiv, 03-31-2005, 10:15 AM
I got into orbit. All graphs for Abovenet and Level3 are completely blank. The others are up and reflect the power issues.
Posted by Mach3, 03-31-2005, 10:15 AM
Yeh, its better then the statement "Oh, don't worry about it, it'll fix itself", they are working on it.
Posted by eric1234, 03-31-2005, 10:15 AM
69.56.164.* still down....
Posted by silentnight, 03-31-2005, 10:15 AM
12.96.161.* still down.
Posted by photodesign, 03-31-2005, 10:16 AM
69.93.193.* still down
Posted by diederik, 03-31-2005, 10:17 AM
I'm also still down... (*@!@
Posted by bugsoft, 03-31-2005, 10:17 AM
69.93.186.* still down
Posted by Lem0nHead, 03-31-2005, 10:17 AM
i got into orbit too there's no reboot ticket for my server or something like that
Posted by Laws, 03-31-2005, 10:17 AM
Can anyone understand the upstream graphs in Orbit? They look truly messed up...looks like they lost many of the gig-es.
Posted by Darkanoid, 03-31-2005, 10:18 AM
69.93.181.* still down
Posted by Mach3, 03-31-2005, 10:19 AM
ay i got into orbit, and looking at network overview, it seems to be all up
Posted by Lem0nHead, 03-31-2005, 10:19 AM
it looks like things are starting to get back now i'd guess 1 hour
Posted by zsindhu, 03-31-2005, 10:19 AM
Orbit is back up for me as well.
Posted by NNNils, 03-31-2005, 10:19 AM
I still have servers in 64.5.* going up and down.
Posted by Snowman30, 03-31-2005, 10:19 AM
Id blame GWB before the Alamo... i have been reading thru all this discussion and i must say i love the people bitching and moaning and sayoing they are going elsewhere... i wonder if these people do this at every data centre everytime something happens..... id hate to bee one of their clients. sure this outage is huge and sure the communication or lack thereof is bad but one major outage in over a year isnt too bad compared to a lot of other DC's out there
Posted by sightz, 03-31-2005, 10:19 AM
Orbit appears to be open for business (but slow). Got my Reboot ticket in: Reboot Request: 659267 added successfully. Only 659 thousand boxes in front of mine!
Posted by dkloke, 03-31-2005, 10:19 AM
very true. but then, that cant be the whole story. there must have been some kind of cascading effect, eg. dns servers being affected making it hard to pinpoint what needed to be rebooted.. (not that i know the answers).. you cant just N+1 at a few points of failure in a many-layerd system. redundancy should cover vertical effects too. not that im complaining.. ill bet TP learns something about failsafing from this event. question becomes, will they share the interesting bits? time will tell.
Posted by Mach3, 03-31-2005, 10:21 AM
levelr3, abovenet and nlayer it aint generating statsistics for though and my server is still down
Posted by ResellerPlanet, 03-31-2005, 10:22 AM
Orbit back up indd. When you check the stats it seems like there was a sudden complete outage. It seems like some ISP's are still not connected... Like AboveNet, they don't show any activity on the graphs.
Posted by NetSender, 03-31-2005, 10:23 AM
Hmm, it looks like some guys just hitting that big U2Share box on the webcam. Heh.
Posted by eric1234, 03-31-2005, 10:23 AM
Once the server are up again, how can i request for refund/compensation? Based on the SLA, can we really get it?
Posted by djdiz-e, 03-31-2005, 10:23 AM
my server is 70.85.68.* and seems its been down for the time
Posted by AKWS, 03-31-2005, 10:23 AM
Its is very simple, and it was a cascading effect. I think that guy nicked the terminal, then those 2 guys went looking for it but also forget to plug your server. But all of this happened while the other tech was getting "serviced" in the cage with the blond tech. I know you wouldnt have belived me without proof. http://www.pc20.com/tmp/
Posted by 0utlier, 03-31-2005, 10:23 AM
67.18.15 still down for me. This makes me nervous. The other 9 servers are running fine now though.
Posted by Lem0nHead, 03-31-2005, 10:24 AM
yes you should write a ticket to Sales but guess/hope TP will publically offer at least 20% or 40% compensation for all servers
Posted by zenfort, 03-31-2005, 10:24 AM
Legal probably isn't told everthing (have to maintain deniability in public statements) - and then again it's legal's need to not tell everthing.
Posted by Gually, 03-31-2005, 10:24 AM
69.93.146.* still down
Posted by Karl Austin, 03-31-2005, 10:24 AM
I'm well aware that you can't just put redundancy in a few select locations, I was merely commenting on the fact that no matter how much redundancy you add, someone has to pay for it, and that it can still go wrong.
Posted by v0ltair3, 03-31-2005, 10:24 AM
Ok. I manage to get into orbit also. If you check the Netflow graphs you can see that there is nathing accounted for more than 4 hours https://orbit.theplanet.com/nav_netw...tatistics.html In the Network overview everything is up and running. https://orbit.theplanet.com/nav_netw..._overview.html The bad part is that servers at 69.93.228.* are still down
Posted by ResellerPlanet, 03-31-2005, 10:25 AM
LMAO! Excellent!
Posted by meldra, 03-31-2005, 10:25 AM
How about the masses of potential US President DNA?
Posted by tinnitus, 03-31-2005, 10:25 AM
69.93.32.* still down traceroute showing packets getting stuck coming out of Dallas Level 3
Posted by Ramprage, 03-31-2005, 10:25 AM
I think they're least concerned about that at this point, I just want the server back up. If they don't offer any kind of refund I think I just might move a few boxes to a new provider. I'm blown away how The Planet responded to this when things weren't fixed. I'm also amazed how they're big fancy redundant network isn't sooo redundant after all.
Posted by dkloke, 03-31-2005, 10:26 AM
ha! ASL? ok lets not go there.
Posted by AlexF, 03-31-2005, 10:26 AM
Which cam is this? Nothing happening on http://69.41.224.20/popup.html and can't get http://64.5.44.170/popup.html to load..
Posted by ResellerPlanet, 03-31-2005, 10:26 AM
So I guess the network is back up but now they have to boot all servers?
Posted by Mach3, 03-31-2005, 10:27 AM
ay theplanet.com is back up too, my server aint though
Posted by photodesign, 03-31-2005, 10:27 AM
LE: 69.93.193.* is still down. When will it REALLY be normal and fixed?
Posted by Snowman30, 03-31-2005, 10:27 AM
Just noticed in orbit that most of the our servers are not in the orbit monitoring system currently and a lot of sections of orbit are a mess
Posted by NNNils, 03-31-2005, 10:27 AM
Other still experiencing there servers going up/down ? I have 3 server in 64.5 range going up/down I have 1 server in same range being down all the time.
Posted by ResellerPlanet, 03-31-2005, 10:28 AM
If we knew, it would have already been mentionned...
Posted by sightz, 03-31-2005, 10:29 AM
I suspect that since all the servers were hard-rebooted that they will all be checking their filesystems now. How long does that take? Is it normal for it to answer pings while doing that?
Posted by Mach3, 03-31-2005, 10:29 AM
some bday this turned out to be T.T
Posted by JenniH, 03-31-2005, 10:30 AM
Has anyone actually had a response from them regarding an ETA, or a diagnosis of the actual problem?
Posted by Karl Austin, 03-31-2005, 10:30 AM
Then you'll be very upset wherever you go, hardly any providers plan for more than one item in a set failing i.e. N+1, the exception generally being chillers for aircon, where most providers opperate at N+2. If you want to make sure something like this doesn't happen to you again, you need to put your hand in your pocket and pay for a provider that opperates N+2, N+3 etc. at every layer of their infrastructure - Then come back and tell us all if it costs the same as services at TP.
Posted by Von Vonder, 03-31-2005, 10:31 AM
Nothing like repairing the good old 2.8 GB mysql table. Good times.
Posted by ResellerPlanet, 03-31-2005, 10:32 AM
My mailbox is full of all those "Reply to post ThePlanet down??" emails
Posted by Laws, 03-31-2005, 10:32 AM
It's posted a few pages back...
Posted by Mach3, 03-31-2005, 10:33 AM
I wonder what tipping point is cause it says Critical Block 2289: MS-RPC: DCOM ISystemActivator Overflow I wonder if that has something to do with the problem?
Posted by coight, 03-31-2005, 10:33 AM
Hey Karl I think the problem well the problem I'm having is their complete lack of contact regarding this issue. It's disgusting. JenniH no, weve rung and they have not been to helpful.
Posted by 1round, 03-31-2005, 10:33 AM
I'm unable to view the cams in IE but they are fine in Firefox, it might be that I don't have the correct applet installed in IE?? Just a thought
Posted by TopHostSupport, 03-31-2005, 10:33 AM
Tell me about it...It's a flood attack.
Posted by Ramprage, 03-31-2005, 10:33 AM
Are you affiliated with them? They advertise a 100% uptime SLA - where can I find the details about this?
Posted by my_forum_id, 03-31-2005, 10:34 AM
The servers WEREN'T all hard rebooted (where are you people getting this stuff from ?) - we had 5 Windows servers that were affected by this and all are still showing 49 days uptime.
Posted by Mach3, 03-31-2005, 10:34 AM
if you want 100% uptime, mirroring on different data facilities is probably the way to go.
Posted by ResellerPlanet, 03-31-2005, 10:34 AM
Btw the reason why they have temporary removed all monitoring services is because at this moment the monitoring system would cause a major email flood most probably...
Posted by navmonkey, 03-31-2005, 10:35 AM
Does your tipping point report contains any information? Mine doesn't.
Posted by bugsoft, 03-31-2005, 10:35 AM
Still down: 69.93.178.* 69.93.186.*
Posted by Karl Austin, 03-31-2005, 10:36 AM
No I'm not, I just happen to understand the ins and outs of infrastructure/network design/costs a bit more than most here do. You all want mega cheap deals, but you also want 100% uptime - The two just don't go together, if you want the utmost uptime, you have to pay, it's as simple as that. Just because a provider offers a 100% SLA, it doesn't mean they are going to achieve it every single month/year etc. it is merely there as a reasurance, basically the provider putting their money where their mouth is usually, nothing more, nothing less.
Posted by VONetwork, 03-31-2005, 10:37 AM
Ticket has been updated at TP =============================== ========================= (asmith-03/31/2005 08:28:38): Good Morning Mr. Oxley, I have an update for you At approx. 4:15 AM CST, a pair of redundant Powerware 500KVA UPS units failed - creating a power failure in Section B of our D2 datacenter. Emergency teams were deployed within minutes and power was restored, however intermittent power outages have continued to occur until 6:45 AM CST. Powerware, JT Packard, and electricians are currently onsite with over 100 Planet technicians working to resolve the issue. We do NOT anticipate any futher outages. A formal RFO will be released once the team debriefs. We apologize for all the issues this has caused you. =========================
Posted by ResellerPlanet, 03-31-2005, 10:37 AM
Some people were saying that they noticed an "unplanned server shutdown" or something...
Posted by friendzone, 03-31-2005, 10:37 AM
EST 9.36 Theplanet is up, but servermatrix is down on my side. and I cannot access my server at servermatrix too, I just called the planet, their UPS (power supply) has problem. And they are fixing it.
Posted by JenniH, 03-31-2005, 10:38 AM
I have a couple of important sites I can swing around to other DCs.... but it takes 30 mins for me to do the setup. The questions therefore.... and I do realize it's ballpark guesswork stuff.... how much longer do you thing it will last? Less Than More than 30 mins More than 1 hour More than 2 hours More than 5 hours What's your best guess? It's not exactly scientific, but it gives me at least something to work with!
Posted by coight, 03-31-2005, 10:38 AM
69.93 is dead 4 hours now. Device Outages Uptime Downtime Other Uptime% EART 1 19:45:55 04:14:05 00:00:00 82.355% URAN 1 19:38:33 04:21:27 00:00:00 81.844%
Posted by ResellerPlanet, 03-31-2005, 10:39 AM
Woohoo I was correct, it was an UPS/Diezel failure
Posted by beachcompcom, 03-31-2005, 10:39 AM
To put in a word for The Planet and Server Matrix, this is a first in nearly 2 years that we've had ANY problems with them. They are always on top of things. However, the lack of communications, and the fact that when you call in, they provide no information is a huge downer.
Posted by sieb, 03-31-2005, 10:39 AM
Last I saw they advertised 99.90%, which is what, 5 hours total downtime or something? Given their record since I have been with them, I am impressed. Everything breaks eventually though, and until something breaks, you can't be sure your protected against it.. *shrug*
Posted by Laws, 03-31-2005, 10:40 AM
How can they know? It basically depends how many servers have broken after the power outage. Each server has to be rebooted and hooked up to console to ensure it is okay and this will take a fair bit of time, I should imagine they won't have enough consoles for 100 techs to all be rebooting servers
Posted by neutro, 03-31-2005, 10:40 AM
Got this in orbit IMPORTANT NOTICE! At approximately 4:15AM CST, a pair of redundant Powerware 500KVA UPS units failed creating a power failure in section B of our DLLSTX2 datacenter. Emergency teams were deployed within minutes and power was restored within minutes but intermmittment power outages did occur until 6:45AM CST. Powerware, JT Packard, and electricians are currently onsite with over 100 Planet technicians working to resolve the issue. We do not anticipate any further outages . A formal RFO will be released once the team debriefs. We apologize for all issues that has caused.
Posted by WebMate, 03-31-2005, 10:41 AM
Very true and I understand that too. Our customers have also been very supportive and understanding!
Posted by kingpcgeek, 03-31-2005, 10:41 AM
Then how come my servers went down at 3:50 CST and are stiill down?
Posted by DR4296, 03-31-2005, 10:41 AM
Gee, that line about "however intermittent power outages have continued to occur until 6:45 AM CST" makes it sound like all issues were resolved 2 hours ago ??? Sure seems to me like it's still going on. -= Dave =-
Posted by UH-Matt, 03-31-2005, 10:42 AM
our TP box is still dead. Is it worth me trying to find my orbit login info or will they eventually boot it if necessary?
Posted by DCM, 03-31-2005, 10:42 AM
That cannot be! Windows boxes must be rebooted daily to keep them working (or that is what the Linux Admin would want you to think).
Posted by Asci, 03-31-2005, 10:42 AM
All of our TMS servers remain down and we're still unable to access TMS's site(s). Anyone heard anything regarding TMS specifically?
Posted by Snowman30, 03-31-2005, 10:43 AM
On the subject of server reboots out of our servers at The Planet and SM we have only had our TCS servers rebooted, all our other servers were not rebooted
Posted by coight, 03-31-2005, 10:43 AM
Matt, I think they are going through the boxes one by one. Who knows?
Posted by Ramprage, 03-31-2005, 10:43 AM
Okay but if a pair of redundant power units fails shouldn't they have some type of backup generators? I never complained about pricing, I don't think you understood my post. Like the event of EV1 last year, they seemed to be a little more on top of the ball as for their emergency plan. I'm NOT saying I expect 100% uptime at all. What I'm saying is I'm not impressed by the matter in which The Planet is handling the situation, and by not keeping us updated isn't giving me much satisfaction. All I want is a simple progress update on the situation at hand. If they claim to have "100 techs on site" why can't one of them take 25 seconds and post and update report? The other issue is that they posted an update report in this thread with inaccurate information! I'd rather not know what is up than telling me "everything is fine" when clearly everyone's servers are still down.
Posted by bheka, 03-31-2005, 10:44 AM
so when do we get 100% refund?? How many hours??
Posted by Laws, 03-31-2005, 10:44 AM
Anyone tried rebooting their servers using the consoles TP provides? This may break them further though...just wondering if any of you have attempted it yet...
Posted by photodesign, 03-31-2005, 10:44 AM
NetRange 69.93.0.0 - 69.93.255.255 still down.
Posted by Lem0nHead, 03-31-2005, 10:45 AM
a 30 days month has 30*24*60 minutes = 43200 so, 0.1% of that is 43.20 minutes
Posted by alexrobonlin, 03-31-2005, 10:45 AM
Yes, they're going box by box - mine is in the reboot queue. Alex
Posted by galantina, 03-31-2005, 10:45 AM
Does anybody have metrix extreme server 67.19.* up?
Posted by eric1234, 03-31-2005, 10:45 AM
Can anyone tell me how i can do mirroring on different data facilities? Any link where i can study about it?
Posted by sightz, 03-31-2005, 10:46 AM
Where do you see those notices in Orbit. I can't find them!
Posted by AlexF, 03-31-2005, 10:46 AM
Same hear (HE & Yipes)...
Posted by WebMate, 03-31-2005, 10:46 AM
Probably not. I've phoned them a few times and apparently this situation has brought the support to a halt. I asked a tech to see if my servers had been rebooted and he said it wasn't possible.
Posted by sieb, 03-31-2005, 10:47 AM
Oribt works for me. Shows both Dall2 and 4 network are all green, though MRTG monitoring has been suspended since 4am (graphs go blank after 4). My server is still unreachable though, 67.18.78.*.
Posted by Laws, 03-31-2005, 10:47 AM
I remember when EV1 had a massive outage and Headsurfer personally jumped in and provided 10 min updates on his own boards and WHT (back when he owned it) and you could tell he was genuinely working his hardest and yelling at everyone in sight to have it fixed. While looking more professional with their statements, they clearly show no emotion towards the situation and it makes them look less responsible...
Posted by SHTech, 03-31-2005, 10:50 AM
> There are currently 851 member(s) and 673 guest(s) on the boards. Our Record is 1538 on 03-31-2005 at 04:42 PM. Nice record though! P.S. Servers in 67.18.* are still not accessible...
Posted by NexusHelm, 03-31-2005, 10:50 AM
I'm also in the 67.18.* range, and my server is still down.
Posted by NNNils, 03-31-2005, 10:50 AM
I still have servers going up and down. Other people experiencing the same?
Posted by Duport, 03-31-2005, 10:51 AM
TP website seems to load here just this second
Posted by Highforge, 03-31-2005, 10:51 AM

Posted by neutro, 03-31-2005, 10:52 AM
was browing orbit suddenly a small page pop up from nowhere
Posted by XN-Matt, 03-31-2005, 10:52 AM
I guess we are all called Mr Oxley - anyone who has a ticket reply from asmith at TP - will know what im talking about.
Posted by bjdea1, 03-31-2005, 10:53 AM
My server in 67.18.* range is STILL DOWN - has been down now for soimething like 5 hours???
Posted by alain, 03-31-2005, 10:54 AM
http://forum.servermatrix.com is loading now.
Posted by XN-Matt, 03-31-2005, 10:55 AM
THe longest our box has been down is; 0d 4h 38m 49s
Posted by Ramprage, 03-31-2005, 10:56 AM
Nick I couldn't agree more. This is just making me questing their backup plan and emergency planning for events like this, and the very large lack of customer communications. Since this entire thing has started I think they posted 1 statement about an hour ago in these forums, now the same statement shows up on orbit..... another update would be nice. My box is still down, what is the eta, what are they doing to fix the problem, are they manually rebooting servers or do i need to put in a support request ticket... these are questions that need answering!
Posted by killer15669, 03-31-2005, 10:57 AM
I guess their phone line is down too, but intentionally for this one
Posted by dkloke, 03-31-2005, 10:57 AM
not that it makes it OK.. buti can see where a dc that hasnt had much catastrophic down-time might not have the most comprehensive emergency procedures in place. nature of the beast. personally, having my hand held through this kind of thing comes in low on my priorities list. very glad of this forum tho ;-) my sympathies to all that are suffering.
Posted by Lem0nHead, 03-31-2005, 10:57 AM
it's back now! 69.56 no reboot
Posted by bjdea1, 03-31-2005, 10:57 AM
Hooray Its ONLINE FINALLY 67.18* Back in business
Posted by nothingg, 03-31-2005, 10:57 AM
im gonna kill somebody
Posted by Gually, 03-31-2005, 10:58 AM
my server hasn't been up yet anytime 69.93.146.x
Posted by jedrek, 03-31-2005, 10:58 AM
it's all back up, no reboot.
Posted by yawsh, 03-31-2005, 10:58 AM
They are just back
Posted by zsindhu, 03-31-2005, 10:58 AM
Phew! Our servers are back up, I can go home now
Posted by coight, 03-31-2005, 10:58 AM
Longest downtime recorded so far on our boxes: 04:42:05 Edit: Our boxes on 69.93 are still not up
Posted by InsideHosts.com, 03-31-2005, 10:59 AM
Team Webhosting: World Police?
Posted by Karl Austin, 03-31-2005, 10:59 AM
In a DC the UPS work inline with the power, so all power goes through them: Mains -> UPS -> Servers Generator -> UPS -> Servers (In simplified terms) So the generators sit behind the UPS, so if the UPS die, there's usually no way for generator power to get to the equipment - Now what you can do, is have a UPS bypass, but, with that you're going to get a blip in the power - which is going to reset all equipment being fed by it, and the same will happen again when you switch back to power going via the UPS systems. What some facilities provide, is two power feeds, a UPS feed and a generator feed - but again, unless they knew the UPS was going to fail, and powered up the generators, you'd still be without power for a period and you'd also have to have the automatic transfer equipment in each rack to handle the distribution from two feeds.
Posted by Tom Pyles, 03-31-2005, 10:59 AM
I'm in the same range as you and we're still down.
Posted by net-trend, 03-31-2005, 10:59 AM
Great, 1 server still down and they just closed the ticket without reply. I'm hoping this has to do with their UPS issue.
Posted by onthespot, 03-31-2005, 10:59 AM
I have to say, booting tons of linux servers after an outage is a NIGHTMARE. Ive had to do it at my previous job and let me tell you its not fun. Keep in mind a majority of these boxes are linux/unix and will typically run fsck on each box upon booting due to the fact that they were not shutdown properly.
Posted by Oleanhst, 03-31-2005, 11:00 AM
209.120.***.** still down
Posted by Gint, 03-31-2005, 11:00 AM
I just got off the phone with tech support. I'm on 69.93.203 and he said that part of the network should be online now as it is routing properly internally. Howerver, I can't see my server online yet. He suggested that it might be doing a file system check. Can anybody on 69.92.203.xxx confirm that they are up?
Posted by NNNils, 03-31-2005, 11:00 AM
64.5.* going up and down (more down than up (( )
Posted by TMS - JoseQ, 03-31-2005, 11:00 AM
Our HE.net and Yipes connection rely on The Planet equipment in order to bridge our link to Equinix at the Infomart. It seems that that part of the bridge is down still, thus why all of our HE.net and Yipes servers are currently offline. The techs are working on checking SM boxes, but the network engineers are still at work making sure all of the network segments are brought back online. Once that is completed, the TMS servers should be back online as well. As far as the power, it doesnt look like our area was affected from the sounds of it, as we're pretty far from their networking equipment, and servers on the premium bandwidth which are back up, don't show any power failures. That's all I know at this time. I wish I had more info but it's been hard to come by... JoseQ
Posted by boeki, 03-31-2005, 11:00 AM
read through all 75 pages while constantly monitoring our servers. still down for 69.93.188. and 69.93.154. ranges
Posted by fire100, 03-31-2005, 11:01 AM
I am still down 66.227.x.xxx
Posted by coight, 03-31-2005, 11:01 AM
Our boxes on 69.93 are still not up....
Posted by Phinneus, 03-31-2005, 11:01 AM
my server back up too 67.18.* and no reboot.......but sm website still down here so I still dont think everything is cleared up yet
Posted by photodesign, 03-31-2005, 11:02 AM
69.93.193.0-69.93.193.255 down
Posted by Oleanhst, 03-31-2005, 11:02 AM
Thanks for the update Jose
Posted by kashaziz, 03-31-2005, 11:03 AM
We are back let see for how long.
Posted by v0ltair3, 03-31-2005, 11:03 AM
69.93.228.* 2 servers up. Have been up for about 3 mins now. Lets hope they stay up. BTW, they had been hard rebooted again.
Posted by Hoojx, 03-31-2005, 11:03 AM
Network is down since 1100 (GMT) Going up and down from 1300 to 1400 It is up for 2 hours now ... so I hope the problem is solved. I'm at 67.18.*.*
Posted by funkytaco, 03-31-2005, 11:03 AM
WHT - Heck Yeah
Posted by Ramprage, 03-31-2005, 11:03 AM
KDAWebServices, I appreciate the explanation of the power system setup. Did you use to work in a DC before or are a systems engineer? Cheers
Posted by c4host, 03-31-2005, 11:04 AM
what is the url for the cams??
Posted by chrisbond, 03-31-2005, 11:04 AM
Thanks Jose
Posted by elcrisol, 03-31-2005, 11:04 AM
From Server Matrix forums:
Posted by daveborg, 03-31-2005, 11:05 AM
Server came back! 69.93.211.*
Posted by AndyGambles, 03-31-2005, 11:05 AM
Well our US site has been down all day and is still not available. I have had no emails or notifications from thePlanet about what is going on which is what is really upsetting me.
Posted by photodesign, 03-31-2005, 11:05 AM
69.93.193.* just started responding to pings after 5 hours
Posted by coight, 03-31-2005, 11:06 AM
One of ours is up
Posted by hosttone, 03-31-2005, 11:06 AM
our server, 64.5.44.* still down.
Posted by beachcompcom, 03-31-2005, 11:06 AM
67.18.13.2** Just back. It HAD lost power.
Posted by Jalberts, 03-31-2005, 11:06 AM
Seems CH's DNS server has gone south, and their support site says they're upgrading, to try back in 15 mins. Been saying that for 3 hours. Anyone know what's going on? I'm paying for a dedicated server from them and this is NOT good.
Posted by onthespot, 03-31-2005, 11:06 AM
70.85.16.228 down (good thing i have nothing on it)
Posted by paulstest, 03-31-2005, 11:07 AM
Mine just this minute returned... hurrah! P
Posted by andrewgmol, 03-31-2005, 11:07 AM
Back here... 69.93.178.x
Posted by Ramprage, 03-31-2005, 11:07 AM
I requested my server be rebooted and I got this response: "We will be performing a sweep of the data center as soon as this issue is completely resolved to reboot and ensure connectivity of all Active servers. Thank you for your patience in this matter. "
Posted by nothingg, 03-31-2005, 11:07 AM
(asmith-03/31/2005 08:50:38): Good Morning Mr. Oxley, I have an update for you At approx. 4:15 AM CST, a pair of redundant Powerware 500KVA UPS units failed - creating a power failure in Section B of our D2 datacenter. Emergency teams were deployed within minutes and power was restored, however intermittent power outages have continued to occur until 6:45 AM CST. Powerware, JT Packard, and electricians are currently onsite with over 100 Planet technicians working to resolve the issue. We do NOT anticipate any futher outages. A formal RFO will be released once the team debriefs. We apologize for all the issues this has caused you. We will be performing a sweep of the data center as soon as this issue is completely resolved to reboot and ensure connectivity of all Active servers. Thank you for your patience in this matter.
Posted by Tom Pyles, 03-31-2005, 11:08 AM
Our final server just came back up 69.93.***.**
Posted by Gually, 03-31-2005, 11:08 AM
uoo! it's up! 69.93.146.x i'm happy!
Posted by coight, 03-31-2005, 11:08 AM
Both are backup now uptime 15:07:58 up 31 days, 2:08, 1 user, load average: 10.89, 6.08, 2.46 Looks like it was not rebooted.
Posted by fire100, 03-31-2005, 11:08 AM
Does anybody have any idea how many servers are actually located at The Planet? Do they document this anywhere?
Posted by Laws, 03-31-2005, 11:09 AM
Is everyone named Mr Oxley or what?
Posted by meldra, 03-31-2005, 11:10 AM
Enough that if I had a dollar for each, I would be quite happy
Posted by onthespot, 03-31-2005, 11:10 AM
going to take hours... I can assure you to get all these boxes back online. The only thing windows is good for is the fact that it can recover from something like this (usually) without manual intervention.
Posted by nothingg, 03-31-2005, 11:10 AM
not me
Posted by ellinas, 03-31-2005, 11:11 AM
69.93.18* UP !! That's good.
Posted by hosttone, 03-31-2005, 11:11 AM
our server, 64.5.44.* still down.
Posted by Gually, 03-31-2005, 11:12 AM
they have reboot my server 2 hours ago. 5 minutes ago the server is up.
Posted by whmcsguru, 03-31-2005, 11:12 AM
I'm still showing a couple of servers down , after a 5 hour grid of 10 or so monitored servers down, so they're getting it back online slowly. It truly is a pain to deal with bringing up every single affected server and router by hand, so be patient, it could take anywhere from 5-90 minutes to get everything back online at 100%. Most likely those that are up now weren't affected directly by the power failure, but by routers/switches/etc that were affected. Seems someone should train their staff better in denying stuff like this before they know 100% about what's going on. Obviously it was a power issue, all the signs pointed to it, and they (finally) admitted it. Typical TP response though, not knowing much about what's really going on. Last edited by whmcsguru; 03-31-2005 at 11:15 AM.
Posted by kaosweaver, 03-31-2005, 11:12 AM
69.93.186.* up Thanks for the work getting us back online.
Posted by photodesign, 03-31-2005, 11:13 AM
No reboot here: 09:11:57 up 56 days, 6:19, 0 users, load average: 0.83, 2.06, 1.29
Posted by sightz, 03-31-2005, 11:14 AM
Did any of you folks who have come back up file reboot tickets? Do you know the reboot request number? Mine was 659267. I assume they go sequentially. I wonder where I am in the line up?
Posted by djdiz-e, 03-31-2005, 11:15 AM
my server hasent responded at all for the whole time ..no domains can resolve at all
Posted by LucasGE, 03-31-2005, 11:15 AM
If the planet went all down, there would be no internet. O_o
Posted by ResellerPlanet, 03-31-2005, 11:16 AM
Same for me, been up for 59 days...
Posted by whmcsguru, 03-31-2005, 11:16 AM
LOL So THAT's where they hide the master on/off switch to the internet. I always wondered where it was.
Posted by osv, 03-31-2005, 11:17 AM
69.93.xx.xx is back up for us now.
Posted by lessthanthree, 03-31-2005, 11:17 AM
maybe its me, but it almost sounds like they mean they are gonna reboot ALL servers, thatd so piss me off, but im reading it wrong, right?
Posted by issbul, 03-31-2005, 11:17 AM
69.56.180.* is up for 2 hours now Just back: ------------- 12.96.164.* 64.5.36.*
Posted by LucasGE, 03-31-2005, 11:17 AM
I assume it might be hosted on The Planet and well, part of The Planet is all down, you can see "The Planet Down" thread.
Posted by fire100, 03-31-2005, 11:17 AM
I wonder how many servers you can fit into a square foot? Lets see, if a rack of 15 servers can fit into a square foot, that would be at least 1,245,000 servers. I am guessing it's more than that though.
Posted by v0ltair3, 03-31-2005, 11:18 AM
can someone tell me what is a RFO ?
Posted by fashion, 03-31-2005, 11:18 AM
all our servers are back up
Posted by Oleanhst, 03-31-2005, 11:18 AM
My TMS Server is back up
Posted by TrevorN, 03-31-2005, 11:19 AM
you have to minus all the random area for people to walk and stuff.
Posted by sieb, 03-31-2005, 11:19 AM
67.18.*.* back up for me. Server still shows 51 days of uptime.
Posted by djdiz-e, 03-31-2005, 11:19 AM
anyone in 70.85.68.* ..up yet?
Posted by ResellerPlanet, 03-31-2005, 11:20 AM
According to Orbit there has been a downtime for about 5 hours, is that possible? https://orbit.theplanet.com/nav_netw...tatistics.html
Posted by meldra, 03-31-2005, 11:20 AM
I've noticed, reading through all this, that it's really shown which hosts actually know what they're talking about and which do not.
Posted by AlexF, 03-31-2005, 11:20 AM
TMS servers are up... Wohoo
Posted by sightz, 03-31-2005, 11:20 AM
Pageviews on this thread have just surpassed the 'Dinix" megathread! 51711 to 50906. Wahoo! We rock! Planet! Planet! Planet!
Posted by universal2001, 03-31-2005, 11:21 AM
Imagine going around hitting the reboot button on all 30,000 servers. lol
Posted by lucis, 03-31-2005, 11:21 AM
finaly i'we got mail....
Posted by whmcsguru, 03-31-2005, 11:22 AM
I think what they mean is they'll reboot all that NEED to be rebooted, not all servers, but, that is just how I understand it. Yes, the servers stopped responding between 4 and 415 am CST and it's now 921 CST, so the outage was close enough to 5 hours long. Some weren't affected the entire time I'm sure, but it started around 4am
Posted by BenS2, 03-31-2005, 11:22 AM
Yes, it went down shortly after 11am UK time, and its now 4:20pm UK time, and some servers are still down, so thats about 5 hours downtime! Ben
Posted by Joshua, 03-31-2005, 11:22 AM
Looks like TMS's HE.net line just came back up, but I guess the techs haven't gotten around to hitting the power button on the ServerMatrix.com server. Our server at TMS on TP bandwidth has been up for a while now, as has our LayeredTech box at TP.
Posted by WebMate, 03-31-2005, 11:22 AM
Our severs are up again! Bloody hell, that was too long :-))
Posted by DR4296, 03-31-2005, 11:23 AM
Hooray! All servers here are back up now. So that makes this something like 5.5 hours of downtime. -= Dave =-
Posted by MatrixAu, 03-31-2005, 11:24 AM
ServerMatrix is back up: http://forums.servermatrix.com/
Posted by fire100, 03-31-2005, 11:26 AM
Back up. 66.227.x.xxx
Posted by junglecat, 03-31-2005, 11:26 AM
*whew* My 5 that were down just came back up. I'm breathing again.
Posted by djdiz-e, 03-31-2005, 11:26 AM
anyone in 70.85.68.* up yet?
Posted by whmcsguru, 03-31-2005, 11:26 AM
ChronicHosting deals a lot with TP servers, so most likely this is part of that problem. TP servers are returning shortly, though, so you should see some signs of life shortly.
Posted by net-trend, 03-31-2005, 11:28 AM
69.93.114.* still down. SM/TP's gonna hurt with the SLA.
Posted by NetSender, 03-31-2005, 11:28 AM
Now for the hordes of support tickets Yay
Posted by nothingg, 03-31-2005, 11:29 AM
67.19.220 up but slow as hell
Posted by djdiz-e, 03-31-2005, 11:32 AM
i was just getting ready to do a backup after some major changes in my main site ..before this happened ..i hope nothin happened to my server
Posted by killer15669, 03-31-2005, 11:33 AM
Question: How did people survived before internet? We are so enslaved and dependant (myself included) to technology, thats amazing. 5 hours is nothing in a whole life but damn it, it hurts so badly >_<
Posted by djdiz-e, 03-31-2005, 11:34 AM
yes it does ..im gonna cry .. lol
Posted by TMS - JoseQ, 03-31-2005, 11:34 AM
I was able to get a hold of one (if not the) senior engineers at The Planet, and he was able to figure out what the problem with our connection was. As a result, TMS connectivity has been restored. All of our IP ranges are accessible again. Thanks for your patience and understanding. JoseQ
Posted by pplante, 03-31-2005, 11:34 AM
5 hours and counting now... 69.93.106.* is still down
Posted by onthespot, 03-31-2005, 11:34 AM
wonder what it was like before tv as well.... and radio.... things must have been really boring.... but im sure more relaxed.
Posted by hosttone, 03-31-2005, 11:36 AM
we are crying. server still down, 64.5.44.*
Posted by snJohn, 03-31-2005, 11:36 AM
*finally* my server is back and it is running pretty fast
Posted by bheka, 03-31-2005, 11:37 AM
67.19.* range seems to be up .. but really really crawling
Posted by djdiz-e, 03-31-2005, 11:37 AM
jiggyjon are you rubbing it in ..lol
Posted by photodesign, 03-31-2005, 11:38 AM
I don't think it was very relaxed when you had to work 60-80 hours a week and your children died on a regular basis. Or how about when the plague hit? That wasn't too relaxing. Ever been chased by a lion or bear while you are trying to get dinner? Not too relaxing either.
Posted by onthespot, 03-31-2005, 11:39 AM
this is true
Posted by vluther, 03-31-2005, 11:39 AM
At approximately 4:15AM CST, a pair of redundant Powerware 500KVA UPS units failed creating a power failure in section B of our DLLSTX2 datacenter. Emergency teams were deployed within minutes and power was restored within minutes but intermmittment power outages did occur until 6:45AM CST. Powerware, JT Packard, and electricians are currently onsite with over 100 Planet technicians working to resolve the issue. We do not anticipate any further outages . A formal RFO will be released once the team debriefs. We apologize for all issues that has caused.
Posted by beachcompcom, 03-31-2005, 11:40 AM
What's an RFO and where is it in Orbit?
Posted by snJohn, 03-31-2005, 11:40 AM
I have already had to disk out a load of refunds though
Posted by photodesign, 03-31-2005, 11:41 AM
RFO=Report Friggin' Overdue
Posted by daveborg, 03-31-2005, 11:43 AM
69.93.* still MIA
Posted by djdiz-e, 03-31-2005, 11:44 AM
im gonna be hurtin after this one i had trouble 2 months ago i was on velocity.net (99% downtime) was down for like 6-7 hours ..and now this less then 2 months later
Posted by barry[CoffeeSprout], 03-31-2005, 11:45 AM
Actually for some weird reason I did not get any downtime (got disconnected for a minute, but all was back up after my lunchbreak) Whatever it was, thanks
Posted by Laws, 03-31-2005, 11:45 AM
http://forums.servermatrix.com/viewtopic.php?t=14361 Post your thoughts.
Posted by snJohn, 03-31-2005, 11:48 AM
Did many notice the downtime?
Posted by kenzo, 03-31-2005, 11:50 AM
http://www.pc20.com/tmp/
Posted by djdiz-e, 03-31-2005, 11:50 AM
yes about 70% of them emailed me already and the last time i lost a few customers
Posted by nothingg, 03-31-2005, 11:51 AM
ok... going down again. WTF!!!!!!!!!!!!!!!!!!!!!!!!!!!
Posted by UH-Matt, 03-31-2005, 11:53 AM
http://ipis.co.uk/nope.jpg
Posted by djdiz-e, 03-31-2005, 11:54 AM
my server is in the 70.84.0.0 - 70.85.127.255 range nothing at all yet
Posted by Fair Dinkum, 03-31-2005, 11:55 AM
The servers we have that were down are all up. The good thing is our servers for our Reseller clients never went down. The bad was almost all of our Dedicated server clients were down during the outage. Glenn
Posted by hosttone, 03-31-2005, 11:57 AM
Any one server still down?? Our server still down.
Posted by djdiz-e, 03-31-2005, 11:58 AM
lol thats a good one
Posted by eric1234, 03-31-2005, 11:58 AM
If all of us complain and ask for a compensation, do you think its possible? I had already received calls from my 3 important customers wanted to change host. I'm going to lose US$50 a month for this 5 hrs outrage...
Posted by wings900, 03-31-2005, 11:58 AM
67.18.3.* down..
Posted by snJohn, 03-31-2005, 11:58 AM
hahahaha
Posted by bheka, 03-31-2005, 11:59 AM
just looking ... ***** The Planet Monitoring System: IPAlert DLLSTX4 ***** down from Date/Time: Thu Mar 31 04:32:03 CST 2005 to Date/Time: Thu Mar 31 09:13:43 CST 2005 I will take my 4hours 45mins refund now thank you
Posted by nothingg, 03-31-2005, 11:59 AM
anyone can tell me how do i get the evidence of the times of downtime ?
Posted by AlexF, 03-31-2005, 12:00 PM
Even though the problem is solved, the headache continue...
Posted by 6PS-Jake, 03-31-2005, 12:00 PM
Hello all, First of all, we have a rack and several other servers at the planet. We appear to be up and running now. However, I was just thinking about the impact of this situation on the hosting world. I am sorry all this happened. I was just wondering... You all think there are about 1.25 million servers collocated there? What percentage do you think went down? Of the servers that failed, how many will suffer hardware failure / data corruption do to the improper shutdowns? For example, Hard Drive Failures. (guess a percentage). Finacially, what impact will this have on The Planet? How is this going to affect you all? I am just wondering your all's thoughts. I am flying to Lousianna in a few hours and will read more tonight. ---------------------- I am serious about this.... I am going to Lousinanna to rollout out a network for a very large Ortho group. I am going to be in the hotel bored out of my skull. If you all need any help at all, please contact me. My information is in my profile. Best is email for now since i will be in a plane. As in any Disaster, there are people who volunteer to help others. Please don't think this some fishing trip for business. My offer to help is that, help. Our community is made up of a collection of different business, some of the largest host only have a handful of people. I am lucky because we were not hit that hard. Please see this as a friend helping a friend. Someday, I may need the return favor. I will not be able to assist until tonight when I am in the hotel. I am a great Windows Engineer, but my skills in Linux are limited. My plane takes off around 1. If you need help, please let me know via email. Good Luck.
Posted by Eleven2 Hosting, 03-31-2005, 12:00 PM
can anyone say EV1 just incase you forget www.ev1servers.net
Posted by hosttone, 03-31-2005, 12:02 PM
Are you sure all server already up?? How you very sure??
Posted by akuo, 03-31-2005, 12:03 PM
The interesting thing for me is that TP is claiming it only affected "a small portion" of their 'DLLSTX2' facility, but from customer comments this seems to be incorrect? It would be very interesting if this actually affected more than one DC...
Posted by Eleven2 Hosting, 03-31-2005, 12:03 PM
1.25 million servers collocated there? Who says 1.25 million? Try only about 25,000 servers.
Posted by Laws, 03-31-2005, 12:05 PM
They won't learn from this unless you tell them what you thought.... http://forums.servermatrix.com/viewtopic.php?t=14361
Posted by 6PS-Jake, 03-31-2005, 12:05 PM
Elven2 that is more in line with what I thought. I saw someone posted over a million. Anyone have some close numbers?
Posted by Melitaweb, 03-31-2005, 12:09 PM
My servers at theplanet are back up but just lost one at dedicatednow. Probably just coincidence.
Posted by fire100, 03-31-2005, 12:10 PM
He was going from my guess of the amount of servers that could ideally in rack situation could be located there. I was curious as to the exact number, but nobody seems to know.
Posted by ServerZB, 03-31-2005, 12:11 PM
still no annoucnement?? wow....
Posted by hosttone, 03-31-2005, 12:12 PM
our server still down. Unbelievabe??? Believe it.
Posted by 1980, 03-31-2005, 12:13 PM
back up here - 69.93.*.*
Posted by eric1234, 03-31-2005, 12:15 PM
Next month bill is coming, can we demand compensation from theplanet? I dont feel like paying them after losing my key customers!!
Posted by Ramprage, 03-31-2005, 12:16 PM
OMG you have to be friggen kidding! They turned off ORBIT! https://orbit.theplanet.com/ "The Support Site is not currently available. Please try again later."
Posted by hosttone, 03-31-2005, 12:16 PM
server still down, 64.5.44.*
Posted by sightz, 03-31-2005, 12:16 PM
Still down 69.93.232.x Orbit is back down now... Customers freaking.
Posted by ResellerPlanet, 03-31-2005, 12:16 PM
I was wondering the same. Can we ask a part of our money back? Also:
Posted by akuo, 03-31-2005, 12:17 PM
Not to start a conspiracy theory, but I have now seen outage reports from customers in DLLSTX2, 4 and 5. Anybody figure how two UPS's (of 14 total) going down can kill connectivity to a bunch of servers across three geographically seperate datacenters?
Posted by whmcsguru, 03-31-2005, 12:18 PM
Linux is pretty friendly in that respect. While I wouldn't exactly advise repeatedly doing this, just like windows you can shut the server down through the power off switch, and it'll be just dandy. Because this happened over an extended period of time, it's reasonably safe to say that no data was being processed when they were shutdown, though checking mysql db's is always a good thing after something like this. Again, not much data loss, or I wouldn't think so. As far as actual hardware failure: Any time you cut the power to the box like that, you run the risk of frying something out. Do it repeatedly, and you increase that risk. Of course, these are Dells (for the most part) we're talking about, so who knows there. They'll probably be under warranty somehow, and the few that do get screwed up will be replaced. I'd say that maybe 1% of the servers that were forced down due to power will have an issue, of course, that is a low estimate, but who knows.
Posted by ResellerPlanet, 03-31-2005, 12:19 PM
70.85 has been online for a while now...
Posted by kmike, 03-31-2005, 12:20 PM
3 of our colocated servers are still down. All in 12.156. subnet. Judging from the fact they're all in the same cab (and some in one rack even), and other servers are up, but were hard rebooted 3 times, it seems something went wrong with their plugging/unplugging procedure. We'd have to wait for our turn of staff attention. Oh well.
Posted by djdiz-e, 03-31-2005, 12:21 PM
im on 70.85.68.* and havent seen any action at all in 5 hours
Posted by sightz, 03-31-2005, 12:22 PM
Just called. The latest news is that the techs on the phone can't even get into orbit to report on ticket status. My box is answering pings again after being dead for the last hour. No HTTP or SSH though!
Posted by nathand, 03-31-2005, 12:23 PM
I've only got one server but oh my this is so bad. What do people think the outcome will be after all is said and done?? nate
Posted by Ramprage, 03-31-2005, 12:23 PM
Oh well? If you're going for the extra effort of colocation and no one is able to help you I don't think I would be satisfied for an "oh well" when no one is able to get your server back online after 5 hours of downtime.... If anything this is a real wake up call for all The Planet customers to rethink which dedicated provider they're using.
Posted by 6PS-Jake, 03-31-2005, 12:24 PM
I was thinking 2 - 3%... Sounds like we are thinking about the same.
Posted by nathand, 03-31-2005, 12:24 PM
And, our server has been up for awhile, but I had to come to this forum to find ANYTHING out. That in itself is frustrating. nate
Posted by SarcasticDwarf, 03-31-2005, 12:24 PM
The larger problem will be database corruption. MySQL does not handle loss of power well.
Posted by fire100, 03-31-2005, 12:26 PM
Probably lots of money out of host pockets for their uptime guarantees. And datacenters and host that tread lightly in their guarantee policies. There is always a chance that something will happen. I imagine some clients will jump ships to another datacenter as well.
Posted by sirius, 03-31-2005, 12:27 PM
Oh for the love of god.... GROW UP! The hosting business isn't for children! Sirius
Posted by pplante, 03-31-2005, 12:27 PM
i requested for my orbit information to be sent to my gmail account about 20 minutes ago and they have not sent it yet. this is really pissing me off, i got the run around on the phone before even getting told i would be emailed it. that'll teach me for using IMAP for email lol.
Posted by Killa200, 03-31-2005, 12:29 PM
o.0, i'm starting to feel that my move to Savvis is more and more meritted everyday. Though i do love TP and their network infrastructure, this situationright now shows the part of TP i hated the most.... Support.
Posted by nathand, 03-31-2005, 12:29 PM
Yeah, that's what I'm envisoining - The Planet will probably take a big hit from this client-wise. All would be seemingly better if we were able to find anything out earlier. I really feel for those companies out there that have frustrated customers - and that have lost customers - due to this. The Planet has always been awesome to us so I guess I'm just puzzled at why it's gotten this bad...
Posted by Mach3, 03-31-2005, 12:31 PM
look its not that bad, atleast the downtime was at night and morning, if this was at the middle of the day. Then the suicide rate among us would go up tremendesly.
Posted by sightz, 03-31-2005, 12:32 PM
The middle of my day is in 28 minutes and we're still down.
Posted by akuo, 03-31-2005, 12:33 PM
Depending what part of the world you're in...
Posted by tree-host, 03-31-2005, 12:33 PM
I know i dont post much, but it might be at night for people over there, but there are a lot of people that it was actually peak time in the business day for (me included). No matter what time something happens, its peak time in atleast 1 country, so i disagree with what your saying completly.
Posted by 6PS-Jake, 03-31-2005, 12:34 PM
I am just thinking that if everyone was compensated from the SLA... Well, first, the planet will take a huge finacial hit. I hope they planned for it. Second, whatever happens, they will increase prices to compensate for the loss. It might not be next month, but it will happen. I would think they have the proper insurance to cover an outage like this, but when it boils down to it... This is a disaster and money will be lost by everyone involved. Another thing..... Has anyone LOOKED at the SLA contract verbage? I am no attorney and I just don't know, but I wonder if there is some piece that protects them from a huge outage like this? Again, don't panic... I am just yaking here and thinking outloud. I was looking at an SLA for someone the other day and although there was 99.9 percent uptime there was a list of exceptions in there like... Equipment Failure doesn't count... Network outage doesn't count.. Emergecy Maintanace doesn't could. Actually I was wondering what did count when I read it... It wasn't the planet though. What do you all think?
Posted by Mach3, 03-31-2005, 12:34 PM
Thats not good, what ip block are you on?
Posted by consultorpc, 03-31-2005, 12:34 PM
In Europe this has happened at peak hours, so ...
Posted by djdiz-e, 03-31-2005, 12:34 PM
this is the busiest time for me and there no site to goto
Posted by crucialx, 03-31-2005, 12:35 PM
For those who have not seen it yet, the latest update:
Posted by photodesign, 03-31-2005, 12:35 PM
Quote: Location: DLLSTX2 Severity: Level 5 Description: At 4:09AM CST, a power failure occurred in a redundant pair of Powerware UPS units feeding power to section B of the DLLSTX2 datacenter. The power failure was caused by a faulty fuse in UPS unit B-1. As the load transferred to UPS unit B-2, the spike in the load created an overloaded breaker and UPS B2 also lost power. This resulted in a power outage to the main power distribution unit feeding section B of the datacenter floor. Emergency teams were notified and the power was restored within 20 minutes. The power continued to cycle until 6:45AM due to the faulty fuse and the inability of the redundant UPS units to remain in bypass mode. Customers may have noticed several power cycles during this time period. Powerware, JT Packard, and electricians found the problem and replaced the faulty hardware and fuses. At 6:45AM, all electrical service was restored to normal and the NOC team began to bring all servers back online. The technical staff is currently placing a console on all servers to verify server restarts. Customers with operating systems that require a file check may have experienced extended downtime during the file check. Powerware and JT Packard will continue to monitor the UPS systems for the next 24 to 48 hours. The Planet does not anticipate any further outages.
Posted by woolly, 03-31-2005, 12:35 PM
Is the router car4-1.dllstx2 working for anyone? Because my server is under that router and it isn't up for me at the moment.
Posted by fire100, 03-31-2005, 12:36 PM
What is Ironic, is most of the clients out there do not realize that host lease or co-lo their box in a larger datacenter. Quite a few host use The Planet, those clients that jump ship, may just end up with another host that is using The Planet. Not everyone knows to use tools like dnsstuff. I just hope this opens some Hosts eyes up, and tell the truth, state your server is located in a datacenter not owned by you, and you had no control over this one. It was the staff at The Planet working hard to restore everything. If clients realize they had no control over this situation, they may trust their service and stay with them.
Posted by Local2, 03-31-2005, 12:36 PM
Who says it was not in the middle of the day? Not everyone is in the USA!!
Posted by Mach3, 03-31-2005, 12:37 PM
:/, im sorry i dint mean it in a negative way, I just wanted to sound a bit positive. After this insident I was thinking maybe making a mirror server...
Posted by snJohn, 03-31-2005, 12:37 PM
It was peak/lunchtime here
Posted by sightz, 03-31-2005, 12:39 PM
69.93.232.x Connected to Uplink Router: CAR2-7.DLLSTX2 Is anyone on this block or under this router up?
Posted by whmcsguru, 03-31-2005, 12:39 PM
Agreed, thankfully you can (kind of) repair corrupted databases. As well, given the fact that this was a 5 hour situation, it's not too likely that mysql was going to be that heavily used during the major part of that. The servers rebooted (or shut down) in the beginning would suffer most from this one.
Posted by CyanoX, 03-31-2005, 12:41 PM
Is this one of the largest thread posted in only a few hours? Seems ThePlanet is one of the most used, if not the most used, on this forums. Has anyone recieved any news on compensation for those who were affected?
Posted by Mach3, 03-31-2005, 12:41 PM
thats weird i thought DLLSTX2 was operational. sightz, have you tried calling them?
Posted by tree-host, 03-31-2005, 12:42 PM
I can imagine you probably didnt.... but also appreciate that i think a lot of people are not in the best of moods, and trying to justify it wil night time doesnt help us all
Posted by Mach3, 03-31-2005, 12:44 PM
well I also wasn't expecting TP to be used by so many people from europe and etc. From what I hear people like to stick with datacenters in their own countries, sicne you get better ping results.
Posted by SarcasticDwarf, 03-31-2005, 12:46 PM
As far as I can tell my own sites only had minor issues that could easily be corrected. From the report, many servers were cycled multiple times, which may have added to the problem.
Posted by Gint, 03-31-2005, 12:50 PM
yeah, the amount of money the lose from their SLA will be a pittance compared to future business lost... A lof ot people that have servers only at the Planet will realize the importance of having multiple servers at multiple DC's. (myself included) This would be a good time for The Planet/Server Matrix to offer an amazing deal on new services to those effected by the outage.
Posted by tree-host, 03-31-2005, 12:51 PM
Depends what you use it for i suppose.... but, for what server i have, would cost me nearly 3 times as much over here!
Posted by cbwass, 03-31-2005, 12:52 PM
Does anyone know why I can ping my sites but can't open them in a browser?
Posted by yaax, 03-31-2005, 12:52 PM
Same problem happened to their dllstx4 datacenter - when it was still new just few months after it was launched, in the August 2004 - it has about 4 hours of downtime because of almost same power failure. Now this happened to their older dllstx2 datacenter.... Fortunately our server in dllstx4 was not affected from this outage.
Posted by Mach3, 03-31-2005, 12:54 PM
cbwass, maybe cause the ip works but port 80 doesnt, check your webserver.
Posted by sightz, 03-31-2005, 12:55 PM
Same thing here. I can ping my IPs but not open any sites or SSH. I wonder if our boxes are actually up but Tippingpoint or something is blocking all traffic? I'm on 69.93.x.x - and you?
Posted by Eleven2 Hosting, 03-31-2005, 12:55 PM
I bet the sales call volume at ev1 is really really high today and for the next couple of weeks.
Posted by fire100, 03-31-2005, 12:58 PM
More than likely the box is just starting back up, or if on a Linux box, the httpd daemon is not running. If you own the box, see if you can SSH into and try ./apachectl start If you have control panel software that should be able to restart services.
Posted by daveborg, 03-31-2005, 12:58 PM
Just got my Nagios cleared. That was one of the worst customer relations effort I've ever been witness to. A great example of how NOT to do it for everyone running their own business. Was that Legal guy for real? Coming on and saying everythings fixed. He posted hours ago while the tecs are still going from server to server with a console watching them boot up. I'm a supporter of TP, but this would have to be one of the most active threads ever on WHT. Head Planet Dude should have been on here - or somewhere - updating everyone. How many people here had this conversation... Customer: I can't see my website, or get my mail. Support: I know, your server's down - looks like something's wrong with the datacenter C: What's wrong with it? S: Dunno... C: When will it be back online? S: Dunno... C: Am I going to lose any data? S: Gee, hope not
Posted by Lem0nHead, 03-31-2005, 12:58 PM
ev1 should compete to TP turning some servers off too so we could see which thread would grows faster!
Posted by jagarco, 03-31-2005, 12:59 PM
Hi I can do a trace to my server but I can't SSh to the server, nor the domain seems resolve to it. Is the planet going to do something else or I'm by my own now? Could the Planet think everything is ok an leave it like that? Thank you Jagarco
Posted by Mach3, 03-31-2005, 01:00 PM
I disagree, don't think everyone ahs the luxury of jsut moving their entire server. That takes time effort and might be more downtime. You can't always switch that quickly and easily.
Posted by Gint, 03-31-2005, 01:03 PM
At the end of the day it is about keeping your customers happy. Even if they cover themselves in their SLA and they don't need to compensate me for downtime, I'll remember how they handled the aftermath of the situation when it comes time for me to order my next server... This will undoubtedly be bad for The Planet financially, but on the flipside it is good for every other DC out there. The Planet needs to figure out a way to reduce any potential churn now...
Posted by sightz, 03-31-2005, 01:05 PM
Anyone else seeing the pattern here? What IP block and router are you on?
Posted by tree-host, 03-31-2005, 01:06 PM
Would be pretty good for all Datacenters (ev1 for example) to offer amazing deals, bet they would get a few people through... not all servers are easy to move, but cPanel has a built in copy system, and then just update the ns ip addresses.
Posted by junglecat, 03-31-2005, 01:06 PM

Posted by daveborg, 03-31-2005, 01:07 PM
Doesn't have to be bad. Disasters happen to the best of companies, it's how they're handled that makes the difference. If TP go above and beyond in cleaning up this mess, and don't stonewall on issues such as SLAs, they can convert customers into loyal customers.
Posted by Nick Jones, 03-31-2005, 01:08 PM
LOL ! http://www.webhostingtalk.com/showth...hreadid=390449
Posted by whmcsguru, 03-31-2005, 01:09 PM
Yeah, let's watch 'em set one of their DC's on fire and then we can compare threads!!!
Posted by MarioR, 03-31-2005, 01:09 PM
It's truly unfortunate to see an outage such as this, we know all to well what something like this can cause. It is always a learning experience for all of us in this industry. Such a catastrophic event such as what has occured should bring on a new learning experience, but this should serve as a reminder to all that it is always good to have a backup plan of your own.
Posted by jagarco, 03-31-2005, 01:10 PM
69.93.128.* Jagarco
Posted by Gint, 03-31-2005, 01:10 PM
I was in the same boat when other on my block were reporting their servers up but mine was not. I didn't take any chances though, so I submitted a support ticket anyways. It turned out my server was just doing an fsck, so I cancelled the ticket once it came back online. -Gint
Posted by HostingInsider, 03-31-2005, 01:14 PM
I bet around half of the internet is down
Posted by AKWS, 03-31-2005, 01:14 PM
Exactly! Saving a few dollars this month is of no concern. As you can not put a price on happy customers. And those un-happy customers that are gone, they will never be back. So its really in how TP dealt with things and is going to deal with. Without updates from TP, you end up telling your cleints the same babble for 4 hours with no progress. How un-professional does that look!
Posted by Mach3, 03-31-2005, 01:16 PM
look I don't think we should worry about the SLA, thats what it is for. If network crashes or power goes down the SLA comes in. I don't think it's just there to look nice. Too bad they don't have much of a description on catagories of SLA, even though they garantee it, it doesnt state what compensation is given depending on cercumstances.
Posted by Nullified, 03-31-2005, 01:18 PM
wow. 88 pages and I missed it all. I guess that's because I didn't seem to have one problem with my server at the planet that is monitored every 5 minutes. No services down, websites up. What happened?????????
Posted by Gint, 03-31-2005, 01:21 PM
I agree whole-heartedly. Keeping everyone in the dark for so long wasn't a good start, but let's see where they go from here.
Posted by Nick Jones, 03-31-2005, 01:21 PM
Heh Nullified .. it's a shame, 5 posts per minute what happend noone knows, we are waiting for official explanation. Someone did put a explanation (sort of): "At approximately 4:15AM CST, a pair of redundant Powerware 500KVA UPS units failed creating a power failure in section B of our DLLSTX2 datacenter. Emergency teams were deployed within minutes and power was restored within minutes but intermmittment power outages did occur until 6:45AM CST. Powerware, JT Packard, and electricians are currently onsite with over 100 Planet technicians working to resolve the issue. We do not anticipate any further outages . A formal RFO will be released once the team debriefs. We apologize for all issues that has caused."
Posted by P_totha_G, 03-31-2005, 01:22 PM
Well, I can relate with all of you. I am with Dinix/WebHostPlus. We have been down since Friday night - came up for a short time Monday afternoon and Tuesday morning, but went back down and have been there ever since. No word, no resolution ...... the whole thing stinks!
Posted by Mach3, 03-31-2005, 01:22 PM
yeh, atleast they coud of told us when they plan on making a full anouncement.
Posted by AKWS, 03-31-2005, 01:23 PM
Basically.. fuse in 1 UPS blew, pushed load to second UPS which overloaded it and blew the breakers. Then they cycled back and forth like that for 3 hours I think. Mean while everyone complaining their servers down. Lots of rude comments, a few guys cried, a mention or two of Ev1 and multiple phone calls. All in all it was exciting, emotional, interesting, spectacular .. I don't know how to sum it up any better than that!
Posted by Killa200, 03-31-2005, 01:23 PM
I feel sorry for section B of Dallas 2, troubles galore today, I guess next time the NOC's get bored at the wee hours of the morning, they won't decide to have a pissing contest next to the UPS's again, will they,
Posted by movielad, 03-31-2005, 01:24 PM
I don't know about you guys, but I've been prepared for something like this. That is to say, I haven't spent the best part of eight years in the web hosting and film industry not to know how to make sensible precautions in case of something happening with the server falling over (hardware or software related), the datacentre or whatever that causes some kind of outage. I've seen Telehouse have power failures, knocking out equipment for some time. Even EV1 had issues if I remember rightly, but admittedly they were very quick to get it resolved. Power outages can and do happen even if you think you're covered. It shouldn't be that way, but there you go. I've another dedicated server with NAC, which runs secondary MX and DNS. Secondary MX makes it so much easier and quicker to spit mail back to the original machine when it comes without it having to rely on the original sender's mail server keeping retrying at ever increasing intervals. As for web serving, although I don't have any web sites on my SM server, I do have a web server on my NAC box, though, and whose contents are mirrored to the SM server every night. In theory, I could just switch between boxes if one fails by using a third party utility or script on ANOTHER box that keeps monitoring the situation between the two other boxes. And the thing about this is that I'm doing all this just for me and some friends - not for commercial or personal wealth. I've got redundancy for a personal server (well, servers) and yet it makes me wonder why some people here, with paying clients, do not take the same kind of precautions and have their own redundancy procedure and contigency plans if this sort of situation occurs. Also by the way that some people are reacting, they're making out that this is a tragedy far greater than that such as the Asian Tsunami or some other major incident in which people and whole families were wiped out. Please get a grip. Yes, The Planet could have made better use of their resources and post (in a new thread to avoid having to go through all xx pages of this thread to find something from them) about what is happening. However, what's done is done. I'm sure lessons will be learnt from this and life will go on. Besides - I'd suspect that The Planet have an SLA with the power supply people, and will be compensated appropriately which will, in turn, go to those that require compensation for their loss of service. Well - that's my thoughts on all of this anyway. M. Last edited by movielad; 03-31-2005 at 01:27 PM.
Posted by AKWS, 03-31-2005, 01:25 PM
Quote from http://forums.servermatrix.com/viewt...r=asc&start=30 "Location: DLLSTX2 Severity: Level 5 Description: At 4:09AM CST, a power failure occurred in a redundant pair of Powerware UPS units feeding power to section B of the DLLSTX2 datacenter. The power failure was caused by a faulty fuse in UPS unit B-1. As the load transferred to UPS unit B-2, the spike in the load created an overloaded breaker and UPS B2 also lost power. This resulted in a power outage to the main power distribution unit feeding section B of the datacenter floor. Emergency teams were notified and the power was restored within 20 minutes. The power continued to cycle until 6:45AM due to the faulty fuse and the inability of the redundant UPS units to remain in bypass mode. Customers may have noticed several power cycles during this time period. Powerware, JT Packard, and electricians found the problem and replaced the faulty hardware and fuses. At 6:45AM, all electrical service was restored to normal and the NOC team began to bring all servers back online. The technical staff is currently placing a console on all servers to verify server restarts. Customers with operating systems that require a file check may have experienced extended downtime during the file check. Powerware and JT Packard will continue to monitor the UPS systems for the next 24 to 48 hours. The Planet does not anticipate any further outages. " Check SM forums for up to date "info"
Posted by sightz, 03-31-2005, 01:26 PM
If your UPS can't handle the spike from the load being transferred to it, isn't that just bad design or using the wrong gear? One would assume they test these things, right?
Posted by Mach3, 03-31-2005, 01:30 PM
AKWS , it can be simple put as "AGONY"(spelling I hope is right)., just sheer agony. ------------------------------------------------------- Also I'm quite sure TP has business insurance and other types of insurance to help cover this. I think this forum is starting to timeout on me...
Posted by AKWS, 03-31-2005, 01:30 PM
Thats what I thought. They should be balanced so that maximum load on 1 UPS is total load produced by the servers on that breaker. The goal of 2x UPS is redundancy, not being able to handle 2x the load. Bad design I would guess. Almost looks like they were trying to jam as many servers in a DC as they could.
Posted by Killa200, 03-31-2005, 01:31 PM
Yes, but when you get down to the nitty gritty, this "spike" their referring to is quite a big spike... your talking about taking 1/2 of B's grid, and all of a sudden surging it over to another breaker (in this case B-2) I dunno how TP has its electrical infrastucture setup, but its a big big datacenter, and that "spike" could be within the area of 1,000 - 7,000 amps of solid 3 phase goodness. Wether its rated to hold that constant or not, equiptment ages, and esp electrical equiptment, as it ages, handles spikes less and less. The instintanious rush of that much current probably caused the heat or electromagnet protection system in the breaker of B2 to falsely go off.
Posted by jagarco, 03-31-2005, 01:33 PM
Is working now. It seems that was the case too I guess. Thank you Jagarco
Posted by Mach3, 03-31-2005, 01:36 PM
hmm, Killa200 ever heard of california blackout and east coast black out in the past year. Seems like everone is having power surge problems....
Posted by Killa200, 03-31-2005, 01:38 PM
could be that too, i was just going in the route of a grid failure. If a surge happend (most likely did once B1 failed), then the current inrush would rise as the voltage dropped that split second, causing the breaker to pop as well due to the heat trigger
Posted by Mach3, 03-31-2005, 01:41 PM
The main problem is why they didn't have backup power. On the surge they could have switched to backup power and then handle the surge or divert the serge.
Posted by DCM, 03-31-2005, 01:43 PM
I was inside the DC 2 weeks ago and yes, they test. In fact, the run 30 minutes on generators each month to verify the system is stable. Of course, no amount of controlled testing can make you 100% sure when it happens for real. As for the lack of information, I can see how it would be difficult to get out a statement. Any statement would have been no better than the lack of one. Be real and consider that it is very difficult to make exact predictions in situations like this one. One would also not want to open the legal door by having some overnight NOC engineer post his best guess on what it is and when it would be back up. People needed to be alerted, then drive down to the data center, and look over the issue and come to a concensus on what was the real issue. There is no way you can start with wild guesses or people speculating what is the problem. Sometimes doing nothing is doing something. Everyone will bitch and yeah, some of you will lose customers that obviously are clueless and want 100% uptime on a budget. If uptime is that important, then those people should invest in their own DC and staff. However, you also own some of the blame for making them think you could deliver 100% uptime. Until you own and are responsible for everything in the operation, you cannot make any promises no matter how good the track record of the companies involved have been in the past. Am I defending SM/TP? No, I am just making a realistic obersvation. Nobody but a handful of people will ever know the exact events that transpired. If you don't trust them to handle it, then the simple thing is to pack up and move somewhere else and wait for their disaster. My advice, instead of pissing about it here, why don't you do something productive and educate your customers that problems can arise outside of anyone's control. Maybe if some of you were not ranting like lunatics, then maybe your customers might be more understanding.
Posted by Killa200, 03-31-2005, 01:50 PM
Mach3, i'm sure they had backup power, and they had ample power from the extrenal grid itself in this case as well, its just the reality that "unhooking" and "hooking" close to 7ka of equiptment of anykind to an already in use grid is gonna cause a mess to equiptment thats not used to doing that regularly. Mistakes happen, and electrical equiptment is nowhere near 100% perfect. They can't pop on backup power to a grid that has a breaker popped after the backup / normal grid connection.
Posted by Mach3, 03-31-2005, 01:50 PM
Well no company can be perfect, you never know a meteor might hit a datacenter and shread it to bits. Is it their fault, not really or were they suppose to predict that? If this happened now, it was bound to happen sometime so hopefully they will improve their datacenter in repsonse to such a senario.
Posted by movielad, 03-31-2005, 01:54 PM
DCM - well said, sir! M.
Posted by woolly, 03-31-2005, 02:04 PM
My server is still down - it doesn't come up from a reboot. What the hell is taking the planet so long?
Posted by SarcasticDwarf, 03-31-2005, 02:05 PM
perhaps the 1000 other people having various problems?
Posted by 6PS-Jake, 03-31-2005, 02:10 PM
Good Point!
Posted by djdiz-e, 03-31-2005, 02:34 PM
server still down
Posted by gorsan, 03-31-2005, 02:36 PM
mine came back
Posted by caker, 03-31-2005, 02:41 PM
For those that are still down, I suggest filling out a ticket and giving them a phone call. One of my cabinets has multiple power feeds -- two of the feeds, and the servers on them, have been up for 5+ hours, but the feed that powers the switch had to be reset by ThePlanet. A simple phone call and everything's back in order.
Posted by Inthrive, 03-31-2005, 02:43 PM
Why does everything think it's helpful to post "my server is down", "this server is up, but this one is down', "no it's back down again" ??? I mean, if you're a customer of TP/SM, you know which servers are down and which are up, what good does it do to tell everyone else? I looked through 40 of the 90 pages of this thread and there doesn't seem to be any facts just a bunch of "they must have this..." or "that couldn't have happened". This thread: http://www.webhostingtalk.com/showth...hreadid=390360 seems to have an offical response from TP/SM about the situation. Personally, I feel it's unacceptable to have redundant systems that cannot handle the entire load if one unit fails. What's the point of having a redundant unit then? We had a similar situation happen at a different datacenter and it's very frustrating. Last edited by Inthrive; 03-31-2005 at 02:50 PM.
Posted by MrMcGoo, 03-31-2005, 02:49 PM
It is so wonderful to see what a vibrant and active group of parasites. Now this is only directed to the SLA crowd... Remember all those times you called up SM and TP and they did some support work "above and beyond" their stated terms.... Remember ALL those 100% uptime months.... Remember all the new and helpful FREE features they have been offering like Remote reboot ports, etc. Ok...Now lets hope next time you need HELP on your server SM/TP treats YOU as well as you are treating them right now...They take the Service limitations conditions and ask for your CC # before they so much as glance at the rack your server is located. I have said it before and I will say it again...Unless the situation is totally abusive (tech support has been unresponsive to a critical situation for a day or so) we at Beachcomber will never request compensation by way of a SLA. If you do not have enough goodwill built up with your customer base to survive a 3-5 hour downtime you HAVE FAR better things to be doing than filling out a SLA claim from believe me...
Posted by Fair Dinkum, 03-31-2005, 02:52 PM
Well, this is a Discussion Forum. Some like to DISCUSS things, and share information. If you need the info directly, you can go to the SM forums and read it there.... though I am afraid you will find Discussion on their Discussion forums too. Cheers. Glenn
Posted by Inthrive, 03-31-2005, 02:58 PM
I was merly noting that there is an excess of posts stating nothing more than "my server is down" or "my server is up." Obviously this is a discussion forum and information about "my server on this IP block is down/up" is helpful. But simply complaining that your server is down (or up) really isn't. How useful can a discussion like that be?
Posted by snJohn, 03-31-2005, 03:07 PM
WHT is mail bombing me! Over the past 3 minutes I received over 60 "Reply to post 'ThePlanet down?? [merged]'" e-mails.
Posted by Killa200, 03-31-2005, 03:13 PM
well if you don't want WHT to do so, there is always the checkbox in control panel to turn that off, under options. Its something like "Notify me when a thread i have posted in is replied to",
Posted by snJohn, 03-31-2005, 03:13 PM
I have ;-)
Posted by snJohn, 03-31-2005, 03:15 PM
hmm, a lot of it is messages that were replied to hours ago
Posted by sirius, 03-31-2005, 03:20 PM
Sorry to burst your bubble, but posting "My server is down" is not a discussion. It's a way for the kids here to increase their post counts and over inflate a thread, where some useful information could have existed. Right? Sirius
Posted by 6PS-Jake, 03-31-2005, 03:24 PM
Your post is right on the money and quite admirable.
Posted by Fair Dinkum, 03-31-2005, 03:24 PM
You are not "bursting my bubble". I am quite old enough to defend my bubble against the negative energy of youth. Quite possibly, but how do *we* judge what constitues "useful information"? If it is useless, report it and let the Mods sort it out. I found much of it to be useful while tracking the downed servers I have VS the servers I have that never went down. Each to his own... I sorted out what I found useful, and easily ignored what was not useful to me.
Posted by zsindhu, 03-31-2005, 03:27 PM
Hear, hear!
Posted by cjscott69, 03-31-2005, 03:32 PM
Yes, for me the "my server is up" posts were helpful to track the progress and get an idea of when I may be up.
Posted by Douglas, 03-31-2005, 03:42 PM
I posted this on the SM forums, I'm posting it here. FYI, I have 6 servers, it appears that only two of them were affected, though I didn't lose power, just connectivity. Okay, I may be going out on a limb here, but I'm one of those kinds of people that would rather see services restored FIRST and then communication later. Was the timing handled poorly? Perhaps. Am I upset about this? Not in the least. We've all become so dependent on technology, we forget that things do break down from time to time. I'm glad that this happened when it did, before my workday. Yes, I've got a couple customers that were irked that their sites were down, however, once I explained to them that it was an issue dealing with a power outtage, they calmed down considerably. Why? Because they understand that NOTHING is 100% failsafe. TP/SM, here's one customer that will NOT be exercising his SLA option. I've been extremely pleased with the hardware and network for over a year now. Customer Service.... eh, different story, lol. Just a suggestion for the future, though... When you experience an area wide outtage, consider having a phone dialer machine that will play a pre-recorded message for all affected customers, or route email to alternate email addresses. That's the ONLY suggestion that I've got. I want to thank the TP/SM staff for their diligence in trying to get through this issue. I certainly do NOT envy them for being in this situation at all. Hat's off to y'all!
Posted by neXolt, 03-31-2005, 03:43 PM
Ok thats nice but i think people would rather know the number to your sales department i suggest your company hires another 50 sales reps for the next few weeks
Posted by sightz, 03-31-2005, 03:45 PM
I, for one, greatly appreciated the posts where people provided details of their servers that were down. For example, knowing that several people on the 69.93.x.x IP block could ping but not connect any other way was useful information which I forwarded to the datacentre, just in case our extended outage was network-related. It is comforting to know when others' boxes are still down and when they are coming back up. It allows me to include lines on our status site such as "10:11am: - We are advised that servers within our network segment are coming up." If you don't like the discussions, just put everyone on ignore (or turn off your computer). PS. My server is back up.
Posted by 00000, 03-31-2005, 03:53 PM
What to do in case when you use Hyperspin monitoring whose service is hosted on ThePlanet?
Posted by universal2001, 03-31-2005, 03:53 PM
Well in my dealings with TP I know they are STRICT on policy. If you ask them to do simple things or things they will not do it because of "policy". Examples: Cancelling an order because I used a different credit card than the one I had on file with them. I asked them to just change the credit card number on the newest order to the one located in ORBIT and their reply was they can't because of POLICY. I had to REORDER. Seems pretty stupid, it would of taken them a few seconds to change the CC#, but noo they don't have time for personal customer service, they forced me to re-order the server, enter in all the details AGAIN and waste another 10 minutes of my life doing that. Then the second issue.. I updated my credit card and guess what? Billing address didn't match, oops! Sorry we cannot accept this card. I offered to call them up to verify the card details. Nope, either I change the billing address on the credit card or they can't accept it.. Despite being a customer of theirs for over 16+ months. Third issue. Ask them to move a hard drive to another server within the same NOC. Sorry no can do, POLICY. Out of all providers I have used, ThePlanet is the most institutionized, everything is done to policy. So now I guess its POLICY to claim what they promised, yet fail to deliver. Some of you's are giving too much credit too ThePlanet, saying "things go down, we can't treat them so harshly". YES that's true, but do you REALIZE why they got to #1 and become one of the world's biggest dedicated providers??? Because they promise 100% uptime on everything and market themselves as the best SO clients expect that. If you can't deliver it then don't claim it.
Posted by Mach3, 03-31-2005, 04:29 PM
universal2001, there is a reason why for all those and they link to security reasons. How will calling them up confirm the credit cards billing address?
Posted by Mach3, 03-31-2005, 04:41 PM
Well you might have a reason to fill one out. for example, assuming you were sending a mailing through your smtp server wich requires almost a wholde day to send. Then in the middle of it it gets shut off and broken so you have to start all the way over and miss a day. or how about even worse, you run a bank and someone needs to fill out their bills on the last day. As he goes to your site its down and he cant fill them out. I dout that would make even some of the loyal customers not very happy. or even you yourself provide an sla. everyone has their reasons why they would want the sla. Ofcourse even taken the acount, but the sla wont be of help ether the ebst you can get is a free month and 300$-500$ isnt that much. As I said before, they probably have business insurance so the sla wont eb a big deal to them. The sla wont get your customers back, but it works liek a privacy policy. Its in place to assure thigns work the way they do. No one forces them to give 100% sla, why do they give it, sicne it gaisn custemer assurance and in providing it you must back it. The problem was those that think about the SLA before their server is even down, thats what really sad. Last edited by Mach3; 03-31-2005 at 04:53 PM.
Posted by bidder, 03-31-2005, 05:03 PM
It is not in their terms and conditions to compensate...please PM me to advise me what to do next?? I have lost a lot of clients because of today's events!!
Posted by Mach3, 03-31-2005, 05:16 PM
no, it isnt in their terms to compinsate. Theya re not responsible for damage or loss off clients..but...as stated under their TC 100% Uptime SLA - Power 100% Uptime SLA - Network since the power did die out. They are responsible for compensationg for the power failure. this again does not account for damages. Whether you lost customers or your server was down and it didnt effect you. You gain the same compensation as everyone else.
Posted by woolly, 03-31-2005, 05:24 PM
The RAID 1 array of my server broke due to the power failure. Filing a ticket in orbit for a tape backup restore but it is taking ages for them to respond to the ticket. Any suggestions to speed this up?
Posted by alexrobonlin, 03-31-2005, 05:27 PM
Not really - apart from be patient- they've had a helluva day! Alex
Posted by mustakal, 03-31-2005, 05:28 PM
Just FYI, Powerware is a Finnish company.
Posted by Hostex Australia, 03-31-2005, 06:30 PM
Orbit is currently 'unavailable'. Probably to stop the huge flow of complaint tickets rolling in....
Posted by StanSQL, 03-31-2005, 06:35 PM
The Planet's website works with me.. Last edited by StanSQL; 03-31-2005 at 06:40 PM.
Posted by woolly, 03-31-2005, 06:41 PM
it's back up now. I wonder why it was down.
Posted by Mark L, 03-31-2005, 06:44 PM
I dont have time to read the entire 1,400 replies, but I dont know if this has been posted.
Posted by StanSQL, 03-31-2005, 06:48 PM
Theirs your answer...
Posted by inteltechs, 03-31-2005, 06:49 PM
call them.
Posted by alexrobonlin, 03-31-2005, 06:50 PM
lol, indeed
Posted by whmcsguru, 03-31-2005, 06:53 PM
More times than anything else, beginning at 749 CST this morning. Seems to me that's like the pot calling the kettle black here. Ever hear of "be nice"? Just because someone wants credit for their unexcusable downtime doesn't mean they're a "parasite". In fact, it only means that they want what they have coming to them. Has TP somehow performed excellent service? I don't think so. Over the years they've gotten worse, rather than better. Have they gone above and beyond? If you can call keeping customers out of the loop for 5 hours "above and beyond", then you need to look at your definition of above and beyond, because it's truly not. If you can call suddenly changing client's ip addresses for no reason at all "above and beyond", then you , again, need to look at your definition of above and beyond, because it's truly not. The service and support from TP has been less than par up to this point and should, indeed be reprimanded , rather than praised. Do you praise someone who does half the job they're supposed to, or someone who constantly says "I don't know what the problem is, but it's on your end", or someone who takes 10x as long as they SHOULD to do their jobs? I certainly don't, but that's exactly the quality and style of workmanship that's been expected from TP / SM techs for a very long time, and they have yet to fail to deliver that.
Posted by MFrag, 03-31-2005, 07:11 PM
Geez, I have been an EE for 18 years on Wall St. We have had outages even before 9/11, which we remained up orderly until we failed over to our Jersey City DR site. It lost power during the east coast black out, our UPS's took the load, and the Generator kicked in with 8 days of fuel. We always design on Wall St for "Busy Hour". aka "Worst Case". We never load our UPS's over 40%, and we balance our racks have a feeder from each. hence if a failure occurs due to any scenario, the other takes the 80% load, and yes, the spike. The professional systems we use are built to take that spike transition. In fact they have 4 phases of conditioning, and enough current to smooth out that spike so the breakers don't trip. I am talking about power systems with breaker switches that are individual components themselves running upwards of $50K each. In 18 years on Wall St, I have seen may blackouts. We suffered in the corporate offices of one brokerage when they lost power for a week when Con-Edison had a major fire in an east side power plant. They re-routed power by ripping up Wall Street and pulling power off the grids that ran the than lively World Trade Center. We immediately went on backup generator. I designed the system such that the UPS's would power the DC, and power the pump in the basement which pumped up diesel oil to the generators on the roof of One Financial Sqaure. The building shook as the generators kicked in... all 52 stories of it. If we under powered, or exceeded budget, yea, we would trip the breakers (which we don't want to do, as you need to replace them. (yea.. $50K disposible's). So we build it right. If this was designed properly, I find it very hard to believe they should fail over a single grid. In terms of a data center, 500KVA is a drop in the bucket really.
Posted by woolly, 03-31-2005, 07:37 PM
Because of the power outage, they say I need an OS reload. (It's The Planet's answer to everything when they can't even be bothered to rebuild a RAID 1 array - whats the point of having RAID 1 then?) Filed a reload ticket, called them a million times, and it has been 10 hours! All I hear from them is "OS reload takes 24-48 hours." Since this was THEIR MISTAKE, the least they could do is help get the server up ASAP. Instead, they make me wait 24 hours for a damn OS Reload. From my experience, The Planet has the worst tech support ever and they don't treat customers very well.
Posted by Ramprage, 03-31-2005, 07:46 PM
Is it me or did someone just edit all the orignal statements made and change them when they originally said the issue was resolved but now they don't say that anymore...... wtf
Posted by BigBison, 03-31-2005, 07:47 PM
16 times out of less than 1400 posts. Don't even get me started on "what's the link to the webcam." Regarding the size of this thread, it's nothing to be proud of. This thread has got to have the lowest signal/noise ratio in the history of WHT. Even if you like to know the exact minute-by-minute status of everybody's server, 80% of the posts in this thread are noise. What's really appalling, though, is not how many people started their own thread on this, but how many of those were within the first five minutes of this downtime. Nobody's impressed with how quickly you can start an outage thread about your host. It doesn't seem like some of you even attempted to contact The Planet before opening a thread here. An SLA is not a guarantee of 100% uptime. I agree with The Bear: Yes, downtime sucks, and this could have been handled better. But, given the unlikeliness of this outage occurring, what's wrong with that first statement they released taking 3 hours? It explained what they were up against. Later, a full explanation came out, after things were fixed. This thread shouldn't have been more than 10 pages, given the actual amount of useful information in it. Instead of running around like a chicken with its head cut off, take the time to read others' posts. If something doesn't make sense, ask someone who has been around here and TP for a while (like coight) to explain it. Why am I ranting like this, even though I'm not a TP customer? Well, if you've ever seen my traceroutes in a thread discussing Cogent, you may recall that all my outbound packets on my personal broadband connection route from Denver to Dallas/Ft. Worth. Whenever there are issues at TP, all traffic through DFW is impacted, and there are many things I can no longer accomplish from my location due to timeouts through DFW, including the work I had planned for this morning. I came here to attempt to ascertain the facts of the situation. To the handful of you or so who made useful posts, thank you. Dual failure in the power supplies like that, yeah that could take all day. It should not have taken me until 10am to get to LegalEagle's post! That was 3 hours after it was made. If this thread hadn't been so damn noisy, I would've known the situation within minutes of that post. For future reference, threads like these are not the place for random speculation about FBI raids, router malfunctions, or even the power problem it turned out to be. When a thread is exploding like that, please resist the urge to make frivolous posts. You only compound the problem, because then nobody wants to read the thread before asking the same question for the umpteenth time. Allow those of us who read before posting to cut to the chase, PLEASE! Last edited by BigBison; 03-31-2005 at 07:51 PM.
Posted by alexrobonlin, 03-31-2005, 07:56 PM
BigBison. I look forward to seeing you so calm, unattached, and unemotional as you suggest you would be when you, your customers, half your business, and thousands of other datacentre customers are in the same position - namely offline... Alex
Posted by PierreB, 03-31-2005, 08:01 PM
there is still downtime for others? Cause we are up since 8:30AM CST Last edited by PierreB; 03-31-2005 at 08:04 PM.
Posted by whmcsguru, 03-31-2005, 08:06 PM
There's bound to be problems resulting from this, like the individual who can't get their raid array up. Sad, but raid takes to outages just about as good as mysql does apparently
Posted by Mark L, 03-31-2005, 08:09 PM
I dont want to read 1,400 posts of "Still down...." "My server has been down for 53 minutes....." "call them!!!!" "what's there number" So, I just posted it. No harm done.
Posted by PierreB, 03-31-2005, 08:09 PM
Well let's hope they all get back up again.
Posted by Mach3, 03-31-2005, 08:15 PM
well maybe the boards should adopt a standard that during situations liek this put something like [IMPORTANT OFFICIAL ANNOUNCEMENT] on top of the psot or soemthing, so that you can jsut use search on posts liek these and get the crucial information.
Posted by alexrobonlin, 03-31-2005, 08:21 PM
Indeed. Much better than people coming on here, ranting, hours after the incident concerned, who isn't even a customer of the TP, telling all the affected people off! Give us a break! We've had an extremely stressful day, and don't need people like you telling us what to do. Sorry to tell it like it is, but hey, I'm only human. If you don't like reading through the whole lot - don't; simple! Alex
Posted by Killa200, 03-31-2005, 08:24 PM
MFrag, i'd have to agree with you in terms of that entire statement, and yet again say that i didn't have any part of TP's electrical system. I am just going over what could happens.
Posted by BigBison, 03-31-2005, 08:24 PM
I was online when this outage occurred, a post I had typed at the IWDN forums wouldn't send. That's unusual, so I did a traceroute and found I couldn't get there. Same with their host, Hostgator. I also know another prominent, active WHTer is at The Planet, Jose - TMS. Traceroutes to them, and The Planet, were flaky as anything and I'm on a direct line to DFW, so to speak. At that point, I knew that as soon as there was anything to know, those two members would post to this thread. As it was, their posts were drowned out in a sea of chud. My point is, this is an awesome place to lurk until verifiable facts are posted. If you need to tell your customers something, start with the truth -- "I'll let you know as soon as I hear" -- which is what HG and TMS did. When official word is forthcoming, it takes it no more than a split-second to arrive here from credible, longtime sources. Be patient like everyone else who isn't posting to the thread. We're talking TP here -- there are thousands of members on this board who were affected and did not feel compelled to post. Look at the thread views. As soon as there's anything to report, it will be, then nobody's inbox gets bombed, either! Right. I mirror my data to two physically separate locations with failover provided by UltraDNS. Wouldn't have affected me, if one of those servers were in TP. Regardless, I have been in that situation before -- I was with Dreamhost last Mar. 29th, when they were down due to DDoS for an entire day. I had to put up with this type of thread then, too, but I never lost my calm. Had a few things to say to them, but not for a few days after the incident so they could get their feet back under them. I agree with you wholeheartedly. You'll notice my scolding actually excused you from that. Doesn't matter that I'm not a customer. This problem had me effectively offline throughout. Made for some stress here, too. Maybe I'm upset like this because people seem so giddy about participating in a thread that's just too large, too fast, to be of much use. Maybe we can break the record for biggest thread, w00t! I've said my piece and I'm done here, goodbye. Last edited by BigBison; 03-31-2005 at 08:36 PM.
Posted by 6PS-Jake, 03-31-2005, 10:04 PM
First things first, exercising the SLA (even if you can) would just hurt yourself and The Planet. We all know this stuff just breaks sometimes. It really doesn't matter what happened.... The fact is it did. All we all can do is learn from the disaster and move on. Two things I would hope the Planet would learn are these: 1. Work on the UPS structure to prevent this double failure thing from happening. 2. Most important, develop some damage control procedures that are customer focused. This post is rather large, but I just feel like its important. I would like to make a comparison. I was on a missle Submarine for a 6 year tour. As you might imagine, COMMUNICATIONS is key to Damage Control on a submarine. The Damange Control Processes on a sub are rock solid and I would like to explain some things about them. A similiar system should be in place with the Planet. First, when a bad thing happens on a submarine a Rapid Response Team fights the casualty (lets use fire for example). The Planet has a Rapid Response Team much like the NOC crew. However, immediately as the RRT deploys a force of briefed, outfitted to combat the casualty, people are deployed to relieve the wore out RRT. Basically, the RRT reacts quickly and on the fly with people who are available while the "true" team (like the Daytime Engineers and Technical Heavy hitters) get to the scene. Pretty much this is what the Planet has already done because its a natural action. On the boat (what we call a submarine), we have a coordinator called a Damage Control Assistant. This guy takes care of directing teams and COMMUNCATION to key points around the submarine. The Planet SHOULD station a person on these forums and their own to provide precise updated information. I am sure some was filling the roll of the DCA during the problem at the planet. You would have to admit that if someone from the Planet was here showing they had the customers interest at heart, making comments, reassuring people, that things would have been better and you could have at least given your customers an explanation. If you think about it, that person who is here is heading off MANY calls to the datacenter. This thread would not be a trillion pages, and you would have a better feeling about the situation. I could go on and on about other submarine positions and things we did to protect ourselves. The point of my babbling here is simple. ITs not a question of would I rather have a person here communication or solving the problem. I think considering the size of the Planet and the obvious amount of customers that visit here, they should consider stationing someone here to help communicate to the customer. This is BOTH communication and working on the problem. This communication would achieve the following: 1. We would have a clue what was happening 2. Many Inbound calls would be cleared up 3. We would have answers to give our customers 4. The Planet would give a vibe that they care and are working. (Please dont flame that... I know they care, but some people here have questioned it) 5. We would be less stressed out 6. Finally, its GOOD business to keep your clients informed when in a disaster. Well, I'll jump off my ol soap box. I hope this babbling post made sense. You may not agree. If I "owned" The Planet, I would make sure that someone was here to communicate. I hope this post wasn't out of line. MOOD STAMP: Concerned, but hopeful we can learn.
Posted by akuo, 03-31-2005, 10:49 PM
I think it all comes down to lack of functional redundancy across many levels. They have stated they couldn't post an update because their office was knocked out. For a company running X thousand servers across three? datacenters, I find it a little hard to swallow that they couldn't find a way to get internet access. Nevertheless... no redundancy. Likewise the redundancy measures in place with their UPS system obviously failed. Bingo, lights out. Likewise their own website and support systems were knocked out by this, being hosted in the same datacentre, with the same power supply. Where's the redundancy there? Hopefully this is a wake-up call.
Posted by dbuckley, 03-31-2005, 10:58 PM
I was hit by this outage as my domains primary mail server is running on a Linode in Theplanet. I understand a thing or two about data centre availability, its quite hard to design a system that operates 24/7 x 25 years. It looks like Theplanet is no more able to cope when things go wrong than just about every other co-lo site I've ever investigated, and it looks like it is for exactly the same reasons. Network and Systems Resilience The power system that failed was rated at 500KVA, which for a large co-lo centre, is not nearly enough. 500KVA is a small datacentre. So my guess is (and this is supported by the wording of the communique) only part of the facility lost power. I could not get theplanet.com to resolve during the outage, with none of the listed nameservers being available. Given that only part of the Theplanet empire had collapsed, how is it that there were no DNS servers available. Best practice for DNS RFC 2182 states I dont think this is news to anyone. What seems far more disturbing is that many servers did not lose power, yet had no connectivity. Thats just bad implementation of networking, schoolboy errors. Power Arrangements Theplanet states the usual marketing hyperbole about how great and resilient it's infrastructure is. As do almost all big co-lo operators. Well, heres the news - every year a few of these facilities have partial or complete power failure. I have a server in Dallas, and a server in a co-lo facility in London, and this year both have been juiceless. I hope this is a dry year. Theplanet, like most co-lo facilities, thinks that having a proprietry, probably modular, redundent UPS with a backup generator will ensure continuous power. It wont. In fact, we've just had a demo. It doesnt. What that system is in reality is a chain of single points of failure, all the way to the plug in the back of the host, and it will from time to time fail. Thus, this will not be the last outage of this nature at Theplanet. In the world of affordable hosting, infrastructure failure is an unfortunate fact of life, and these facilities will, in whole or in part fail. Thus to stay up, you need to be geographically dispersed. Just having DNS and mail elsewhere is a massive improvement when the lights go out... I hope Theplanet will learn something from this experience, but they are unlikely to try and fix the root cause, as sorting out the power properly will be a bit disruptive to the centre, and will itself probably result in more downtime than just staying with what the have and hoping.
Posted by sprintserve, 04-01-2005, 01:18 AM
We are running businesses, not charity. If you can't meet the SLA, don't offer it.
Posted by 6PS-Jake, 04-01-2005, 01:55 AM
Sprint, I can understand what you are saying.... but Its like having a car wreck just one that does minor damage.... You decide, I have Insurance, so you turn in the claim. 3 months after the claim is paid your insurance doubles because you had a claim. I think its a smart business decision to not pursue the SLA because I am sure the price of services will go up. ThePlanet has two choices after paying back: 1. Lose money 2. Increase prices to recoop I think we can either recoop SLA money now and pay dearly for it later or not press and hope to salvage. I guess when it boils down to it, the decision to press the SLA is an individual choice. I am going to press our management not to pursue. This is not a "they tried" or "charity" decision for me. This is a business choice because of what I fear will follow.
Posted by sprintserve, 04-01-2005, 02:06 AM
On the contrary, the punitive effect should make TP work harder at it, since they know there's a cost to failure. At the same time, if they can't afford to pay out based on just 1 incident, they have no right to be in the business. End of the day, it's a simple thing. Many of the hosts will have SLAs to their own clients. You would want to pay out to your clients, some of which is needed to keep them. You wouldn't want to lose money yourself would you? If they have to increase prices, and they remain competitive to the market, no issues there. But if they can't, it's a simple thing nowadays to choose someone else who is competitive. Lastly I seriously doubt it will cause them much damage in the long term financially.
Posted by eric1234, 04-01-2005, 02:48 AM
So when can we receive compensation from TP?
Posted by Douglas, 04-01-2005, 03:08 AM
http://forums.servermatrix.com/ Third forum down, fourth forum down. It explains how to deal with the SLA.
Posted by kmike, 04-01-2005, 03:12 AM
I've yet to see some mention of this "event" on their site, not to say about apology to clients. So far only one PR was coughed up to be posted here (which hardly goes as official statement), and then copy-pasted into support tickets related to downtime. Are they trying to downplay the issue, or just waiting for dust to settle?
Posted by whmcsguru, 04-01-2005, 03:20 AM
They've already made the same post that they have here on their forums. It's not too likely we'll see anything more from them.
Posted by alexrobonlin, 04-01-2005, 03:23 AM
Probably both... Alex
Posted by eric1234, 04-01-2005, 03:24 AM
So does i need to open a support ticket to claim the compensation? Or i just wait until TP mention it?
Posted by alexrobonlin, 04-01-2005, 03:42 AM
Open a support ticket. As SprintServe said; we're running businesses, not charities, and if TP can't get it right, then they need to do something about it... Alex
Posted by bheka, 04-01-2005, 03:44 AM
Simple open a ticket, give server information and downtime and let them do the maths. Additional Info Below
Posted by timdorr, 04-01-2005, 04:40 AM
It's listed on the home page of Orbit with a complete explination. It's not hidden if you're a customer...
Posted by bidder, 04-01-2005, 04:43 AM
so what was the uptime percentage after yesterday's outage?
Posted by Neosmith, 04-01-2005, 04:49 AM
What is sad of all is that this happened. They are making lots of money from the datacenters and it is not like they can't afford to fix the problem. This really should have NEVER HAPPENED. They should have planned for the "What If this, or what if that" which to me is just good thinking. I know you might not be able to plan for everything but you should at least plan for the very worst. Besides I also wonder if they have done any type of test of there backup systems, etc within the last year or less? They should be testing their backup systems at LEAST once a year. Hopefully they have learned from this but then again maybe not. Maybe they are just crossing their fingers hoping that "the next time it happens" that they can get everything up sooner. Everyone that was affected by this SHOULD claim the SLA. This is all just my 2 cents!!!! Last edited by Neosmith; 04-01-2005 at 04:53 AM.
Posted by eric1234, 04-01-2005, 05:05 AM
Ok, i had open a ticket already. Do you think they will compensate me for the 5hrs downtime? Or they will try to deny everything and say that its not covered?
Posted by Neosmith, 04-01-2005, 05:23 AM
My ISP gets its backbone from ThePlanet and I just did a trace route and it looks like my connection goes through one of the dc that was down. Correct me if I am wrong. 2 216.185.103.1 1.216-185-103.reverse.theplanet.com 17.954 xxx 3 67.18.171.9 9.67-18-171.reverse.theplanet.com 139.579 xxx 4 12.96.160.41 dsr1-1-v2.dllstx2.theplanet.com 780.512 xxx 5 70.85.127.65 dist-vlan21.dsr3-1.dllstx3.theplanet.com 25.841 xxx 6 70.85.127.2 ae0-0.ibr2.dllstx3.theplanet.com 27.192 xxx 7 216.200.6.233 gw-cr2.dfw2.us.above.net 34.570 xxx 8 64.125.29.17 so-4-1-0.mpr2.iah1.us.above.net 30.757 xxx 9 216.200.251.162 t289.216-200-251-162.iah1.us.above.net 33.091 xxx
Posted by hosttone, 04-01-2005, 07:31 AM
Yes, we have claim the SLA and they told us need 7-10 days.
Posted by Jalberts, 04-01-2005, 10:08 AM
Is this happening again? Chronichosting, which apparently goes through TP is having the exact same problem as yesterday at this time, DNS not resolving for any of my domains, but the IPs connect fine. This is just plain ridiculous. Once is bad enough, two days in a row is an insult!
Posted by whmcsguru, 04-01-2005, 10:19 AM
Nothing showing up as down here, might contact support @ chronic and see what the issue is. WHT really isn't their helpdesk, or anyone's, for that matter
Posted by Jalberts, 04-01-2005, 10:27 AM
I would except their "support" site has had an announcement that they are upgrading and to try back in 15 mins, since yesterday morning when this fiasco first happened. They have no phone support that I'm aware of, and John logged off of MSN Messenger after I inquired about the outage this morning. That's the service you get for $200 a month...
Posted by Joshua, 04-01-2005, 10:28 AM
Bad comparison. Here in NJ, there's a no-fault law for auto insurance, where if your car is in an accident that you didn't cause, your auto insurance rates do not go up. If the downtime isn't your fault, your rates shouldn't go up by making a claim (and they won't with a company the size of ThePlanet). If a company is so confident in their infrastructure that they can offer a 99.9% or 100% SLA, and back it up with high enough compensation, you have ever right in the world to claim it... You were promised high uptime, and didn't get it. If the company doesn't have the financial resources to pay the SLA claims and not raise rates, well, they shouldn't be operating 3 datacenters with 20,000+ servers. Go ahead and claim the refunds if you want.
Posted by whmcsguru, 04-01-2005, 10:41 AM
Jalberts It's not John's fault that he logged off, MSN is acting shaky right now, nobody can log in or out. Make sure your dns is actually running, if you're running a ded. server, and if that doesn't help, feel free to contact me via pm and I'll see if I can help out. Sorry, forgot about the signature thing
Posted by Jalberts, 04-01-2005, 10:49 AM
I'm logged in to MSN right now, doesn't seem shaky at all to me. I run a ded server, but DNS is hosted on CH's server. I know they're on PEI in Canada, maybe a lobster trawler cut the cable to the mainland, who knows. Though their main site is up. Sorry, it's just incredibly frustraing. I went to CH expecting something different, but it's the same old crap. Lack of support, lack of caring about customers.
Posted by dolay, 04-01-2005, 10:50 AM
You said that before me , it is going to be 4th time that we again apply for a lifetime free credit to our customers by upgrading customers bw. and disk quoatas just because of TP fault. I have applied for a SLA for our Xeons that costs us $2k/Mo. Most of you may have servers for your personal usages but also theplanet customers here are running a web hosting business and offering %99.x uptime guarantee by trusting theplanet guarantee. What i have to say if theplanet do what they offer (i even dont dream they will) we will forward the refunds to those servers customers. Last edited by dolay; 04-01-2005 at 10:53 AM.
Posted by T595net, 04-01-2005, 11:02 AM
Is MSN hosted at TP? Last edited by T595net; 04-01-2005 at 11:08 AM.
Posted by whmcsguru, 04-01-2005, 11:05 AM
Don't take my word for it, take msn's. There is an issue with MSN at the moment, second time this week, but there is a problem. If there really is a problem, email john and he'll respond quick enough. WHT isn't the support forum for any one specific host, this issue has been resolved for a good 24 hours now.
Posted by Jalberts, 04-01-2005, 11:14 AM
I wasn't trying to get support here, was just trying to find out if the same thing was happening again, since my symptoms are exactly the same as yesterday. I did email John, waiting for a response.
Posted by photodesign, 04-01-2005, 12:44 PM
99.3% ?
Posted by jmweb, 04-01-2005, 12:45 PM
Jeff, all your domains are resolving fine ( which appears to be an issue with Apache on your server). Have you tried restarting apache? Also I do apologize for the delay in responding to your ticket that was opened an hour ago, and as of 20 minutes ago it was updated. Also if you feel we don't care certainly let me know how and why you feel this way as we certainly do care however we can not troubleshooting. I was very shocked when I read this as I do not understand why you would post such a thing. Last edited by jmweb; 04-01-2005 at 12:58 PM.
Posted by tuwebfacil, 04-01-2005, 01:26 PM
Hi, Someone has a complete log of this downtime? How long it was, when started and when it was fixed? And the amount of downtime % And other information for file a claim with them Would be very helpfull this information to file the claim. thanks in advance !
Posted by dolay, 04-01-2005, 04:35 PM
one of our filled server go poweroffline 4 hours ago and still TP doesnt response to us have anyone issues like that still?
Posted by wds716, 04-01-2005, 04:47 PM
Is it normal for a provider to NOT proactively let the customer know of an outage? I really hope this is not an industry standard.
Posted by WebsiteNames, 04-01-2005, 07:22 PM
Can anyone confirm any continued Planet issues as we are being hosted at another WHT hosting service and not only were we down for an extended period of time yesterday, But we have also been down since 12 pm est this afternoon. I contacted there tech support and they state that its the planet again but I have spoken with a few people and they have all indicated that my host must be using yesterdays outage as an excuse, I was lucky enough to take some screen shots of my WHM server staus pages just before this latest outage and they show some very concerning problems. Could someone please comment on the latest staus as I really dont want to mention any names before all the facts are known but I will say that I have had the worst hosting experice over the course of the last 45 days and everytime its blamed on either ThePlanet or backbone upgrades ect ect ect. I run a business and I cannot afford to keep being told excuses 99.9% uptime has no bearing I just need my sites online, Not offline for days at a time and be given the option of money back for the month and move hosting or a token 10% disk space and bandwidth That means nothing to me compared to the lost clients we endure when our sites are offline for this lenght of time. Last month we lost a recorded 6238.00 in revenue due to these outages Febrary we lost a recorded 4790.00 in revenue due to outages, I have online hotel booking websites and these revenues are derived from daily sales by hour, We have approx 80 sites that are all still down Someone please help as my host blames ThePlanet is this true??? I can post the screen shots of server status if anyone is interested.
Posted by sightz, 04-01-2005, 07:33 PM
I have had 100% uptime for the last several months from the Planet, with the exception of yesterday's events. Haven't seen any interruptions today. If you are losing $5000 per month due to outages, perhaps you should have some redundancy - like multiple servers in different locations.
Posted by WebsiteNames, 04-01-2005, 07:41 PM
I have documented all the problems I have had for the past 2 months and its always the planets fault. Thats a great idea you submit, I find it very disturbing that you stat that your service has only been interupted once meanwhile my host is down an awfull lot of time and blaming the planet
Posted by dolay, 04-01-2005, 08:02 PM
it is now more than 8 hours one of our Xeon server power offline behind yesterday trouble, which is hosting more than 600 hosting account that sold commerically by Newista and if this last more than 10hours i may also declare here all ThePlanet's dirty laundries at past with their details , it is enough our customers shamed us on WHT due to Planet faults.. i already cancelled all of our servers with ThePlanet. I cant stand that , it is enough to be f#[k{d by TP faults Last edited by dolay; 04-01-2005 at 08:06 PM.
Posted by nothingg, 04-02-2005, 12:05 AM
The Planet Support Ticket 660263PLNT was Updated Please go to https://orbit.theplanet.com/nav_supp...cket_id=660263 to view and/or edit the support ticket. Ticket Details: Queueales Status:OPEN SummaryLA Claim Details:A RFQ has been received. from: Master User: zentz Details: Today around 4:10AM CST my server had a downtime which was going over for hours and finaly came back up at around 9:30AM CST So i would like to request the SLA credit for the downtime that damaged my business with alot of consenquences! Thanks, Zentz ------------------------------------------ (eclark-03/31/2005 11:56:34): We need for you to list to the best of your ability the time the server/s (include server name & IP) went down and when they came back online. Once we receive this we will sent to management for investigation. SLA claims can take up to 7 days to provide to provide the solution. Thank you and we apologize for the problem. ServerMatrix Sales sales@servermatrix.com 1-800-377-6103, Option #4 -------------------------------------- (c20623zent-03/31/2005 13:19:03):here are my monitoring system logs Thu Mar 31 06:08:23 EST 2005 1st HTTP Check failed Thu Mar 31 06:08:53 EST 2005 2nd HTTP Check failed Thu Mar 31 06:08:53 EST 2005 Server Down Thu Mar 31 06:44:23 EST 2005 1st HTTP Check failed Thu Mar 31 06:44:54 EST 2005 2nd HTTP Check failed Thu Mar 31 06:45:15 EST 2005 Apache Down Thu Mar 31 06:45:15 EST 2005 Apache Down Thu Mar 31 07:04:15 EST 2005 1st HTTP Check failed Thu Mar 31 07:04:16 EST 2005 2nd HTTP Check failed Thu Mar 31 07:04:16 EST 2005 Server Down Thu Mar 31 07:07:01 EST 2005 1st HTTP Check failed Thu Mar 31 07:07:31 EST 2005 2nd HTTP Check failed Thu Mar 31 07:07:31 EST 2005 Server Down Thu Mar 31 07:11:11 EST 2005 1st HTTP Check failed Thu Mar 31 07:11:41 EST 2005 2nd HTTP Check failed Thu Mar 31 07:11:41 EST 2005 Server Down Thu Mar 31 07:15:39 EST 2005 1st HTTP Check failed Thu Mar 31 07:16:01 EST 2005 Server resumed proper functionality Thu Mar 31 07:32:12 EST 2005 1st HTTP Check failed Thu Mar 31 07:32:33 EST 2005 2nd HTTP Check failed Thu Mar 31 07:32:33 EST 2005 Server Down Thu Mar 31 07:36:08 EST 2005 1st HTTP Check failed Thu Mar 31 07:36:38 EST 2005 2nd HTTP Check failed Thu Mar 31 07:36:38 EST 2005 Server Down Thu Mar 31 07:38:06 EST 2005 1st HTTP Check failed Thu Mar 31 07:38:37 EST 2005 2nd HTTP Check failed Thu Mar 31 07:38:37 EST 2005 Server Down Thu Mar 31 07:42:10 EST 2005 Reboot Request Ticket submitted to ThePlanet Thu Mar 31 07:45:36 EST 2005 1st HTTP Check failed Thu Mar 31 07:46:06 EST 2005 2nd HTTP Check failed Thu Mar 31 07:46:06 EST 2005 Server Down Thu Mar 31 07:46:36 EST 2005 Server still Down Thu Mar 31 07:50:19 EST 2005 1st HTTP Check failed Thu Mar 31 07:50:49 EST 2005 2nd HTTP Check failed Thu Mar 31 07:50:49 EST 2005 Server Down Thu Mar 31 07:51:19 EST 2005 Server still Down Thu Mar 31 07:57:36 EST 2005 1st HTTP Check failed Thu Mar 31 07:58:06 EST 2005 2nd HTTP Check failed Thu Mar 31 07:58:06 EST 2005 Server Down Thu Mar 31 07:58:37 EST 2005 Server still Down Thu Mar 31 08:00:41 EST 2005 1st HTTP Check failed Thu Mar 31 08:01:11 EST 2005 2nd HTTP Check failed Thu Mar 31 08:01:11 EST 2005 Server Down Thu Mar 31 08:01:41 EST 2005 Server still Down Thu Mar 31 08:04:13 EST 2005 1st HTTP Check failed Thu Mar 31 08:04:43 EST 2005 2nd HTTP Check failed Thu Mar 31 08:04:43 EST 2005 Server Down Thu Mar 31 08:05:13 EST 2005 Server still Down Thu Mar 31 08:08:36 EST 2005 1st HTTP Check failed Thu Mar 31 08:09:06 EST 2005 Reboot Request Ticket submitted to ThePlanet Thu Mar 31 08:09:07 EST 2005 2nd HTTP Check failed Thu Mar 31 08:09:07 EST 2005 Server Down Thu Mar 31 08:09:37 EST 2005 Server still Down Thu Mar 31 08:11:05 EST 2005 1st HTTP Check failed Thu Mar 31 08:11:36 EST 2005 2nd HTTP Check failed Thu Mar 31 08:11:36 EST 2005 Server Down Thu Mar 31 08:12:06 EST 2005 Server still Down Thu Mar 31 08:13:50 EST 2005 1st HTTP Check failed Thu Mar 31 08:14:20 EST 2005 2nd HTTP Check failed Thu Mar 31 08:14:20 EST 2005 Server Down Thu Mar 31 08:14:50 EST 2005 Server still Down Thu Mar 31 08:15:11 EST 2005 1st HTTP Check failed Thu Mar 31 08:15:41 EST 2005 2nd HTTP Check failed Thu Mar 31 08:15:41 EST 2005 Server Down Thu Mar 31 08:16:11 EST 2005 Server still Down Thu Mar 31 08:19:02 EST 2005 1st HTTP Check failed Thu Mar 31 08:19:32 EST 2005 2nd HTTP Check failed Thu Mar 31 08:19:32 EST 2005 Server Down Thu Mar 31 08:20:02 EST 2005 Server still Down Thu Mar 31 08:21:19 EST 2005 1st HTTP Check failed Thu Mar 31 08:21:49 EST 2005 2nd HTTP Check failed Thu Mar 31 08:21:49 EST 2005 Server Down Thu Mar 31 08:22:19 EST 2005 Server still Down Thu Mar 31 08:23:15 EST 2005 1st HTTP Check failed Thu Mar 31 08:23:41 EST 2005 1st HTTP Check failed Thu Mar 31 08:23:45 EST 2005 2nd HTTP Check failed Thu Mar 31 08:24:11 EST 2005 2nd HTTP Check failed Thu Mar 31 08:23:45 EST 2005 Server Down Thu Mar 31 08:24:15 EST 2005 Server still Down Thu Mar 31 08:24:11 EST 2005 Server Down Thu Mar 31 08:24:41 EST 2005 Server still Down Thu Mar 31 08:32:16 EST 2005 1st HTTP Check failed Thu Mar 31 08:32:46 EST 2005 2nd HTTP Check failed Thu Mar 31 08:32:46 EST 2005 Server Down Thu Mar 31 08:33:17 EST 2005 Server still Down Thu Mar 31 08:33:32 EST 2005 1st HTTP Check failed Thu Mar 31 08:33:40 EST 2005 1st HTTP Check failed Thu Mar 31 08:34:02 EST 2005 2nd HTTP Check failed Thu Mar 31 08:34:10 EST 2005 2nd HTTP Check failed Thu Mar 31 08:34:02 EST 2005 Server Down Thu Mar 31 08:34:32 EST 2005 Server still Down Thu Mar 31 08:34:10 EST 2005 Server Down Thu Mar 31 08:34:40 EST 2005 Server still Down Thu Mar 31 08:37:07 EST 2005 1st HTTP Check failed Thu Mar 31 08:37:37 EST 2005 2nd HTTP Check failed Thu Mar 31 08:37:37 EST 2005 Server Down Thu Mar 31 08:38:07 EST 2005 Server still Down Thu Mar 31 08:42:08 EST 2005 1st HTTP Check failed Thu Mar 31 08:42:38 EST 2005 2nd HTTP Check failed Thu Mar 31 08:42:38 EST 2005 Server Down Thu Mar 31 08:43:09 EST 2005 Server still Down Thu Mar 31 08:47:09 EST 2005 1st HTTP Check failed Thu Mar 31 08:47:39 EST 2005 2nd HTTP Check failed Thu Mar 31 08:47:39 EST 2005 Server Down Thu Mar 31 08:48:09 EST 2005 Server still Down Thu Mar 31 08:52:04 EST 2005 1st HTTP Check failed Thu Mar 31 08:52:34 EST 2005 2nd HTTP Check failed Thu Mar 31 08:52:34 EST 2005 Server Down Thu Mar 31 08:53:05 EST 2005 Server still Down Thu Mar 31 08:57:09 EST 2005 1st HTTP Check failed Thu Mar 31 08:57:39 EST 2005 2nd HTTP Check failed Thu Mar 31 08:57:39 EST 2005 Server Down Thu Mar 31 08:58:09 EST 2005 Server still Down Thu Mar 31 09:02:14 EST 2005 1st HTTP Check failed Thu Mar 31 09:02:44 EST 2005 2nd HTTP Check failed Thu Mar 31 09:02:44 EST 2005 Server Down Thu Mar 31 09:03:14 EST 2005 Server still Down Thu Mar 31 09:07:13 EST 2005 1st HTTP Check failed Thu Mar 31 09:07:43 EST 2005 2nd HTTP Check failed Thu Mar 31 09:07:43 EST 2005 Server Down Thu Mar 31 09:08:14 EST 2005 Server still Down Thu Mar 31 09:12:10 EST 2005 1st HTTP Check failed Thu Mar 31 09:12:40 EST 2005 2nd HTTP Check failed Thu Mar 31 09:12:40 EST 2005 Server Down Thu Mar 31 09:13:10 EST 2005 Server still Down Thu Mar 31 09:17:07 EST 2005 1st HTTP Check failed Thu Mar 31 09:17:37 EST 2005 2nd HTTP Check failed Thu Mar 31 09:17:37 EST 2005 Server Down Thu Mar 31 09:18:07 EST 2005 Server still Down Thu Mar 31 09:22:13 EST 2005 1st HTTP Check failed Thu Mar 31 09:22:43 EST 2005 2nd HTTP Check failed Thu Mar 31 09:22:43 EST 2005 Server Down Thu Mar 31 09:23:13 EST 2005 Server still Down Thu Mar 31 09:32:11 EST 2005 1st HTTP Check failed Thu Mar 31 09:32:41 EST 2005 2nd HTTP Check failed Thu Mar 31 09:32:41 EST 2005 Server Down Thu Mar 31 09:33:12 EST 2005 Server still Down Thu Mar 31 09:37:09 EST 2005 1st HTTP Check failed Thu Mar 31 09:37:39 EST 2005 2nd HTTP Check failed Thu Mar 31 09:37:39 EST 2005 Server Down Thu Mar 31 09:38:09 EST 2005 Server still Down Thu Mar 31 09:42:06 EST 2005 1st HTTP Check failed Thu Mar 31 09:42:36 EST 2005 2nd HTTP Check failed Thu Mar 31 09:42:36 EST 2005 Server Down Thu Mar 31 09:43:07 EST 2005 Server still Down Thu Mar 31 09:47:10 EST 2005 1st HTTP Check failed Thu Mar 31 09:47:40 EST 2005 2nd HTTP Check failed Thu Mar 31 09:47:40 EST 2005 Server Down Thu Mar 31 09:48:10 EST 2005 Server still Down Thu Mar 31 09:52:09 EST 2005 1st HTTP Check failed Thu Mar 31 09:52:39 EST 2005 2nd HTTP Check failed Thu Mar 31 09:52:39 EST 2005 Server Down Thu Mar 31 09:53:10 EST 2005 Server still Down Thu Mar 31 09:57:15 EST 2005 1st HTTP Check failed Thu Mar 31 09:57:45 EST 2005 2nd HTTP Check failed Thu Mar 31 09:57:45 EST 2005 Server Down Thu Mar 31 09:58:15 EST 2005 Server still Down Thu Mar 31 10:02:17 EST 2005 1st HTTP Check failed Thu Mar 31 10:02:47 EST 2005 2nd HTTP Check failed Thu Mar 31 10:02:47 EST 2005 Server Down Thu Mar 31 10:03:17 EST 2005 Server still Down Thu Mar 31 10:05:15 EST 2005 1st HTTP Check failed Thu Mar 31 10:05:16 EST 2005 1st HTTP Check failed Thu Mar 31 10:05:18 EST 2005 1st HTTP Check failed Thu Mar 31 10:05:34 EST 2005 1st HTTP Check failed Thu Mar 31 10:05:45 EST 2005 2nd HTTP Check failed Thu Mar 31 10:05:49 EST 2005 2nd HTTP Check failed Thu Mar 31 10:05:49 EST 2005 2nd HTTP Check failed Thu Mar 31 10:06:04 EST 2005 2nd HTTP Check failed Thu Mar 31 10:05:45 EST 2005 Server Down Thu Mar 31 10:06:16 EST 2005 Server still Down Thu Mar 31 10:05:49 EST 2005 Server Down Thu Mar 31 10:05:49 EST 2005 Server Down Thu Mar 31 10:06:20 EST 2005 Server still Down Thu Mar 31 10:06:20 EST 2005 Server still Down Thu Mar 31 10:06:04 EST 2005 Server Down Thu Mar 31 10:06:34 EST 2005 Server still Down Thu Mar 31 10:13:47 EST 2005 Server resumed proper functionality Thu Mar 31 11:52:39 EST 2005 1st HTTP Check failed Thu Mar 31 11:53:10 EST 2005 2nd HTTP Check failed Thu Mar 31 11:53:10 EST 2005 Server Down i am also uploading you RTG graph... please let me know if you need any other info thanks, zentz ------------------------------------------ (eclark-03/31/2005 13:20:47): Again, please be aware of the 7-10 time frame for the SLA be investigated and determined as appropriate for your account. Thanks, ServerMatrix Sales sales@servermatrix.com 1-800-377-6103, Option #4 -------------------------------------- (c20623zent-03/31/2005 13:26:05):i forgot to include my server name and ip so here it is: Label: cakic IP: 67.19.220.122 i am aware of the 7-10 time frame that may occur. thanks, zentz ------------------------------------------ (jreinis-04/01/2005 14:06:50): I do apologize but D4 datacenter was not effected by the outage. This SLA claim has been denied. Resolution: Please do not reply to this email. If you have additional information, please update the ticket using the link to the support system provided. If the support ticket was closed, and the issue was not resolved to your satisfaction, please feel free to open another ticket and reference this ticket number. Thank you, The Planet
Posted by dolay, 04-02-2005, 01:37 AM
Ticket Number: 662640PLNT Ticket Type: Sales Server: Fixxxer(70.84.249.52) Status: OPEN Opened By: Newista Web Services Re-Assign To: The Planet (THEPLANET) Bandwidth Monitor (bwmonitor) Deleted User (dolay) Deleted User (nw-kamil) Deleted User (nw-lukas) Deleted User (nw-ewa) Newista Web Services (c16452newi) Remigiusz Jackowski (nw-remik) Robert R (er2er) Summary: SLA request on Fixxer- 70.84.249.52 Existing Details: Hardware Object: ALL THE SERVERS 67.19.32.36 + 70.84.249.52 + 67.19.190.164 + 70.85.48.4 I am sure you can understand why i listed all of our servers. Look, this was not a suprise for me, it is not my business, your business and i dont care the reason.. You know 67.19.32.36 which is reclaimed today by you and which was a filled server that was cancelled by me due to your untolerable hardware faults at past. You know what happened yesterday while we were carrying 600 hosting accounts that was commercially sold by NEWISTA WEB HOSTING SERVICES BY 99.99% UPTIME GUARANTEE from this server to our new own server as it seen on NEWS of http://www.newista.com/ YOUR WHOLE NETWORK OUTGATED FOR MORE THEN 6 HOURS! And we have not carried the 600 commercially sold account from this server due to enermous outgate and now i am in damn problems cause we cant get the last backups from this server to the new server(outsie of TP), all of our customers complaining everywhere because of how a trouble service we are ... Dont say me the reason or any other explain. This is what happened to me. You are crashing with me as a 4 Dual Xeon server owner BUT I AM CRASHING HERE WITH 2000 HOSTIN SITE OWNERS WHICH ARE FROM THOSE SERVERS! I am asking of this 4 servers all services MARCH PAID AMOUNT! Thats it. Only pay or not, yes or no? P.S. I am really getting more and more sick while i am opening accounting related tickets here, I am going to cancell the other 3 dual xeon servers too but i am really afraiding that you again bomb a bingo while we are transferrring those servers... BTW. i know you are not going to refund anything, if i really know you from past 2 years but emergencely confirm this ticket. God knows when and where will i use this ticket for whats? To your knowledges, Ramazan Dolay Newista Founder http://www.newista.com/ ------------------------------------------ (mhall-04/01/2005 15:50:20): Theere is only one server on this account: Fixxxer 70.84.249.52 Are you asking for an SLA on this server? The Planet will not honor false SLA claims, or servers on different accounts. If you are uninterested in the reason for the outage, that's fine, but you must follow procedure. I have dealt with enough of your requests for credit to know that you are well aware of the procedure. Please supply us with the approximate times this server went down and came back up. We will base your credit off of the information you supply to us. -------------------------------------- (c16452newi-04/01/2005 16:52:52):As you know (I asked TP support to not this), Newista has two orbit accounts: 1- c16452newi have 67.19.32.36 + 70.84.249.52 2- c22324mahm have 67.19.190.164 + 70.85.48.6(VPS .4) One with servers 67.19.32.36 + 70.84.249.52, the second orbit's with 67.19.190.164 + 70.85.48.6 I will open a similiar ticket on the second account, if you prefer? The downtime of a big part of ThePlanet network has not only been approved by an official information from ThePlanet, but also by 1.500 of posters and 80.000 viewers at http://www.webhostingtalk.com/showth...hreadid=390336 on WHT which is also known by you and i am a member of this Board since 2003 with user "Dolay" also you approved that by your "Legal Eagle" user. Your network was cut like knife at the mentioned hours so it does not need to approve seperate reports per server. You know that better than us. Also you have deleted yesterdays https://orbit.theplanet.com/nav_hard...stgroup=C16452 statistics that shows 8 hours of outgate on whole network(no connection at all that was coming or outgoing from your network). SLA CREDIT REQUEST: 1) server Xeon, 67.19.32.36. + 70.84.249.52 + 67.19.190.164 + 70.85.48.6(VPS .4) The server 67.19.32.36. have been reclaimed today, however, it has been paid for full March (see accounting) and the outage appeared in March. 31.03.2005 - 4 AM CST, approx. 8 hours (http://www.webhostingtalk.com/showth...43#post2982643) Subject to service and power SLA, this was a 'well-known' power downtime (bad fuse, according to TP official information). *****ADDITONAL server Fixxxer, 70.84.249.52 more credit request: Downtime history: https://orbit.theplanet.com/nav_hard...stgroup=C16452 Fri Apr 01 11:01:19 CST - about 6 hours. Server is in "Power: offline" state and does not react to "Power On" since then. Because of that, it should be covered by Power SLA. Also REF: Ticket Number: 663318PLNT NOTE: Do you want me to open same SLA request ticket from second orbit account for - c22324mahm have 67.19.190.164 + 70.85.48.6(VPS .4) ? NOTE2: Fixxxer Total downtime exceed 7 hours for today behind yesterday outgates by right now REF: Ticket Number: 663318PLNT and i have really bad headache. And i am swearing myself for the day i choose to start a web hosting business with you ThePlanet. Ramazan Dolay Newista Founder http://www.newista.com/ -------------------------------------- (c16452newi-04/01/2005 17:09:10):Extra Note: I have said to the 400 web hosting resellers of Newista that i will refund all the refunds you made for those outgates and thats what i am going to do if you are going to do. ------------------------------------------ (mhall-04/01/2005 18:58:51): I can only use this ticket here to address the SLA claim you are filing on the server attached to this account. For the other servers on your other account, I will need a separate SLA credit request ticket. We are working on Fixxer right now, but the function of this ticket needs to be on the SLA claim on this server. I cannot grant an SLA for "yesterdays trouble" for this server, as this server is not located in the affected datacenter. -------------------------------------- (c16452newi-04/01/2005 19:11:38):"I cannot grant an SLA for "yesterdays trouble" for this server, as this server is not located in the affected datacenter. " so why the world cant not reached to the 600 sites that hosted on this server yesterday? Look "mhall" pray that any of our customers dont able to think yesterday there were Google quarterly(one a 3 month) Toolbar PageRank update which is still going an those fixxxer resellers whom are all webmasters and sites are crawling with a blank page while Google Dancing..., The outgate is the seenable part of the iceberg, there are hell of problems at the bottom due to this problem. I promise you i am going to cancel all of our server one by one as we carry them to the another reliable datacenter.. My mental health and wallet is totally destroyed. I know you are not in a pleasure of those troubles but say to your Boss , if he is selling and growing too fastly than he had to gain more supporters and more ehanced systems for ThePlanet or you will always mess with those kind of problems... ------------------------------------------ (mhall-04/01/2005 19:18:51): Fixxer will be back online within the hour. What was the cause of your outage yesterday? -------------------------------------- (c16452newi-04/01/2005 19:50:52):eh ), i think aliens made a magic things and all of our ThePlanets accounts(2k websites) gone unreachable, really gaving a refund is like a giving a part of your body for you right? rotfl, i really never be able to understand why some people are so greedy.. however i know the meaning of the honor which will i kept at the end of my life whatever cost me. - i am not going to interest with any orbit jobs or you will make me loose all of my kindness here, our cPanel specialist will prepare you all the details that you already know better than us,
Posted by WATYF, 04-02-2005, 12:39 PM
What's the status of this whole thing??? My hosting company uses The Planet and every single one of my websites went down starting Thrusday night, and are STILL not up yet... is The Planet still recovering, or is my host the problem? Any info would be appreciated... I can't even get to my host's website... I'm totally in the dark about this. WATYF
Posted by WebsiteNames, 04-02-2005, 12:54 PM
I would like to say to Dolay, Thanks for dealing directly with me during this problem and yes our google ranks will be effected by this shut down, I am a newista user that has had problems but I also understand that there is circumstances beyond their control. I just wish that my business wasnt effected in such a dramatic way. Dolay you operate a decent business and I for one appreciate the fact that you deal directly with me. I would like to suggest that you install a live chat with your customers as this could calm alot of tempers having to wait for answers really sucks in these situations. I would hope that we ( customers) will not have to deal with anymore outages for the year as we have lost business, ( not your fault this time ) we have lost google indexing for another quarter, and we have lost valued customers. I would also just like to state that Dolay does go out of his way for the most part but if theplanet is going to disrupt your business in this very menacing way your threat of moving would probably be the best solution, because if theplanet keeps screwing up they will continue to loose customers such as yourself. I still am hosted at newista and look forward to a trouble free remainder of 2005
Posted by patrick24601, 04-02-2005, 01:04 PM
Look at how Dolay talks to the planet and pay attention to the details of his ticket. Mr Dolay seriously lacks professionalism in interpersonal and communication skills. He may stroke you via customer support, but I'd hate to be a client of his and get on his bad side. I don't care how upset he is or what his SLA is. And if he leaves the planet it will be one more PITA they do not have to deal with. WATYF: There are no more planet outages right now? Have you picked up the phone and called ?
Posted by PierreB, 04-02-2005, 01:11 PM
Most of us are back online, I heard a few still have troubles with their servers. Try doing a whois on your host and see a phone number, call them if possible.
Posted by dolay, 04-02-2005, 07:03 PM
"seriously lacks professionalism " what professionalism are you talking about? what professionalism can you show after you do not slept more than 50 hours? what professionalism can you show after your servers taken down more than 14 hour? what professionalism will you show after the Planet kidding such like below that (mhall-04/01/2005 19:18:51): What was the cause of your outage yesterday? ???? What the hell monkey business is this? While the world knowing that the whole network cut like a knife all the incoming and outgoing data ... and look at that reply ThePlanet is denying and claiming that the servers not effected and asking me why our servers outgated yesterday.... I see they doing this to everyone Who can approve tha outgate and tell me whos SLA CLAIM ACCEPTED BY THEPLANET HA? Yes i gone littlebit rude and kiddy but this not due for the last 2 day only right, it is the saving of a year that past with untolarable faults... Since that time i kept the silence , since that time we pay to our own customers due theplanet faults... Anyways here are the results, we do not loose any customers and i do not think we are going to loose any customer because of those outgates too but what about ThePlanet? They already lost 2 $2k/Mo. payer customer like me. How you do business is too important but how a reliable and promised business you are doing is the most important thing. This is not the way i like to do business but hose are the naked results. Thats it, i wish our private wont be discussed here more and people discuss here WHAT SLA are they getting according to thePlanet 100% POWER and NETWORK UPTIME GUARANTEE
Posted by whmcsguru, 04-02-2005, 07:26 PM
What good will attacking each other in this thread do? None Keep in mind that the only "bad guy" here is theplanet, not the individual users. Posting tickets in here, won't do a bit of good either. Those should be kept between yourselves and theplanet. If it comes down to it, yes, you need to use those in a court of law, but this is neither the time, nor the place for that. Keep in mind that we're all human here. everyone had a rather LARGE problem, and SOME are still trying to recoup from that. I agree, theplanet dropped the ball, and in a big way. They should have had more juice going through the backup machines, but, sadly they don't, didn't, and WON'T fix the problem. They won't, because to fix the problem would cause much more downtime than would be expected, as someone else said.
Posted by WebsiteNames, 04-02-2005, 11:44 PM
I for one am happy at the way Dolay represented us clients. Because no matter what I know myself I have left a few heated emails for the tech support staff at newista and I am only one client, if i multiply that by 100 I too would be pissin mad at the planet. I continue to support newista even thru the problems I have had over the past few months. As a matter of fact it was dolay that approx a month ago personally moved all my hosting from one server to another server at the time was most reliable, It just so happens that this server i was moved to was effected this time around. Thats customer service. I am not saying I forgive for the loss my business has suffered but let me say this I would rather a pitbull fighting for me to get my sites back online as opposed to a poodle I still use newista even after the problems of the past few months and i have been assured that I should not encounter any more outages for the year, as my business is getting ready for peak season and we will book about 50-70 k rooms in the next few months and I hope we never have to face these problems again. Best Regards S.Alek President and CEO Canadian Niagara Hotels
Posted by Lem0nHead, 04-02-2005, 11:48 PM
if you have a high loss (ie more than $50/hour) when your site is down, i'd recommend that you set redundant DNS
Posted by WebsiteNames, 04-03-2005, 10:47 AM
Lemonhead, We are in the process of doing so as another member also recommended the same.
Posted by DeveloperMCT, 06-15-2005, 02:40 AM
I just signed up with ServerMatrix / ThePlanet. I am interested to know how ServerMatrix kept their garantee?
Posted by beachcompcom, 06-15-2005, 09:06 AM
Never did get anything official from them, or a refund.
Posted by richyc, 06-15-2005, 09:08 AM
We had multiple credits applied to our account as per their SLA
Posted by kneadingu, 06-15-2005, 11:06 AM
Likewise.
Posted by DeveloperMCT, 06-15-2005, 11:12 AM
this is the only time this has happened, right? ....the downtime, I mean
Posted by crucialx, 06-15-2005, 11:15 AM
No, they have had downtime in the past. They did a network upgrade (moved all their incoming providers from one DC to another - an entire rework of their network) without notifying their customers. During that period there was about 24 hours of packet loss. Those are the only 2 main outages I can think of, most of the time their network is fairly reliable.
Posted by Lem0nHead, 06-15-2005, 11:22 AM
their outages are really rare, but you can forget the refund they usually will give 15%, even if they should give 40% and use excuses like "you have too many tickets", "you've changed your credit card", etc... to justify things that makes no sense IMO
Posted by DeveloperMCT, 06-15-2005, 11:25 AM
For the price, are there any other hosts that are better? I would hope Network Solutions and Godaddy would be reliable, but they come to mind only because how big they are. And I'm sure those two are much more expensive than SM
Posted by Patryk, 06-15-2005, 11:32 AM
I got two servers at LayeredTech.com (in Savvis DC) and I'm very happy with it. They had some routing problems last week, but any serious outages since I remember. They have also excellent support.
Posted by Lem0nHead, 06-15-2005, 11:35 AM
I like 15minuteservers
Posted by crucialx, 06-15-2005, 11:36 AM
Size does not mean reliability You may want to do a search on Network Solutions and Godaddy first.
Posted by DeveloperMCT, 06-15-2005, 01:08 PM
well, i guess this was just a notion.. big profile = public = making sure no mistablse so stock prices dont crash. if a host goes down for an hour during peak time, i'd be upset. If a host goes down for like 12+ hours, I'm looking for a new host. I know we live in a imperfect world, but this IS not acceptable in the hosting world. when you're on a boat going across the atlantic and drown with the ship, you don't say "crap happens", you ask the cruise line about procedures and policies in place for life boats and life jackets. These large hosts have generators in place and all, but it seems these dont work when you really need them, then you have to ask why aren't they tested once or twice a month to make sure they do work when you need them. ...sorry, I was forseeing someone playing devil's advocate saying "no host is perfect" (which isn't what I'm trying to refute anyway, but you get the idea). maybe 99.9995 uptime is much much more rare and unrealistic than me and my clients realize. when you're a one man or family owned company new clients can have a hard time getting over "are they running this out of their house" or maybe this guy isn't as reliable as a large programming or hosting company. An outage for a small company makes the client think it happened because you're small, not because "crap happens"
Posted by ffeingol, 06-16-2005, 12:01 AM
It took a bit (based on the volume of the claims) but we got all our credits. The key (we've found) is to have good monitoring and being able to document the outage. We have our servers monitored by several (respected) outside monitoring services. We simply suppled the down/up times from those services and requested the refund. Frank
Posted by bugsoft, 06-17-2005, 06:20 AM
Complete network down? I have about 15 servers with then in dc1 and dc2 and all down. Orbit down. Can anyone confirm?
Posted by bugsoft, 06-17-2005, 06:33 AM
Actually it's not all the network that is down but rather some pops I can connect to all my servers from other servers but there is something wrong with the savvis pop I guess Works: root@server1 [~]# traceroute www.theplanet.com traceroute to www.theplanet.com (216.185.111.40), 30 hops max, 38 byte packets 1 177.90.32.216.reverse.layeredtech.com (216.32.90.177) 0.466 ms 0.438 ms 0.441 ms 2 10.1.3.21 (10.1.3.21) 2.237 ms 5.232 ms 0.594 ms 3 216.39.69.225 (216.39.69.225) 0.565 ms 0.461 ms 0.563 ms 4 chr2-g1-0.FortWorthda1.savvis.net (216.39.64.41) 0.466 ms 0.410 ms 0.498 ms 5 dpr1-so-1-0-0.DallasEquinix.savvis.net (204.70.134.17) 1.745 ms 1.723 ms 1.704 ms 6 aer1-po10.DallasEquinix.savvis.net (204.70.134.14) 1.477 ms 2.702 ms 1.657 ms 7 208.175.175.42 (208.175.175.42) 1.683 ms 1.650 ms 1.605 ms 8 dist-vlan31.dsr3-2.dllstx3.theplanet.com (70.85.127.30) 1.710 ms 1.684 ms 1.655 ms 9 dist-vlan22.dsr1-2.dllstx2.theplanet.com (70.85.127.76) 2.203 ms 2.003 ms 2.128 ms 10 car1-1-v1.dllstx2.theplanet.com (12.96.160.11) 1.999 ms 2.214 ms 2.032 ms 11 37.216-185-111.reverse.theplanet.com (216.185.111.37) 2.286 ms 2.256 ms 2.286 ms Doesn't work: Tracing route to 216.185.111.37 over a maximum of 30 hops 9 72 ms 75 ms 73 ms Savvis-3561.tlh.giga.parix.net [198.32.247.97] 10 73 ms 73 ms 75 ms bcs2-so-1-1-0.Parisprx.savvis.net [204.70.193.138] 11 152 ms 155 ms 149 ms bcs2-so-3-0-0.Washington.savvis.net [204.70.192.125] 12 163 ms 167 ms 169 ms dcr2-so-7-2-0.Atlanta.savvis.net [204.70.192.57] 13 189 ms 189 ms 195 ms bcs2-so-2-0-0.Dallas.savvis.net [204.70.192.70] 14 191 ms 187 ms 187 ms bcs1-so-6-0-0.Dallas.savvis.net [204.70.192.49] 15 189 ms 187 ms 189 ms dpr1-so-0-0-0.DallasEquinix.savvis.net [204.70.134.10] 16 188 ms 189 ms 187 ms aer1-po10.DallasEquinix.savvis.net [204.70.134.14] 17 189 ms 193 ms 191 ms 208.175.175.42 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out.
Posted by the--dud, 06-17-2005, 08:20 AM
In case you folks didn't know, servermatrix has a network status forum here... http://forums.servermatrix.com/viewforum.php?f=26 It isnt terribly interesting, but it has a lot of router maintenance etc for the last few weeks. My SM box has been up all the time, but ORBIT isn't available...
Posted by bugsoft, 06-17-2005, 08:25 AM
Well, since I can't access theplanet, nor servermatrix nor orbit nor my servers, nor server matrix forums it's not very usefull that link
Posted by the--dud, 06-17-2005, 09:27 AM
Strange hehe.. Must be some gigabit providers that works, and some that don't then