ServerMatrix down [merged]

Portal Home > Knowledgebase > Articles Database > ServerMatrix down [merged]

Posted by Shaw Networks, 03-20-2004, 01:31 AM
Can anyone access their SM server? Mines down
Posted by c0bra, 03-20-2004, 01:33 AM
Mine was down, came back, down again now......
Posted by thedavid, 03-20-2004, 01:33 AM
Servers are bouncing - one dropped, came back... 2 others dropped and stayed down...
Posted by The3bl, 03-20-2004, 01:35 AM
Down up down up.. Phone is busy. I assume they are having a issues..
Posted by null, 03-20-2004, 01:35 AM
I spoke to Neo (tech guy) - he told me its a router problem
Posted by JasonID, 03-20-2004, 01:36 AM
Can't get through to support.. everything is down. What's up?
Posted by Matt, 03-20-2004, 01:36 AM
Mine are up and have not gone completely down according to monitor. I have been in one for the past half hour and did notice a slow down at one point, but only lasted a few minutes. I am still in the one and speeds are normal for it now. It's on the 69.93.240.** block. Just checked another on the 69.56.172.** block and it is ok as well.
Posted by null, 03-20-2004, 01:37 AM
I have to servers and only one is down My ip is 69.93.103.*
Posted by thedavid, 03-20-2004, 01:37 AM
All our servers that are down are on CAR2-8.DLLSTX2 - how about you guys?
Posted by cybexhost1, 03-20-2004, 01:38 AM
The car2-8.dllstx2 router is currently down. That is a customer access router, so the entire datacenter isn't down.
Posted by thedavid, 03-20-2004, 01:39 AM
Yup, just confirmed it with their nifty new 'network overview' thingy in orbit. That's pretty handy
Posted by inteltechs, 03-20-2004, 01:39 AM
69.93.82.* 69.93.222.* down too.
Posted by odsdc, 03-20-2004, 01:40 AM
down here
Posted by Shaw Networks, 03-20-2004, 01:40 AM
Anyone find out an ETA on it?
Posted by cybexhost1, 03-20-2004, 01:40 AM
69.93.35.***
Posted by null, 03-20-2004, 01:40 AM
We are back
Posted by Matt, 03-20-2004, 01:40 AM
http://www.webhostingtalk.com/showth...hreadid=250047
Posted by JasonID, 03-20-2004, 01:42 AM
Thanks Matt.
Posted by cybexhost1, 03-20-2004, 01:42 AM
The router shutdown about 10 minutes ago - it is still a little early.
Posted by Shaw Networks, 03-20-2004, 01:42 AM
69.93.186.* here seems like an isolated IP range
Posted by c0bra, 03-20-2004, 01:42 AM
Yeah we're back too now... hopefully with no more drop outs.
Posted by Xoopiter-Jeff, 03-20-2004, 01:43 AM
69.93.217.*** up and fine.
Posted by JasonID, 03-20-2004, 01:43 AM
69.93.181.*** is down here.
Posted by thedavid, 03-20-2004, 01:43 AM
A server on CAR2-6.DLLSTX2 just dropped. Same one that dropped before 2-8 dropped.
Posted by cybexhost1, 03-20-2004, 01:44 AM
69.93.35.*** still down here. The router is still shown as being offline.
Posted by inteltechs, 03-20-2004, 01:45 AM
all of our servers are up again
Posted by c0bra, 03-20-2004, 01:45 AM
There's no point posting every few seconds... they're working on it. Give them time.
Posted by odsdc, 03-20-2004, 01:45 AM
UP again 69.93.40.*
Posted by cybexhost1, 03-20-2004, 01:46 AM
car2-6.dllstx2 confirmed down in addition to car2-8.dllstx2.
Posted by JasonID, 03-20-2004, 01:47 AM
dsr2-v2.dllstx2.theplanet.com Seems dead.
Posted by sysc, 03-20-2004, 01:48 AM
still down here
Posted by coight, 03-20-2004, 01:49 AM
Attached Images switch2.gif (26.7 KB, 201 views)
Posted by cybexhost1, 03-20-2004, 01:49 AM
There is no way a Distribution Switch Router is down. It is just a few Customer Access Routers.
Posted by thedavid, 03-20-2004, 01:49 AM
Hah, nice graphic robert.
Posted by Shaw Networks, 03-20-2004, 01:50 AM
Wow ORBIT is great, minute to minute live updates, it's like a comptuer nerds version of ESPN. "Look, look, they're going for the GigE router! They're replacing it like I've never seen before! Oh ****! He dropped the hardware! Oh the humanity!"
Posted by coight, 03-20-2004, 01:51 AM
Yes paint is a wonderful tool
Posted by JasonID, 03-20-2004, 01:51 AM
Slap my face, bad copy and paste. Sorry bout that.
Posted by cybexhost1, 03-20-2004, 01:51 AM
"Back, Back, Way Back, and the Connection is GONE!"
Posted by thedavid, 03-20-2004, 01:52 AM
Heh, doesn't update as fast as my nagios does though Still really neato.
Posted by cybexhost1, 03-20-2004, 01:53 AM
Bad Copy and Paste? What is that supposed to mean?
Posted by thedavid, 03-20-2004, 01:53 AM
It means he copied the wrong thing. No need to jump all over someone. Sheesh.
Posted by sysc, 03-20-2004, 01:53 AM
anyone have an ETA?
Posted by Shaw Networks, 03-20-2004, 01:54 AM
Lol nice we need commentation like that on WHR
Posted by cybexhost1, 03-20-2004, 01:54 AM
Just didn't make sense, but now I get it. He accidently copied the Distribution Switch instead of the Customer Access Switch.
Posted by JasonID, 03-20-2004, 01:54 AM
I copy and pasted the wrong line. Human error. Again you have my most sincere apologies.
Posted by cybexhost1, 03-20-2004, 01:56 AM
car2-6.dllstx2 is back up. But that isn't where I am.... Come on car2-8.dllstx2 The router that could... I think I can.. I think I can..
Posted by Shaw Networks, 03-20-2004, 01:58 AM
Yay, one router back up, too bad my server isn't on it :-/
Posted by JasonID, 03-20-2004, 01:58 AM
Just our luck, I guess. I am on car2-8.dllstx2 also. Hopefully it will be back up presently.
Posted by thedavid, 03-20-2004, 02:01 AM
Makes me feel good ours are spread around the dc Still, 2 are connected to that one. forums.servermatrix.com is sloooow when I tried too, no real explanation I saw posted yet though.
Posted by sysc, 03-20-2004, 02:01 AM
no eta huh ?
Posted by thedavid, 03-20-2004, 02:02 AM
"A short while" is what was posted. No real ETA.
Posted by coight, 03-20-2004, 02:03 AM
WE have a server on car2-4 thats down. I am assuming it's not our server as httpme.com is also down.
Posted by JasonID, 03-20-2004, 02:03 AM
They have to be down for 44.64 minutes before the SLA kicks in. I am betting we will be 'saving' money this month.
Posted by Nessun, 03-20-2004, 02:03 AM
i have pretty well had it with theplanet as far as all the downtimes i have had in the last 2 months on my server. This is just getting flat out stupid.
Posted by cybexhost1, 03-20-2004, 02:04 AM
Hope you got your stopwatch handy.
Posted by Shaw Networks, 03-20-2004, 02:05 AM
I made the first post here about 5 mins after it started, we can use it as the start time.
Posted by thedavid, 03-20-2004, 02:05 AM
Got better, got nagios 0d 0h 38m 52s so far
Posted by JasonID, 03-20-2004, 02:05 AM
I am really starting to dislike your attitude. (I am assuming you are being sarcastic here.) Note 'saving' in quotations. Every second we are down costs me money.
Posted by cybexhost1, 03-20-2004, 02:05 AM
Ohh..Don't even get me started. You want downtime? Go to Burst.Net - an entire weekend. The Burst.Net outage led to the biggest post in WHT history, and had the most number of people on at one time in WHT as well.
Posted by Nessun, 03-20-2004, 02:07 AM
im talking about a $500 a month theplanet server not some $60 pos burstnet server so i dont want to hear it.
Posted by cybexhost1, 03-20-2004, 02:07 AM
If we are going to have downtime, we mine as well get something out of it. "Every second we are down costs me money" - then get your money back with the SLA.
Posted by thedavid, 03-20-2004, 02:08 AM
Ooohhh she's comin back.
Posted by Shaw Networks, 03-20-2004, 02:10 AM
We're UP again! What was the stopwatch time?
Posted by JasonID, 03-20-2004, 02:10 AM
Time check please, thedavid. Routers are up.
Posted by StartAnISP, 03-20-2004, 02:11 AM
All routers show up now but my server is still dead..
Posted by thedavid, 03-20-2004, 02:12 AM
39 minutes, 50 seconds as measured from ev1.
Posted by cybexhost1, 03-20-2004, 02:12 AM
"The CAR2-4 router is currently experiencing issues that are causing the customers routed through that router to be down. Engineers are working on the problem and the issues should be resolved shortly." -Server Matrix Forum @ 12:35AM EST It is back on as of 1:10AM EST.
Posted by Shaw Networks, 03-20-2004, 02:12 AM
DAMN IT, just missed the SLA, SM did that on purpose!
Posted by aleck, 03-20-2004, 02:13 AM
i'm up. 40-42 min down
Posted by StartAnISP, 03-20-2004, 02:13 AM
Oh I think I am covered, they had this same server of mine down for over 9 hours the other day.
Posted by Nessun, 03-20-2004, 02:15 AM
still not back and very pissed.
Posted by ThePrimeHost, 03-20-2004, 02:15 AM
One of ours is still down too....
Posted by techsla, 03-20-2004, 02:16 AM
Is the planet down?
Posted by StartAnISP, 03-20-2004, 02:16 AM
I am on car2-4 and it just came back.. For now..
Posted by cybexhost1, 03-20-2004, 02:17 AM
And it just had to be down when one of my SiteUptime pings occured. My downtime for the Month of March is now 0.113 with 99.887% uptime.
Posted by Nessun, 03-20-2004, 02:18 AM
back but lagged bad
Posted by thedavid, 03-20-2004, 02:20 AM
That's not a bad thing - really... Uptime checked once per hour (or whatever) isn't really a good indicator of 'real' uptime at all.
Posted by cybexhost1, 03-20-2004, 02:21 AM
It is checked 3 times an hour..btw.
Posted by ThePrimeHost, 03-20-2004, 02:22 AM
Back up. That was quick!
Posted by thedavid, 03-20-2004, 02:22 AM
Still not a real good indicator. But ok.
Posted by cybexhost1, 03-20-2004, 02:23 AM
What do you want for free? Eh?
Posted by Nessun, 03-20-2004, 02:24 AM
I am really getting tired of this I'm starting to wonder why I'm paying theplanet prices for something with an uptime similar to managed.com these last 2 months little over exagerated but this keeps up and it won't be.
Posted by thedavid, 03-20-2004, 02:25 AM
http://www.nagios.org 3 checks, cycles once every 90 seconds by default. Can be made to go at shorter intervals if you want. Free
Posted by cybexhost1, 03-20-2004, 02:25 AM
BACK DOWN AGAIN!
Posted by thedavid, 03-20-2004, 02:25 AM
Yup, the pager's going off again
Posted by sysc, 03-20-2004, 02:25 AM
yup down here as well
Posted by cybexhost1, 03-20-2004, 02:26 AM
Start up the stopwatch again.
Posted by null, 03-20-2004, 02:28 AM
down
Posted by cybexhost1, 03-20-2004, 02:28 AM
"Update: The CAR routers that were down are all now back up and performing normally. We are investigating the cause of the problems and will issue a formal RFO once it have been determined." - ServerMatrix Forums @ 1:24 AM EST. But too bad it is back down again. {I hate this 90 second rule}
Posted by thedavid, 03-20-2004, 02:28 AM
Don't worry, it's automatic
Posted by Ramprage, 03-20-2004, 02:29 AM
What do you use to monitor your servers? Alertra? Doesn't nagios have to be running on a server that is actually online to work?
Posted by Nessun, 03-20-2004, 02:29 AM
I think ill get out a calender to check my downtime.
Posted by Matt, 03-20-2004, 02:29 AM
Merged another...
Posted by cybexhost1, 03-20-2004, 02:30 AM
This is starting to bring back memories of the Burst.Net outage.
Posted by thedavid, 03-20-2004, 02:31 AM
Nagios on an ev1 server. Shoots me an email and pages when 'bad stuff' happens, whether that's bad load, too many processes, disk warnings, or just plain down. I highly reccomend it
Posted by StartAnISP, 03-20-2004, 02:32 AM
Dead again, mine is on car2-4
Posted by c0bra, 03-20-2004, 02:32 AM
Down here as well.
Posted by JasonID, 03-20-2004, 02:33 AM
Yay, SLA. Down here too on car2-8.dllstx2.
Posted by cybexhost1, 03-20-2004, 02:33 AM
Nagios looks very good, but all of my servers are either The Planet or ServerMatrix.
Posted by inteltechs, 03-20-2004, 02:33 AM
yeah down again...
Posted by sysc, 03-20-2004, 02:34 AM
yup, down.
Posted by TLott, 03-20-2004, 02:34 AM
69.93.72 down. Is ThePlanet down as well, or is this isolated to SM? (hope it's isolated)
Posted by c0bra, 03-20-2004, 02:34 AM
I run serversalive on my own local machine. Works very well and runs off Windows. The free version can monitor (I think) upto 20 different devices or servers. Very happy with it.
Posted by Ramprage, 03-20-2004, 02:35 AM
LMAO That might not be far off if this keeps up. What day is it again? Thedavid, thanks for that info on your monitoring script. The funny thing about this outage is that I JUST migrated a box from Ev1's network to Server Matrix.... BIG mistake on my part... sigghhh.
Posted by thedavid, 03-20-2004, 02:35 AM
Get a servint vps or something to run it on. Seriously, I wouldn't consider running without it, it 'saves the day' that often. When problems happen, they get resolved fast because of it.
Posted by odsdc, 03-20-2004, 02:35 AM
all down here again
Posted by cybexhost1, 03-20-2004, 02:36 AM
How often does it check? And - URL?
Posted by c0bra, 03-20-2004, 02:36 AM
My box with ThePlanet is fine. Although we recently had an outage with them that lasted nearly two hours.
Posted by thedavid, 03-20-2004, 02:36 AM
heh, ev1's dc2 had some latency issues today, and they're having some issues with ALGX. Nagios noted that one too
Posted by Nessun, 03-20-2004, 02:36 AM
Im normally a big fan of theplanet but they have totally taken that away with the last 2 months services they have gone out the window and into a trash can.
Posted by Aussie Bob, 03-20-2004, 02:37 AM
Yep.
Posted by Ramprage, 03-20-2004, 02:38 AM
Or just use a third party monitoring service like Alertra, SitePulse, Easy Monitor or the other ones out there. The VPS is probably on the same network as you anyways
Posted by StartAnISP, 03-20-2004, 02:38 AM
Why the hell does this happen constantly over there? Why don't they have hot swap routers? Why do they keep letting inexperienced 11 year olds screw with their equipment? What is the problem.. This is beyond silly.. Wish there was some better solution.. A company that offers what The Planet offers but actually delivers reliable services. What the hell good is all their redundant bandwidth to the Internet if the problem is that they can't keep their own internal network up for more than a few weeks at a time? Too many from the cheap labor pool with passwords to the routers I guess.. Oh well, looks like we will need to move the 5 servers we have there.. Just not sure where to go.. EV1 offers pretty much zero support but is clearly more reliable network wise. Rackspace is WAY too proud of their pricing... Sigh..
Posted by thedavid, 03-20-2004, 02:38 AM
CAR2-6.DLLSTX2 back up...
Posted by c0bra, 03-20-2004, 02:38 AM
As often as you like. I set it to run every 60 seconds... play a loud sound effect if my server goes down and page me on three cycles of downtime (that's three minutes). It also pages me whenever the server(s) come back to life as well. http://www.woodstone.nu/salive/
Posted by coight, 03-20-2004, 02:38 AM
David, do you have a howto for nagios. I have tried installing myself and it seems a major pain?
Posted by thedavid, 03-20-2004, 02:39 AM
Naw, I was talking about one of these: http://www.servint.com/vps/
Posted by JasonID, 03-20-2004, 02:39 AM
Called them. No ETA. Call sales for your SLA refund.
Posted by c0bra, 03-20-2004, 02:40 AM
Unfortunately all the bargain hosts have their faults. But trust me on this.... YOU DO NOT want to put your trust in ev1s support personnel. They are beyond useless.
Posted by thedavid, 03-20-2004, 02:41 AM
Not really, what I did is read through *all* of the docs prior to setting it up. That's really required, since the config can be so variable and flexible, and since there's so many config files... Then, just take and modify the example configs. It's not too bad once setup though. Takes no care/feeding, just works. If you have any questions on it I'd be happy to help though. We actually run separate instances of it on one server, which works well too.
Posted by Ramprage, 03-20-2004, 02:41 AM
If anyone finds a good how to please forward it to me, I'll post it on my site for future referance. I wonder how many websites and other things this one outage is affecting. Just to think 1 server can hold hundreds of different sites on it alone... the stats might be mind blowing!
Posted by JasonID, 03-20-2004, 02:41 AM
thedavid, do me a huge favor and let me know how long car2-8.dllstx2 was down total tonight. (When, IF, it comes back up) I guess I need to start running WebWatchBot again.
Posted by StartAnISP, 03-20-2004, 02:42 AM
Back up for now car2-4
Posted by Nessun, 03-20-2004, 02:42 AM
I wouldnt be all that mad if this was a servermatrix box paying $100 like everyone else but this isn't it is a THEPLANET box big difference in price and therefore I expect a big difference in uptime. Time to give rackspace a call.
Posted by thedavid, 03-20-2004, 02:43 AM
Oh, I'll post a total once she's back. It's set to 'obsess' over downed services, so it's accurate to about 30-60 seconds, usually.
Posted by thedavid, 03-20-2004, 02:44 AM
None of the servers are $100 that we have But I understand what you mean.
Posted by JasonID, 03-20-2004, 02:44 AM
Their sales people are certifiably insane about making their potential customers happy.
Posted by Ramprage, 03-20-2004, 02:45 AM
Guys, quick question here. My main site for my hosting company is on a box on SM that is down. How can I at least mirror my email to another server so my clients can contact me, or mirror my site to a second machine? I don't want to get 2 copies of all emails though, just have it redundant because this is hurting me badly.
Posted by c0bra, 03-20-2004, 02:46 AM
My only SM box is $59 a month so I can't really complain. If this was my $599 theplanet box then I would be far from impressed. Oh my $59 box is pingable again now.
Posted by thedavid, 03-20-2004, 02:46 AM
You'll want distributed DNS servers (like a vps elsewhere to do secondary, at least) and a secondary MX for your main domain name. That way, mail spools on the 'secondary' mail server, and DNS is resolvable at all times.
Posted by StartAnISP, 03-20-2004, 02:46 AM
I know.. We have some servers there too.. We have them in several data centers and to be honest, we are happy with NONE of them. We are considering moving it all in house again just to get everyone's hands off our stuff as much as possible. This is a big hassle though for us but I think it will be our only choice soon. Downtime is just not an option for us. The whole reason we went with The Planet and similar companies was to gain reliability, redundant badnwidth etc.. We have had MUCH more trouble than we had running our own network. At least with our own network we only had to worry about one thing, the connection being out. With a back using BGP4 this will likely not happen. SIGH...
Posted by sysc, 03-20-2004, 02:47 AM
still down here
Posted by thedavid, 03-20-2004, 02:49 AM
Ya know, I was just settling down to watch a netflix too... Sad...
Posted by mp3sattack, 03-20-2004, 02:49 AM
mine has been up and down for like an hour, right now is down, that's why i came here, hahahhaa, if you ask me, i'm less worried now, at least were not my users :p
Posted by thedavid, 03-20-2004, 02:50 AM
Up again
Posted by JasonID, 03-20-2004, 02:51 AM
Time, please, thedavid.
Posted by StartAnISP, 03-20-2004, 02:52 AM
Their graph shows 2-8 down now..
Posted by Nessun, 03-20-2004, 02:52 AM
heh give it 5minutes im sure they can knock it down again.
Posted by Shaw Networks, 03-20-2004, 02:52 AM
Finally... total downtime?
Posted by cybexhost1, 03-20-2004, 02:52 AM
Back up here as well, do we have the official time?
Posted by thedavid, 03-20-2004, 02:53 AM
Total combined is 1h 6m 0s I knew you'd ask
Posted by cybexhost1, 03-20-2004, 02:54 AM
The network overview isn't automatic - all of those pings and crap would knock them offline. It is manually updated by NOC monkeys, based upon their reports, etc.
Posted by odsdc, 03-20-2004, 02:56 AM
its down again - been bouncing up and down for over an hour now
Posted by Ramprage, 03-20-2004, 02:56 AM
Wow, honestly I'm trying to understand that but I'm having trouble. Could you go into a little more details because I'm either not reading properly or half asleep... From what I understand: SM Server - hosts mysite.com ns1.mysite.com ns2.mysite.com are both on that server. I would setup a second server that is: ns3.mysite.com ns4.mysite.com For the DNS entry at the registrar I would enter all 4 name servers so the domain would use both boxes right? Then where do I put the MX record, on the first or second server? Do I need to setup the domain on the second box as well? Sorry I've never done anything like this so excuse my n00bish questions
Posted by StartAnISP, 03-20-2004, 02:57 AM
I think they have one network cable and they keep switching it back and forth between 2 or 3 routers to at least give everyone a little uptime. They will probably continue this practice until the new guy get's back from Walmart with a few new cables. Problem is, he will probably bring back RJ11 phone cords. So, we will probably have to go through this all weekend until little Johnny gets the Walmart order right. Finally little Johnny (the newest Network Engineer at SM) will get a flat tire on his bike just as he is about to make it back with the correct cables.. OK, I will stop.. But you know I have to wonder if this is really the sort of thing that is going on over there? Of course I am joking here about Johnny and such..
Posted by thedavid, 03-20-2004, 02:57 AM
Oh good god... CAR2-6.DLLSTX2 dropped again.. What're they doing, unplugging one and running back and forth with 1 working router? Humor is required during these times, no?
Posted by Nessun, 03-20-2004, 03:00 AM
told u they could knock it down again
Posted by cybexhost1, 03-20-2004, 03:01 AM
car2-8.dllstx2 is now back online. car2-6.dllstx2 is now back offline.
Posted by thedavid, 03-20-2004, 03:01 AM
Right, up to the MX record thing. The *best* way to do it is make the second server a 'slave'. You'd put something like this in your named.conf file: zone "domain.com" { type slave; masters { 127.0.0.1; }; file "/var/named/domain.com.db"; }; Replace 'domain.com' with your real domain, and 127.0.0.1 with your primary nameserver. Then, any dns changes that you make will automagically be applied to the secondary nameserver. Make all dns changes within the primary (first) nameserver. Then you're golden.
Posted by Ramprage, 03-20-2004, 03:01 AM
Server Matrix didn't tell anyone what is really happening? Did you know they were hit by a massive snow storm and the datacenter floor is like a skating rink! All the techs keep knocking down the servers and grabbing the wires from the routers for balance.
Posted by StartAnISP, 03-20-2004, 03:03 AM
car2-4 dead again
Posted by cybexhost1, 03-20-2004, 03:03 AM
I left Burst.Net to avoid crap like this.
Posted by thedavid, 03-20-2004, 03:05 AM
back up again...
Posted by StartAnISP, 03-20-2004, 03:05 AM
Guess you have the cable for the moment.
Posted by JasonID, 03-20-2004, 03:05 AM
You will have to keep a running tally for each router now.
Posted by aleck, 03-20-2004, 03:06 AM
was down for 16 more min
Posted by Ramprage, 03-20-2004, 03:06 AM
thedavid, ok so the DNS slave settings would only affect that one domain and not all others right?
Posted by Matt, 03-20-2004, 03:06 AM
I feel for you guys. None of my servers are located on either of those segments so they have been spared this time. I'd share my cable if it would reach....
Posted by cybexhost1, 03-20-2004, 03:07 AM
I thought Musical Chairs were for 10 year olds. Now these NOC Monkey's are playing Musical Routers.
Posted by StartAnISP, 03-20-2004, 03:07 AM
car2-4 back up, yes cable is mine... All mine.. HA HA HA <----evil laugh
Posted by Nessun, 03-20-2004, 03:07 AM
your very lucky some of us have had a lot of downtime over the last month or 2 due to downtimes from one of these very same routers and I have had it I am so mad right now I want to drive my car into theplanet datacenter.
Posted by Ramprage, 03-20-2004, 03:08 AM
LMFAO That's the funniest thing I've read in a long time! Thanks man, that was toooo good! LMAO runs to Wal mart for their network cables LOL I'm on the floor over here.
Posted by thedavid, 03-20-2004, 03:09 AM
Yup Doesn't need a control panel setup (if you're using one) or anything. Just that config in the named.conf and bind should handle the rest. Just make sure the secondary mail server is setup to act as a 'store and forward' smtp service rather than a 'deliver to local users' service. So when the primary comes back up, it'll auto-deliver to that (rather than holding on to it)
Posted by odsdc, 03-20-2004, 03:09 AM
lol, it seems that as one router come online, the others go offline
Posted by cybexhost1, 03-20-2004, 03:09 AM
Evil Laugh = muh huh huh huh The first 'muh' is essential.
Posted by thedavid, 03-20-2004, 03:09 AM
It just did. Mymy.. The other router is again down.
Posted by Shaw Networks, 03-20-2004, 03:09 AM
I also left Burst.net to avoid stuff like this :-/ ServerMatrix needs to make an update, best to keep customers posted. Improves reputation in my mind.
Posted by StartAnISP, 03-20-2004, 03:10 AM
You know, I was really thinking that after a long week I would stay up all night answering tickets from pissed off customers.. Weee... Oh how I love The Planet.. I think they are on another planet.. Is that why they call it The Planet?
Posted by thedavid, 03-20-2004, 03:11 AM
One thing's for sure, I'm about ready to turn off the pager (phone). Sick of it going off constantly
Posted by cybexhost1, 03-20-2004, 03:11 AM
car2-8.dllstx2 DOWN AGAIN! And the Stopwatch starts yet again....
Posted by JasonID, 03-20-2004, 03:11 AM
2-8 IS DEAD AGAIN!!!
Posted by aleck, 03-20-2004, 03:11 AM
what's the ETA? i mean i can switch to my backup box (20-30 min of work), but not sure if it worth it.
Posted by BizB, 03-20-2004, 03:12 AM
one of my servers keep bouncing up and down but the other is ok yeah i can see in orbit car2-8.dllstx2 is down
Posted by Ramprage, 03-20-2004, 03:13 AM
You had to get more technical again just when I was starting to figure this stuff out... Ok, but I'm using the second servers mail server for that machines local clients... will this cause any problems?
Posted by Shaw Networks, 03-20-2004, 03:13 AM
Musical routers... sounds catchy...
Posted by thedavid, 03-20-2004, 03:14 AM
Nope - it should be configurable per-domain. What control panel/os/mail software you using?
Posted by cybexhost1, 03-20-2004, 03:14 AM
Can anyone get these guys on the phone?
Posted by thedavid, 03-20-2004, 03:15 AM
I'd rather talk to you Oh, and answer support tickets about servers being unreachable
Posted by Nessun, 03-20-2004, 03:15 AM
Nope busy everytime for me but I dont think my cursing them out is going to help.
Posted by StartAnISP, 03-20-2004, 03:15 AM
Last time, I got one update to my ticket in over 9 hours of downtime. Everytime I called they just told me the same EXACT thing.. It's broke, we are fixing it.. OK, what's the ETA? Don't know OK, what's wrong? Don't know Same crap every time.. They have not even had the decency to update my ticket this whole time.. No answer to my ticket at all. Their phone is busy.. Tickets don't get answered and servers are down WAY too much.. So... In my opinion, this is suckin' pretty bad! I think there is a need for a company right in the middle of The Planet and Rack Space. I don't quite need Rackspace (at least not their pricing) but I need something better than The Planet. Wait, I would just settle for well.. You know.. A FREAKIN PINGABLE SERVER!!!!
Posted by JasonID, 03-20-2004, 03:16 AM
They have no clue, don't bother phoning. Evidently the monkeys are trying to figure out what's wrong. They also know nothing about the SLA.
Posted by cybexhost1, 03-20-2004, 03:16 AM
thedavid, This is starting to become way too similar to the BURST outage.
Posted by Shaw Networks, 03-20-2004, 03:16 AM
Well SM tech support told me everyone was called down there and they're all looking at it now. Should be taken care of soon :-/
Posted by Nessun, 03-20-2004, 03:17 AM
Im glad they can all sick back and look at it but I'd rather them fix it and not temporarily for 5minutes and break it again.
Posted by thedavid, 03-20-2004, 03:17 AM
Nah! I think they'll get it sorted soon. Bad stuff happens. Everywhere. How well it's dealt with separates the cream from the rest.
Posted by Ramprage, 03-20-2004, 03:18 AM
lmao, this thread is turning into comedy night on WHT I'm using Cpanel with Red Hat boxes!
Posted by JasonID, 03-20-2004, 03:19 AM
2-8 UP. Time Please, thedavid.
Posted by thedavid, 03-20-2004, 03:19 AM
Back up (again)
Posted by cybexhost1, 03-20-2004, 03:19 AM
"Thank You for calling ServerMatrix - How can I help you" "Yes, I was calling to inquire about the SLA" "What about him" "Excuse me, him?" "Yes, him - what about him?" "I am sorry, I must not be aware of what you are talking about." "SLA - He is sitting next to be." "Who is SLA" "Sorry Lazy (Use Your Imaginiation)" -NOC Monkey's in a nutshell- Last edited by cybexhost1; 03-20-2004 at 03:24 AM.
Posted by thedavid, 03-20-2004, 03:24 AM
On 2-8: 1h 20m 5s On 2-6: 0d 0h 27m 21s downtime and 0d 0h 25m 20s of 'unknown' (ping didn't return anything, as though the IP's were unroutable)
Posted by BizB, 03-20-2004, 03:25 AM
in 60 days for both servers i had no downtime and i think its great spcialy for the low price i am paying
Posted by Nessun, 03-20-2004, 03:26 AM
im talking about theplanet though not servermatrix big difference in price also most all the downtime was by one of these specific routers which i am referring to.
Posted by StartAnISP, 03-20-2004, 03:28 AM
car2-4 back down again.. Yay, this is fun..
Posted by Nessun, 03-20-2004, 03:28 AM
and down again.
Posted by tichuot, 03-20-2004, 03:29 AM
https://orbit.theplanet.com/resource_net_overview.html is incorrect
Posted by thedavid, 03-20-2004, 03:30 AM
Jeez... Never gonna get to that movie... Hallie berry is upset at me and the planet now...
Posted by anand247sm, 03-20-2004, 03:30 AM
My server is up now
Posted by Ramprage, 03-20-2004, 03:31 AM
Well I'm sure this yo-yo like activty will endure when my head is snuggled up against my pillow for hours to come, no point it sitting here watching them disconnect and plug in the wrong cables.. err .... power cords.... or phone lines..... whatever they're doing I hope they figure it out before I wake up Night all, I'll check back in the morn'! thanks for keeping me humoured about this btw, and david for the help
Posted by cybexhost1, 03-20-2004, 03:32 AM
25 bucks says that car2-8.dllstx2 will be the next one down. Anyone else in?
Posted by JasonID, 03-20-2004, 03:33 AM
2-8 IS DEAD. Response times are through the roof.
Posted by aleck, 03-20-2004, 03:33 AM
back up. that's ridiculous. i'm going outside with the handy, so i won't have itching fingers to switch dns. Last edited by aleck; 03-20-2004 at 03:37 AM.
Posted by cybexhost1, 03-20-2004, 03:35 AM
Who collects the life insurance?
Posted by JasonID, 03-20-2004, 03:35 AM
We can split it, we deserve it.
Posted by Shaw Networks, 03-20-2004, 03:36 AM
Come on SM let's keep it up this time, actually it's down for me now.
Posted by anand247sm, 03-20-2004, 03:37 AM
SM down for me again
Posted by cybexhost1, 03-20-2004, 03:38 AM
And.... car2-8.dllstx2 is yet again back down. "This is the tenth time tonight Bob, tell him what he has won"
Posted by Nessun, 03-20-2004, 03:42 AM
and up again. I have the wire for now and Im greedy they better not give it to any of you.
Posted by WebHostingNeeds, 03-20-2004, 03:42 AM
seems again down. I got error from SM monitoring system. Thought some thing wrong with my server as only one is having problem. ***** The Planet Monitoring System: IPAlert ***** Notification Type: PROBLEM The following host appears to be: DOWN
Posted by anand247sm, 03-20-2004, 03:43 AM
still i cant access
Posted by cybexhost1, 03-20-2004, 03:43 AM
Do a little dance, Make a little love, Go back down tonight.
Posted by JasonID, 03-20-2004, 03:43 AM
Which router? 2-8 is still dead.
Posted by cybexhost1, 03-20-2004, 03:46 AM
car2-6.dllstx2 AND car2-8.dllstx2 CONFIRMED DOWN!
Posted by Akash, 03-20-2004, 03:50 AM
The night couldnt get any better for me According to the guy i just talked to on the phone, the "entire senior staff is down there working on it. No eta" There was some shouting in the background....woulndt be surprised if someone fell asleep at the wheel and is getting a good talking to
Posted by cybexhost1, 03-20-2004, 03:50 AM
car2-6.dllstx2 Back Up car2-8.dllstx2 Still Sleeping
Posted by Nessun, 03-20-2004, 03:52 AM
if they want Ill come down and give him more then a talking to.
Posted by JasonID, 03-20-2004, 03:52 AM
2-4 is gone now.
Posted by tichuot, 03-20-2004, 03:53 AM
car2-4 is dead
Posted by cybexhost1, 03-20-2004, 03:53 AM
Damn Senior Citizens.
Posted by BogdanM`, 03-20-2004, 03:53 AM
Down again. Just lovely
Posted by thedavid, 03-20-2004, 03:54 AM
That's amusing But also worrisome It is friday night... maybe someone spilled beer on the router? (ok, just trying to get as many smilies in this post as possible )
Posted by anand247sm, 03-20-2004, 03:54 AM
car2-8.dllstx2 is still down
Posted by cybexhost1, 03-20-2004, 03:56 AM
car2-8.dllstx2 is down for the count.
Posted by anand247sm, 03-20-2004, 03:56 AM
car2-4.dllstx2 also down
Posted by BF-Gary, 03-20-2004, 03:56 AM
They both seem to go up and down at the same time.
Posted by thedavid, 03-20-2004, 03:57 AM
Maybe the routers are fighting a steel cage match...
Posted by anand247sm, 03-20-2004, 03:58 AM
Not able to handle each other
Posted by cybexhost1, 03-20-2004, 03:59 AM
Yawn....I don't know how much more I can handle.
Posted by anand247sm, 03-20-2004, 03:59 AM
car2-4.dllstx2 again up, down no how much time it will stay
Posted by BF-Gary, 03-20-2004, 04:00 AM
Well I hope they both win!!!
Posted by JasonID, 03-20-2004, 04:00 AM
2-8 is back. thedavid, can we get a time?
Posted by anand247sm, 03-20-2004, 04:01 AM
me 2
Posted by cybexhost1, 03-20-2004, 04:01 AM
Now everyone will flood with SLA refund requests, resulting in even more downtime.
Posted by Imago, 03-20-2004, 04:02 AM
And a bad example for all your fans! :-) Me included, as my she is down for more than 4 hours. Yesterday she was down for almost 6 hours at about the same time. I was thinking this is my limited memory, so precipitated to order some addon RAM. Do you think SM are going to compensate us with, say, additional RAM or free HDD? Just to make us psychologically overcome the big disappointment.
Posted by thedavid, 03-20-2004, 04:02 AM
I'll post 'er when everything's done. Sick of redoing the reports So check this thread after the dust settles.
Posted by JasonID, 03-20-2004, 04:03 AM
I don't blame you. Thanks very much for the service, thedavid.
Posted by BF-Gary, 03-20-2004, 04:03 AM
According to the SM SLA you will get money off your next bill.
Posted by anand247sm, 03-20-2004, 04:04 AM
I just managed to change ti DC to SM but..... damn same problem again
Posted by cybexhost1, 03-20-2004, 04:04 AM
I would like to trade in my Celeron 1.7Ghz for two Dual Xeon 2.8Ghz's please. The emotional distress of this outage is just too great.
Posted by Nessun, 03-20-2004, 04:04 AM
heh the SLA getting close to refunding me next months money in advance.
Posted by JasonID, 03-20-2004, 04:06 AM
Is there a page on theplanet website about the SLA, or is that in the contract?
Posted by Nessun, 03-20-2004, 04:07 AM
it was a joke :/
Posted by amzhost, 03-20-2004, 04:07 AM
My 2 servers at server matrix have been up and down for a while. Any body has a problem with them.
Posted by thedavid, 03-20-2004, 04:07 AM
Still bouncing back and forth, back and forth.. Like watching tennis, this is.. Swapping between browsers...
Posted by BF-Gary, 03-20-2004, 04:07 AM
Wouldn't the server need to be down 99.9% of the month for you to get it for free?
Posted by cybexhost1, 03-20-2004, 04:08 AM
The Planet's Total Control Line's SLA isn't 99.9% - it is 100%. If your server is down at all, your supposed to get dollars off of your bill.
Posted by JasonID, 03-20-2004, 04:08 AM
Drat.
Posted by Nessun, 03-20-2004, 04:13 AM
still up id say this is a record but i think the TP admins will pull the line on purpose.
Posted by thedavid, 03-20-2004, 04:15 AM
When it's up for 15 minutes straight (both) I'll post times
Posted by BF-Gary, 03-20-2004, 04:16 AM
down again here
Posted by JasonID, 03-20-2004, 04:16 AM
2-8: Dead.
Posted by Nessun, 03-20-2004, 04:16 AM
and the TP admin just kicked my server off the shelf and said take that. (down again.)
Posted by SW-Ray, 03-20-2004, 04:17 AM
Try http://www.webhostingtalk.com/showth...hreadid=250047
Posted by anand247sm, 03-20-2004, 04:19 AM
car2-8.dllstx2 up
Posted by Imago, 03-20-2004, 04:20 AM
cybexhost1Emotional distress is not covered by SLA but still is a valid reason for compensation far beyond the SLA-specified. And since distress is a dual stress, I think, a Dual Xeon will be right for you to make you forget the painful experience.
Posted by JasonID, 03-20-2004, 04:21 AM
Dead here.
Posted by cybexhost1, 03-20-2004, 04:21 AM
car2-8.dllstx2 Back Down.
Posted by ChowSumDung, 03-20-2004, 04:22 AM
My server is up! oh wait, it's down now. Ohh, it's up! Oh, it's down. Up. Down. Up. Down. Up. It's down.
Posted by Nessun, 03-20-2004, 04:24 AM
my server is starting to remind me of sex but its lasting longer then 30 seconds.
Posted by JasonID, 03-20-2004, 04:25 AM
This is getting old.
Posted by BF-Gary, 03-20-2004, 04:26 AM
Hey I have a bunch of spam emails I can forward to you that may solve that other issue for you.
Posted by anand247sm, 03-20-2004, 04:26 AM
car2-8.dllstx2, now i can c it down , all my customers shouting on my head
Posted by JasonID, 03-20-2004, 04:27 AM
2-8: Up.
Posted by anand247sm, 03-20-2004, 04:29 AM
2-8 is still down
Posted by anon-e-mouse, 03-20-2004, 04:29 AM
Moderators dream: That members would look in this forum first, if their host isn't here, post in this forum first
Posted by cybexhost1, 03-20-2004, 04:29 AM
ServerMatrix's Official Response as of 3:13 AM EST: "We are currently experiencing significant network connectivity difficulties that are affecting certain server's uplink. We have our full senior network engineering staff on-site and working on this problem, and we are doing our very best to correct these problems now. The servers have not been affected in any way other than their connectivity. We appreciate your patience in this matter, and we will have this problem corrected as soon as is possible. Thanks."
Posted by thedavid, 03-20-2004, 04:31 AM
On a bright note... "The Trunk Murders" is playing on the history channel... Fascinating to watch when I'd rather be sleeping...
Posted by JasonID, 03-20-2004, 04:31 AM
17 65 ms 66 ms 66 ms sl-theplanet-2-0.sprintlink.net [144.228.250.126 ] 18 65 ms 66 ms 66 ms car2-8-v1.dllstx2.theplanet.com [12.96.160.24] 19 66 ms 65 ms 66 ms 154.69-93-181.reverse.theplanet.com [69.93.181.1 54] Trace complete.
Posted by anand247sm, 03-20-2004, 04:34 AM
thanx for the info, now i am able to access my server. Relaxed........
Posted by Nessun, 03-20-2004, 04:34 AM
wait 30 seconds and try again
Posted by JasonID, 03-20-2004, 04:35 AM
I hope it stays up, but for some reason my sensible side doubts it.
Posted by anand247sm, 03-20-2004, 04:36 AM
keep your fingures crossed
Posted by cybexhost1, 03-20-2004, 04:37 AM
I have had enough..I am going to B-E-D. When I wake up at noon, all I can say is my servers better be up.
Posted by anand247sm, 03-20-2004, 04:38 AM
Gudnight....
Posted by odsdc, 03-20-2004, 04:39 AM
all my servers still down
Posted by anand247sm, 03-20-2004, 04:40 AM
I am able to access all of mine but car2-6.dllstx2 down
Posted by BizB, 03-20-2004, 04:41 AM
LOL
Posted by JasonID, 03-20-2004, 04:43 AM
2-8 Is gone again.
Posted by amzhost, 03-20-2004, 04:45 AM
It's back now, isn't it.
Posted by thedavid, 03-20-2004, 04:45 AM
If I weren't on the laptop, I'd be photoshopping two routers playing tug-o-war over a mud pit with some cat-5 cable. But I am, so I'm not. I do want to go to bed tho.
Posted by anand247sm, 03-20-2004, 04:45 AM
2-8 again down
Posted by amzhost, 03-20-2004, 04:46 AM
Posted: Sat Mar 20, 2004 1:24 am Post subject: -------------------------------------------------------------------------------- Update: The CAR routers that were down are all now back up and performing normally. We are investigating the cause of the problems and will issue a formal RFO once it have been determined. _________________ William Charnock VP Network Engineering ThePlanet.com
Posted by anand247sm, 03-20-2004, 04:47 AM
I think you forgot to keep ur fingures crossed
Posted by thedavid, 03-20-2004, 04:48 AM
That's old.
Posted by JasonID, 03-20-2004, 04:49 AM
Woops, you are right. My bad.
Posted by eXecution, 03-20-2004, 04:49 AM
omg my server keeps crashing too, I thought it was me but its servermatrix.
Posted by BizB, 03-20-2004, 04:50 AM
damn car2-8.dllstx2 is down again !
Posted by anand247sm, 03-20-2004, 04:50 AM
So next time your server crashes make sure its SM or you
Posted by thedavid, 03-20-2004, 04:50 AM
Wait... Maybe it is eXecution.. Everyone get 'em!
Posted by jessfx, 03-20-2004, 04:51 AM
still down here...
Posted by thedavid, 03-20-2004, 04:52 AM
It keeps bouncing back between the two routers guys - it'll be on and off till they fix it.
Posted by anand247sm, 03-20-2004, 04:52 AM
car2-8.dllstx2 again down
Posted by amzhost, 03-20-2004, 04:52 AM
don't know but it seems to be fine now. .
Posted by JasonID, 03-20-2004, 04:53 AM
2-8 is back. 18 78 ms 79 ms 71 ms car2-8-v2.dllstx2.theplanet.com [12.96.160.56] Trace complete.
Posted by coight, 03-20-2004, 04:54 AM
car 2-4 is now down. It's either 2-8 down or 2-4's down.
Posted by anand247sm, 03-20-2004, 04:54 AM
down for me
Posted by coight, 03-20-2004, 04:55 AM
Everything is up for the moment
Posted by anand247sm, 03-20-2004, 04:56 AM
yea just for a MOMENT.....
Posted by barko, 03-20-2004, 04:57 AM
"Moment" being the operative word. Fingers now glued in crossed position.
Posted by Boost, 03-20-2004, 04:57 AM
it is really too much, come on cant they fix it almost about 4 hours up and down
Posted by JasonID, 03-20-2004, 05:02 AM
I'm willing to bet that it's over. This may be a bit early, but I think its fixed.
Posted by ChrisTech, 03-20-2004, 05:03 AM
one box up, one box down...bleh....
Posted by odsdc, 03-20-2004, 05:05 AM
down again
Posted by Constantin, 03-20-2004, 05:07 AM
You lost
Posted by thedavid, 03-20-2004, 05:07 AM
Purdy picture time Attached Images 2-6.gif (15.8 KB, 64 views)
Posted by thedavid, 03-20-2004, 05:08 AM
And again, cause I love you guys (sniff) Attached Images 2-8.gif (15.6 KB, 55 views)
Posted by anand247sm, 03-20-2004, 05:08 AM
All boxes working for me
Posted by JasonID, 03-20-2004, 05:08 AM
Oh yay, 2-8 is gone again. Way to make me look bad, LOL.
Posted by anand247sm, 03-20-2004, 05:10 AM
..... down again
Posted by coight, 03-20-2004, 05:12 AM
2-8 down
Posted by anand247sm, 03-20-2004, 05:16 AM
car2-8.dllstx2 up again
Posted by BizB, 03-20-2004, 05:18 AM
so far so good
Posted by JasonID, 03-20-2004, 05:18 AM
Maybe that is it? Please Santa... let the servers stay up....
Posted by amzhost, 03-20-2004, 05:18 AM
i think we should take a nap, nothing we can do right now.
Posted by odsdc, 03-20-2004, 05:22 AM
all up here on 2-8 what about 2-6?
Posted by JasonID, 03-20-2004, 05:24 AM
Everything is up.
Posted by mnu, 03-20-2004, 05:26 AM
--- 12.96.160.52 ping statistics --- 12515 packets transmitted, 8881 packets received, 29% packet loss round-trip min/avg/max/stddev = 87.382/106.614/1776.233/47.749 ms ...which is car2-4-v2.dllstx2.theplanet.com go TP
Posted by thedavid, 03-20-2004, 05:27 AM
Ok, totals as promised (if it drops again, I'll just post tomorrow in the daylight hours): CAR2-8.DLLSTX2 Total ping unavailable: 0d 2h 22m 48s Total ping unknown (IP routing issue): 0d 0h 5m 20s CAR2-6.DLLSTX2 Total ping unavailable: 0d 1h 16m 21s Total ping unknown (IP routing issue):0d 0h 25m 20s
Posted by odsdc, 03-20-2004, 05:28 AM
and down AGAIN!
Posted by JasonID, 03-20-2004, 05:30 AM
Egads, I am going to bed!
Posted by thedavid, 03-20-2004, 05:30 AM
I jinxed it. They were both up for some time. (sigh) Last edited by thedavid; 03-20-2004 at 05:33 AM.
Posted by panopticon, 03-20-2004, 05:39 AM
Still going up and down for me at 4:30 AM, down since 12:25 AM
Posted by JasonID, 03-20-2004, 05:40 AM
2-8 is back.
Posted by panopticon, 03-20-2004, 05:43 AM
Anyone hear from servermatrix what the issue is? Is it a ddos attack or something else?
Posted by JasonID, 03-20-2004, 05:45 AM
You can see people on the DC cam, kinda neat.
Posted by JasonID, 03-20-2004, 05:53 AM
Yay, 2-8 is down again.
Posted by panopticon, 03-20-2004, 05:59 AM
Just came up again. Going on 4 hours 35 minutes now of up down up down up down up down
Posted by JasonID, 03-20-2004, 06:00 AM
Yep... I am going to bed now. This is getting boring. It can't seem to stay up for 15 minutes.
Posted by mark1hos, 03-20-2004, 06:01 AM
FYI - 4 Hours and counting! I have been speaking with Techs and they said its a router on just a section of their network which is affecting a few 100 servers, so they say. But like I said we are now in the forth hour of network issues. I feel for you all :-(
Posted by odsdc, 03-20-2004, 06:10 AM
hmm, been up for about 15mins now ..
Posted by JasonID, 03-20-2004, 06:11 AM
Not quite... don't jynx it
Posted by JasonID, 03-20-2004, 06:14 AM
OMG... it is gone again.
Posted by odsdc, 03-20-2004, 06:16 AM
still up here, been solid at my end for coming 20mins
Posted by JasonID, 03-20-2004, 06:17 AM
2-8 is down.
Posted by mark1hos, 03-20-2004, 06:18 AM
Done here too.
Posted by SW-Ray, 03-20-2004, 06:26 AM
http://forums.servermatrix.com/viewtopic.html?t=5298 So hopefully, all will be well from now on.
Posted by BizB, 03-20-2004, 06:26 AM
so far its up for me
Posted by andrewbrooks, 03-20-2004, 06:43 AM
Heres what i get from my server at 69.56.198.154 Pinging 69.56.198.154 with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Ping statistics for 69.56.198.154: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), Any ideas?
Posted by anand247sm, 03-20-2004, 06:45 AM
WTF is up with SM ?? Lookes like the extra burstnet customers have thrown them overboard and degraded their performance and support delivery.
Posted by anand247sm, 03-20-2004, 06:46 AM
My 5 boxes are going up and down every now and then
Posted by anand247sm, 03-20-2004, 06:48 AM
back again
Posted by aitor, 03-20-2004, 06:48 AM
I am getting same problem with my server ip 69.93.196.178. I have requested reboot 4 hours ago and no reply to it. Strange part is that Monitoring Report say is online and i can´t access from other sm server, but sometimes it is up. I am thinking that is line problem. Bad part is sm no info about it.
Posted by frozen, 03-20-2004, 06:49 AM
So submit an outage report, they do stand by their 99.9% uptime garantee and you will be credited the difference.
Posted by andrewbrooks, 03-20-2004, 06:54 AM
Alrighty then, at least i know we arent alone.
Posted by webrats, 03-20-2004, 06:54 AM
my theplanet server is up www2.webrats.com
Posted by Imago, 03-20-2004, 06:55 AM
Yesterday my box was down 6 hours, today - more than 7 hours (so far). I wonder how much would this make in terms of differential credit.
Posted by anand247sm, 03-20-2004, 06:56 AM
LOL they take 3-4 days to reply on a support ticket, i wonder how much time they will take to reply to outage report
Posted by frozen, 03-20-2004, 07:00 AM
About 20 minutes for me. Guess they love me
Posted by Imago, 03-20-2004, 07:00 AM
Yesterday it took them 20 minutes to answer the outage report (first time) and 90 minutes the second time.
Posted by anand247sm, 03-20-2004, 07:01 AM
well yes, looks like they surely do.
Posted by anand247sm, 03-20-2004, 07:03 AM
i had 1 box ordered with custom partitioning, and they screwed up the partition sizes. LOL and replied back to me stating it was a complex install so it was messed up. changing partition sizes mean complex install ??? wow Gave me atleast 5 days of downtime on that box. And even after fixing the partitions, they didn't install cpanel on it. I did the complete install myself. 3 Days and still waiting for their reply on downtime compensation.
Posted by anand247sm, 03-20-2004, 07:06 AM
car2-8.dllstx2 down again
Posted by Imago, 03-20-2004, 07:06 AM
Anand, you seem to be very much nirananda :-(
Posted by anand247sm, 03-20-2004, 07:09 AM
what do u mean ???
Posted by Imago, 03-20-2004, 07:11 AM
niranandam - lack of happiness in Sanskrit
Posted by anand247sm, 03-20-2004, 07:16 AM
Dude you could sit tight but its my business on fire right now. Customers are calling left right and center and I am answerable to them. I went to SM because of their reputation, but the first month itself is turning to be terrible.
Posted by forumtalk, 03-20-2004, 07:17 AM
upnow
Posted by desman, 03-20-2004, 07:18 AM
Is that language absolutely needed? SeverMatrix is a solid company IMO. I have 2 boxes there and only one affected; people, they did their best in the bad situation; they were faced with this late on Friday evening... They fixed the issue, and as a customer Im happy with the result. They have my vote as a rock solid company in a crunch situation.
Posted by anand247sm, 03-20-2004, 07:19 AM
yup its up now Hope it keeps that way "fingers crossed"
Posted by anand247sm, 03-20-2004, 07:23 AM
As i said earlier, i went with SM because of their reputation only. Sorry about the language, probably i m too frustated listening to customers for so long I don't doubt the company, but i expect this to be solved fast as its costing me a lot already. I have 5 boxes affected going up and down every now and then.
Posted by dredy, 03-20-2004, 07:30 AM
my server (at theplanet/servermatrix) has down and come back continious on today. There is a network problem. Server is alive but unreachable. Server has down 3-4 times on today, each downtime take 20-30 minutes! Total downtime is approximately 1,5-2 hours for today!!! I contacted theplanet about this issue but they didn't give me an answer yet... (i have this problem for 3-4 hours!) Has anyone got this problem @theplanet?
Posted by Imago, 03-20-2004, 07:30 AM
It is still down for me. Rock solid down.
Posted by anand247sm, 03-20-2004, 07:32 AM
mine working perfect rite now. Just praying now
Posted by desman, 03-20-2004, 07:34 AM
Sure its tough sometimes, but as far as I can see the issue has been resolved Correct me if Im wrong
Posted by Imago, 03-20-2004, 07:38 AM
Unfortunately, you are wrong! :-(
Posted by anand247sm, 03-20-2004, 07:38 AM
Yes for now. I believe that post of mine was posted before the issue was solved
Posted by desman, 03-20-2004, 07:39 AM
I hear that
Posted by anand247sm, 03-20-2004, 07:41 AM
All routers show up status inside orbit.
Posted by desman, 03-20-2004, 07:42 AM
Nevermind
Posted by desman, 03-20-2004, 07:44 AM
Same here
Posted by anand247sm, 03-20-2004, 07:44 AM
now what happened to you ?
Posted by Imago, 03-20-2004, 07:45 AM
Try to reach any of my sites http://www.indology.net http://www.slavica.org http://www.orientalia.org http://www.iztok.net http://www.indology.ru http://www.vdsp.net http://www.medicum.net
Posted by ToOnZ - SGWHT.com, 03-20-2004, 07:46 AM
Actually....if you bothered to check out their forums or look slightly lower down this thread... you will find out what you want to know
Posted by desman, 03-20-2004, 07:47 AM
Talking to you Joking
Posted by desman, 03-20-2004, 07:49 AM
Cannot here in Australia. DSL
Posted by anand247sm, 03-20-2004, 07:50 AM
i can't reach them either. If you give me the ip address i could try to trace from inside my servers at SM. You can PM me the ip.
Posted by anand247sm, 03-20-2004, 07:52 AM

Posted by TopQHost, 03-20-2004, 07:52 AM
yep. check out the other forums: http://www.webhostingtalk.com/showth...hreadid=250047
Posted by desman, 03-20-2004, 07:52 AM
Are these on SM boxes?
Posted by coight, 03-20-2004, 07:54 AM
Imago, bind is down on your machine.
Posted by frozen, 03-20-2004, 07:55 AM
Maybe you should request a reboot, all servers I have have been up and running for a bit now wiith no problems(they were down at the same time as yours earlier). We run a business too, we have customers that get mad, etc. This is something you have to factor in when planning your business. Every company will have errors here and there, from SM/TP to EV1 to RackSpace.
Posted by desman, 03-20-2004, 07:56 AM
From what
Posted by coight, 03-20-2004, 07:57 AM
And apache it seems
Posted by www, 03-20-2004, 07:57 AM
Imago, your DNS servers seem to be down. Have you called them yet?
Posted by Imago, 03-20-2004, 07:58 AM
Yes, desman, they are. Thank you, Anand: "the traceroute fails."
Posted by anand247sm, 03-20-2004, 07:58 AM
check my PM.
Posted by coight, 03-20-2004, 07:58 AM
Adverse effects of alcohol
Posted by desman, 03-20-2004, 08:00 AM
Sounds fimilar - For me that is Last edited by desman; 03-20-2004 at 08:03 AM.
Posted by anand247sm, 03-20-2004, 08:00 AM
guess i was late in posting :-P
Posted by anand247sm, 03-20-2004, 08:02 AM
i am gonna take a break now probably. Imago let me know if you need nething else from me.
Posted by desman, 03-20-2004, 08:05 AM
Yes I think this one is just about finished
Posted by anand247sm, 03-20-2004, 08:12 AM
LOL if i could help the topic be buried long down and never a situation arise for the same topic to come up. After all who wants the SM to get in trouble again.
Posted by Imago, 03-20-2004, 08:12 AM
Thank you, Anand. Good night, or whatever time it is. Here is early afternoon. :-)
Posted by anand247sm, 03-20-2004, 08:13 AM
np, netime. Its day for me stil neways good day to you.
Posted by kaikai, 03-20-2004, 08:17 AM
is it fixed yet? Thanks all
Posted by anand247sm, 03-20-2004, 08:19 AM
yup from the looks of it. Check your boxes.
Posted by aleck, 03-20-2004, 08:21 AM
looks ok. *still crossing fingers*
Posted by anand247sm, 03-20-2004, 08:21 AM
Recieved a reply from SM about outage complain, and ticket closed. Outage Report March 19, 2004 Summary: At 23:00 CDT, ServerMatrix engineers were alerted that one of our customer VLAN routers (CAR) was not responding. It was determined that the router had crashed and it was reloaded and returned to normal service. Within several minutes 2 more customer routers crashed as well. Upon inspection, it was determined that the cause of these crashes was a vulnerability in the BlackIce personal firewall program. The details of the vulnerability are located at http://www.eeye.com/html/Research/Up...20040213.html. In order to stop the routers from crashing, Planet engineers were required to locate and disconnect all affected servers. In addition, filters were applied at our border routers to prevent further exploitation of the vulnerability. Planet engineers are continuing to monitor all of the routers to make sure the problem does not re-manifest. Future Mitigation: The filters will be left at the border for an indefinite time to prevent future exploits. In addition, we would suggest that if you utilize the BlackIce firewall system, that you patch your software as soon as possible. If you have any questions about this, please contact our network security team at security@theplanet.com. Thank You, William Charnock VP Network Engineering The Planet Internet Services Hmm... i never use that firewall.
Posted by desman, 03-20-2004, 08:28 AM
Call me confused with that statement. BTW LOL means Lots of laughs & more... Check it out here http://en.wikipedia.org/wiki/Internet_slang Cheers Desman
Posted by desman, 03-20-2004, 08:30 AM
Still haven't received a reply, thanks for this... Maybe because I have only 2 servers Anyway SM still rock
Posted by anand247sm, 03-20-2004, 08:32 AM
plain and simple i meant, if i could help it, the servers / network of SM never go down and a situation like this never arise that we need to discuss here for network downtime. But ...
Posted by desman, 03-20-2004, 08:34 AM
I Understand
Posted by anand247sm, 03-20-2004, 08:34 AM
yup they do.
Posted by anand247sm, 03-20-2004, 08:36 AM
I am glad
Posted by desman, 03-20-2004, 08:38 AM
Hmmmmm Thanks
Posted by dpny, 03-20-2004, 10:22 AM
Amazing... Live update @ WHT. much better then TP / SM lolz this thread is like getting more update then their tech's have hair on their head
Posted by aleck, 03-20-2004, 10:29 AM
sure
Posted by aitor, 03-20-2004, 10:33 AM
I receive also the firewall reply, and i don´t also use firewall. What i see as really bad is they take 6 hours to say me that. If they know problem why they didn´t say me so i can say to my customers? Big headache is this.
Posted by CyberBabe, 03-20-2004, 10:40 AM
Give SM a break. I imagine that it must have taken them a while to figure out that (of all things) some customer's firewalls were screwing up the routers. It's not the first thing that pops in your mind when a router goes down. Then they have to walk around and manually disable BlackIce on all those servers. So I think they were pretty busy last night. I'd rather have them work on the problem and notify me later, than spending time answering tickets and not fixing the problem.
Posted by aitor, 03-20-2004, 10:50 AM
I take a break to sm, they make a good work. But 6 hours without know what is happening is bad. I have calls from my customers asking why didn´t work and i don´t know why. As my father says. If someone writes to you at least you must read it.
Posted by Aussie Bob, 03-20-2004, 10:57 AM
You need to do both - Make quick announcements informing folks there's an issue and that you're working towards a resolution. Give 15 to 30 minute progress updates. Takes about 2 minutes to announce a problem and minutes to make updates. There is no excuse from anyone who calls themselves a hosting professional, to leave paying customers in the dark, while you rectify the situation. Do both and your clients will love you for it. If you leave them in the dark, then you are biting the hand that feeds you, and that's bad for business.
Posted by CyberBabe, 03-20-2004, 11:14 AM
Sure, but if there is a major outage and there aren't enough resources to do both, I'd rather have them concentrate on the former. Fortunately the outage was virtually being "live blogged" on their support forums so I think most people knew at least what the reason for the outage was.just not how long it would take to fix. Then again, I don't think SM knew that either since I doubt they knew who had BlackIce and who didn't.
Posted by beady, 03-20-2004, 12:17 PM
I work at a large webhosting company, and from aproximately 12:30am this morning we experienced the effects of a very nasty worm believed to be either "Witty" or a variant thereof that was replicating like wildfire on our network using a compromise on Windows servers running BlackIce Defender. The traffic pattern was so severe that it caused extremely high latency and even outage on our Extreme Networks equipment causing the Extreme routers to throw error logs almost identical to the Slammer WORM. After blocking port 4000 inbound and outbound, the network stabilised. However, we then discovered that this worm had compromised several of both our own and our customers Windows Servers and damaged many of them to the extend that they will not even boot into Windows, some of them even "looping" on boot, and others bluescreening on loading Windows. These are currently being rebuilt from backups, but the damage is so severe that even the partition is not recognised on some of the drives even when using advanced recovery software. I believe this is just the beginning of an extremely nasty future WORM epidemic and that we may have been one of the very first affected in this manner, because very little material is currently available on the NET, especially regarding the damage caused to compromised servers. This post is both a warning to other ISP's and a question as to whether other ISP's or hosting customers reading this post have or are currently experiencing identical symptoms on their networks / servers. I think the limited public alerts available completely understate the seriousness and "maliciousness" of this worm from our experience. e.g securityresponse.symantec.com/avcenter/venc/data/w32.witty.worm.html
Posted by thedavid, 03-20-2004, 12:31 PM
As promised last night. Total downtime for servers behind CAR2-6.DLLSTX2 was: 0d 1h 50m 41s Total downtime for servers behind CAR2-8.DLLSTX2 was: 0d 3h 30m 18s This was independently monitored from the ev1 datacenter, at 90 second intervals.
Posted by Shaw Networks, 03-20-2004, 12:35 PM
Nice monitoring thedavid, I knew I was going to be able to get some exact statistics from you
Posted by cybexhost1, 03-20-2004, 12:36 PM
This very well could be the causes of all of the Routers crashing last night.
Posted by cybexhost1, 03-20-2004, 12:41 PM
The Planet/ServerMatrix's OFFICIAL Report: At 23:00 CDT, ServerMatrix engineers were alerted that one of our customer VLAN routers (CAR) was not responding. It was determined that the router had crashed and it was reloaded and returned to normal service. Within several minutes 2 more customer routers crashed as well. Upon inspection, it was determined that the cause of these crashes was a vulnerability in the BlackIce personal firewall program. (Looks like you are right Beady ) In order to stop the routers from crashing, Planet engineers were required to locate and disconnect all affected servers. In addition, filters were applied at our border routers to prevent further exploitation of the vulnerability. Planet engineers are continuing to monitor all of the routers to make sure the problem does not re-manifest. Future Mitigation: The filters will be left at the border for an indefinite time to prevent future exploits. In addition, we would suggest that if you utilize the BlackIce firewall system, that you patch your software as soon as possible. If you have any questions about this, please contact our network security team at security@theplanet.com. Thank You, William Charnock VP Network Engineering The Planet Internet Services Total Downtime: A little over 3 1/2 hours. As per SM's SLA - only ~46 minutes is needed for a refund.
Posted by null, 03-20-2004, 12:49 PM
How much can we get as a refund?
Posted by thedavid, 03-20-2004, 12:53 PM
Little. 99.9% uptime is awful close to what they're providing, if their service stays up for the entire rest of the month without interruption. Best bet - if they have any more significant downtime later this month, submit a SLA request for the combined times of that downtime plus this one.
Posted by Nessun, 03-20-2004, 01:56 PM
some of us have 100% SLA those on the ThePlanet network.
Posted by cybexhost1, 03-20-2004, 01:58 PM
As far as I know, The Planet doesn't use the routers effected.
Posted by Nessun, 03-20-2004, 02:09 PM
heh Considering my server is a Theplanet server they do.
Posted by cybexhost1, 03-20-2004, 02:29 PM
What kind of Server is it?
Posted by StartAnISP, 03-20-2004, 03:15 PM
So umm... How does one tell if he has this virus already? Symantec says it only sits in memory and can't be detected by virus software. I could reboot and then find out but then it may not reboot.. Yuck.. Any way to find out if I a server is already hosed?
Posted by cybexhost1, 03-20-2004, 04:06 PM
It effects the network routers, etc. as it did with The Planet/Server Matrix. You don't have to worry about this effecting your individual server. If you do utilize the BlackIce Firewall, it is recommended that you update your software so it is more compatible with the software update on the router. But that's about it.
Posted by StartAnISP, 03-20-2004, 04:29 PM
Yes but from what Symantec says, one of the problems with this Virus is that it trys to overwrite the first 128 sectors of the hard drive or something to that effect. If it's on one of my servers it would be nice to know
Posted by cybexhost1, 03-20-2004, 04:34 PM
Do you utilize the BlackIce Firewall? If so, update your software. If not, that is how this bug gets in - through a vulnerability in it. So if you don't utilize it, you won't have to worry.
Posted by cybexhost1, 03-20-2004, 05:05 PM
The problem was a Vulnerability - a Vulnerability in which they have since fixed. I believe it is now safe to close this thread.
Posted by Homer-HB, 03-20-2004, 08:51 PM
Damn I was asleep through all of this, sounds like quite a drama! Dave, whats the total downtime of car2-4?
Posted by Aussie Bob, 03-20-2004, 08:57 PM
It takes less than 1 minute to make a quick announcement, letting folks you know there's an issue [even if they don't really know what it is], and they're working towards a resolution. Then quick updates every 15 to 30 minutes. There are no excuses for no communication. I can't understand how hosts can think differently.
Posted by thedavid, 03-20-2004, 09:14 PM
I don't have any servers behind that switch.
Posted by joshiee, 03-20-2004, 10:05 PM
You know.. I feel bad for the people whom switched from burst to SM particularly.. They just switched from burst to avoid their down time.. now right after SM has one too.
Posted by cybexhost1, 03-20-2004, 11:51 PM
I am one of those. I being CybexHost.
Posted by iNCLiP, 03-21-2004, 01:35 AM
yeph it sucks, but my server was down 3 times for about 15 min. last night. my box at burst was down for more then 20 hours just a 2 weeks ago, it's a huge difference
Posted by cybexhost1, 03-21-2004, 02:08 AM
Again.....
Posted by panopticon, 03-22-2004, 03:04 AM
I assume the BlackIce Personal Firewall exploit which was the root cause of the problem was on customer servers, right? What I don't understand is why exactly this caused three of the servermatrix routers to keep going down? Did the attackers use the Personal Firewall vulnerability to intentionally take the network offline? Or was that an inadverntent side effect of too much traffic from the exploited servers?
Posted by wrcharnock, 03-22-2004, 12:34 PM
The routers crashing was an inadvertent side effect of the worm. There was a larger number of servers affected by the worm on the 3 routers that crashed. Unfortunately, it took some time to isolate which servers were infected and remove them from the network. William Charnock VP Network Engineering The Planet Internet Services
Posted by eXecution, 03-23-2004, 07:15 PM
I know this is a bit old, but can I get the SLA money back guarantee still ?
Posted by BizB, 03-24-2004, 03:13 AM
yes you can get back some of the money just put a ticket to accounting they just gave me back 20% on the server
Posted by mpjetta, 03-26-2004, 11:28 PM
I agree. All of my routes are now going through level3 and everything is working 100% again.