Alabanza Down !

Portal Home > Knowledgebase > Articles Database > Alabanza Down !

Posted by u2web, 09-17-2003, 08:31 PM
Appears that Alabanza's data center in Baltimore is down since about 7:10pm EST, 9-17-03. Their web site is down, my reseller site is down, all my client's are down. Not one to panic, but it's been over an hour! Thanks for any information.
Posted by gromit, 09-17-2003, 08:40 PM
Anyone have any info? thx
Posted by Sirealtor, 09-17-2003, 08:49 PM
all numbers are either busy or not in service? hope they are still in business. must be something serious hope it is fixed soon anyone get in touch with them?
Posted by centrahost, 09-17-2003, 08:52 PM
Alabanza? You use this company? They still around? You better run and do it now!
Posted by gromit, 09-17-2003, 08:55 PM
There was a small fire on the 3rd floor of the NOC and the FD wanted to shut everything down to be sure. I hear things should be back up shortly. No servers were affected. Fire was not in alabanza's space
Posted by Sirealtor, 09-17-2003, 08:56 PM
how did you find that out?
Posted by gromit, 09-17-2003, 08:57 PM
Why do you say that?
Posted by Sirealtor, 09-17-2003, 09:00 PM
just ignore him, and he will go away
Posted by gromit, 09-17-2003, 09:02 PM
I found out from sources inside the belly of the beast itself. Only thing I know so far is that there was some type of fire - described to me as "small". I asked about ups's, redundancies and all that and was only told that the FD wanted to make sure all electricity was off in the building and im sure that would affect all the incoming lines anyway which the ups's are not really designed to handle.
Posted by centrahost, 09-17-2003, 09:11 PM
10 years in the industry counts for something.
Posted by gromit, 09-17-2003, 09:18 PM
true - but cant we all say that? Im just wondering what your reasons are for saying what you do. Perhaps your 100% on the mark but im interested not only in the opinion but the reasoned opinion.
Posted by Sirealtor, 09-17-2003, 09:21 PM
since 1998 + 10 years = 2008 so are you early a few years?
Posted by u2web, 09-17-2003, 09:21 PM
Thanks for the update ... sounds like NAC!
Posted by centrahost, 09-17-2003, 09:26 PM
I dont know much about real estate either.
Posted by u2web, 09-17-2003, 10:27 PM
Alabanza is still down, over 3 hours ...
Posted by serverdummy, 09-17-2003, 10:49 PM
Still down...
Posted by marksy, 09-17-2003, 10:55 PM
Anyone know anything about this for sure?
Posted by Andrew, 09-17-2003, 11:11 PM
Well, I know Alabanza is still in business...they called me the other day, requesting to send me more information about their services. I said "You ARE aware we are a competitor of yours, yes? It's not as though we're low profile and hide that." They didn't seem to care...lol
Posted by centrahost, 09-17-2003, 11:33 PM
This thread is a good indicator of the uneducated market they continue to attract and feed off of by emailing the client databases.
Posted by u2web, 09-17-2003, 11:33 PM
hello gromit ... any updates? Over 4 hours seems like more than a "small" fire to me. Anyone remember how long NAC went down a few mos. ago due to a "small" fire in their UPS room? ~ money flys out of pockets ~
Posted by marksy, 09-18-2003, 12:09 AM
no doubt, morons clearly should have seen a fire - same with those dip****s in NAC when they had theirs...man, those dumb uneducated people.....
Posted by gethybrid, 09-18-2003, 12:20 AM
Over 450,000 websites are affected by Alabanza's outage. If the entire data center in Baltimore is down - nearly a million websites could be without service. One of Alabanza's resellers - Christian Web Host (www.christianwebhost.com) has reported that a DOS (Denial of Service) attack has occured and closed down much of the data center's network. With Alabanza's phone systems down, and many of the numbers out of commission, we will have to wait before we know for sure what has happened. However, rest assured that hundreds of thousands of dollars are being lost each hour the server is down.... Hopefully, we can all keep praying and hoping that our internet lives come back soon...
Posted by Elen47, 09-18-2003, 12:23 AM
PG&E mandated that alabanza shut down due to an electrical emergency in the basement of the building. Alabanza made a quick post on Alabanza.info and kept it up for as long as possible before shutting down. They were able to shut down and will begin bringing them back up once PG&E installs a new piece of equipment in the building. Again, Alabanza didn't lose power, they were forced to shut down due to the emergency. PG&E estimates begin able to bring things up at 9:40 EST however, you can't hold Alabanza to tthis. Once power is restored and Alabanza is given the okay the servers will come back on line. They've already brought extra staff in to make sure things are brought up as quickly as possible. As you can see it is well past 9:40 PM ........... just info for your consumption.
Posted by WorkinIt, 09-18-2003, 12:40 AM
http://abw.infopop.cc/6/ubb.x?a=tpc&...9&m=6396087924 It is ugly everywhere! I am wondering, how can the redlands CA servers be down as well? Wasnt the "fire" in Maryland???
Posted by pj50, 09-18-2003, 12:41 AM
Guys I am getting a little worried, I have been with Alabanza for over 1 year and never have had more than 5-30 min of downtime every couple months. It's 1am EST time, and everything is still down. I called at 11pm and got to speak with support they said it was a small fire in the lobby like everyone explained, and the data is all secure. But now all the phones are down. I hope things are back up before 7am tomorrow morning, this is getting serious. Think about it, more than a couple percent of the Internet is down right now, over half a million domains. Regards.
Posted by WorkinIt, 09-18-2003, 12:43 AM
I cant believe people got suckered into buying accounts from alabanza in the first place!
Posted by gethybrid, 09-18-2003, 02:48 AM
Alabanza has definately experienced hard-times. I just got off the phone with support personnel from Alabanza's Tech Support office located in Redland, CA. It took about an hour to get someone on the phone, repeating the "hang up and try again" about 10,000 times. Finally, I got a human being. His name was Steve, and he said that there was a fire in the lobby of Alabanza's Baltimore office. The fire department came, put the fire out and evacuated the building. Per fire code, the building had to be shut down and electricty to the building was cut as well. The NDC (National Data Center) is pending re-opening after the investigation is completed and the building is turned back over from the fire department. According to the official from Alabanza, this could be anywhere from a few hours to days. They have no way to know how long it will take for the investigation to be completed. For those of you who are setting, as I am, pushing the refresh button - all we can do is hope and wait. Many people around the world has been affected. Alabanza reports having over 100,000 customers and the data center provides service to nearly 2 million websites worldwide. Therefore, we are definately not alone. The world seems much smaller tonight, in retrospect we all hope things are restored soon. In our business, time is money. So, if anyone wants me to refresh their pages for them - while they go take a nap. Please feel free to post the URL's in the board. I'll keep refreshing. I know I can't help, but for some reason - it seems someone has stolen my business from me... so I can't sleep anyway. Waiting through the night... and morning..
Posted by gethybrid, 09-18-2003, 03:09 AM
Hey Everyone.. It is now 3:02am EST and the servers appear to be coming back online. A recent scan of IP addy's registered through Internic for Alabanza, show that they are coming back online. Everyone who is still up with me, keep that refresh button going - hopefully we will all be back up in a matter of just a few minutes. When we're back up... everyone should start the email cycle going to Alabanza. In a couple other forums that I've participated in this evening, everyone is saying that they are going to demand service credit. Seven hours of lost service is an incredibly large amount of time for any host provider. I plan on contacting someone tomorrow (we'll later today) at Alabanza about my account. They either will credit me for time lost or I'll be moving to rackspace. Maybe they will get a couple of my customer's emails too... I'm sure they will want to read the dirty emails I will have tomorrow. I wish everyone luck in dealing with this issue... it's been a pleasure holding fort with you on WHT tonight. Everyone take care!
Posted by gvard, 09-18-2003, 07:24 AM
Good evening from Greece, The response I had in my e-mail was the following: ---snip!---snip!---snip!---snip!---snip!--- Dear Valued Alabanza Client, Late last night BG&E, the Baltimore electric company, took the alabanza building offline in order to repair damages due to a fire. This action was conpletely unexpected, however our top notch sysadmin and noc team executed our emergency plans immediately. They brought all servers offline and shutdown in an orderly manner. BG&E had at one point brought power back up, however it was greatly fluctuating. Our Sys team made the decision to not bring the servers back up until the power surges stopped in order to protect our client's harward and data. The power has been stablized and all servers are coming online at this time. If you have any questions, please write or call in to support. Sincerely, Alabanza Corporation Alabanza Technical Support --------------------------------------------- My server was down for about 6,5 hours (02:00-08:30 GTM+2). It's been 5 hours since then and many servers from Greek customers are still down.... Sincerely, George Vardikos HyperHosting http://www.hyperhosting.gr/
Posted by WorkinIt, 09-18-2003, 08:39 AM
So I dont get it, did this happen in Redlands California or Baltimore Maryland? Last edited by WorkinIt; 10-01-2003 at 05:07 PM.
Posted by marksy, 09-18-2003, 09:40 AM
Happened in Baltimore. If you get compensation, let us know. Ala is notorious for for not giving the same across the board unless called to the carpet on it.
Posted by WorkinIt, 09-18-2003, 09:56 AM
Compensation??? Funny Last edited by WorkinIt; 10-01-2003 at 05:07 PM.
Posted by marksy, 09-18-2003, 10:07 AM
Just start another thread so those of us with a clue about Alabanza can get some discussion about affected business as opposed to answering statements like this. Alabanza hosts hosts, not sell individual websites, nobody here is in a $30 range much less 4 digits for many.
Posted by gvard, 09-18-2003, 12:05 PM
Greetings from Greece, Here is the announcement I've just received from Alabanza..... ---Snip!---Snip!---Snip!---Snip!--- Dear Valued Alabanza Client, Last night, we suffered an fire that was not related to the power source but within power switching equipment within the building. The fire department mandated around 7pm that all main and auxiliary power be closed down. Alabanza activated its Disaster Recovery Plan which worked nearly flawlessly. The plan is available to Alabanza clients and will be updated and resent to you. The only thing that went wrong in the plan is the alabanza.info site which is supposed to be giving you constant updates during a disaster is being overhauled and wasn't available. However, the good news is all technical positions reported quickly to their posts and were ready, willing, and able to help -- despite the risk of smoke inhalation they were working hard to make sure client servers were shut down properly. Around 9:45 power was restored, but then the electricians found another problem and forced a hard shut down of power which forced most of our servers that were coming up to go down hard. This caused the unfortunate situation of a longer term downtime due to servers needing to do disk checks when the power came back on. I chose the location for Alabanza for two reasons; Power and Bandwidth. Alabanza literally sits on top of a major central point of bandwidth for the east coast. Way before the internet, every long distance call out of the Baltimore/Washington area passed through our building. When data started traveling over wire, our building was one of the first places to be built out in the world, and when data started travelling over fiber, our building was well equipped by companies like MFS and UUNET. This was all thanks to the former major tenant of the building, the WR Grace Company. Today, our basement is home to major routing equipment for ATT, MCI/WorldCom, FGC, Sprint, Qwest and several others. Anytime you see 'bwi' in a traceroute you are most likely in our basement. The second reason I chose the location is power. The power is on a redundant grid with three power stations feeding into it. Everything between the power generation and the building is underground so there is no fear of losing power due to high winds. The grid has not failed for over 20 years, which is when it suffered a brown out due to an explosion at one of the power stations. The power source still did not fail last night -- internal power switching did. What are our future plans? We were already working on bringing up two separate feeds from the triple-generator power source in case the one internal feed fails again. Clearly, it would not have helped last night due to the mandate from the fire department, but it could help in the event of different scenarios. Thank you for your business, Thomas Cunningham Chairman and CEO Alabanza Corporation ---Snip!---Snip!---Snip!---Snip!--- Sincerely, George Vardikos HyperHosting http://www.hyperhosting.gr/
Posted by marksy, 09-18-2003, 12:15 PM
We got it too, this piss anyone else off? Hey everyone - here's why it's not our fault - blah, blah, blah...You pay us for power and bandwidth and we didn't have it up but it's not our fault. My clients don't care why their sites were down, they pay us to have it up - the rest is up to us. Rightfully so I might add. I pay them for power and bandwith I expect it to be there - if they chose to have the DC in a crackhouse I guess that wouldn't be their problem? Worthless letter.
Posted by WorkinIt, 09-18-2003, 12:30 PM
i especially love the cut and paste crap that alabanza gives its resellers to tell their clients.... Industrial-Grade Equipment for the Ultimate Performance and Reliability XXX recognizes that downtime is not an option for your servers. That's why we invest heavily in hardware and facilities that ensure that your clients are up and running on the web 24 hours a day, seven days a week. XXX's Network Operations Center (NOC) features raised flooring. This enables a constant flow of conditioned air and helps to maintain uniform room temperature at all times. In addition, raised flooring reduces static and ensures a professional, computer-grade environment for your servers. The facility is equipped with a Raytheon fire suppression system, designed to immediately extinguish fire and protect equipment and personnel. The command center is controlled via automatic doors to further secure and protect the equipment. Finally, the NOC is located in a secure, monitored, class A building with a minimum number of approved personnel allowed access to highly sensitive areas and equipment. A detailed record of employee and visitor entry is maintained at all times. Uninterruptible Power System To guard against local power failures, XXX has two industrial-grade, three-phase Liebert UPS systems. These act as back-up batteries, maintaining uninterrupted power in case of surges or power outages. With these back-up systems in place, we can keep our network up and running indefinitely without relying on external power. Industrial-Grade Air Conditioners XXX's NOC has two Liebert 10 ton industrial air conditioners that condition our computer rooms and operations center. Air temperature is maintained at an optimal 65 degrees Fahrenheit. Custom Web Servers XXX's web servers are custom-built industrial machines designed for a 24/7 web serving environment. All XXX servers are equipped with dual-redundant, 450-watt power supplies, swappable drives and force-filtered cooling systems. In addition, our NOC is equipped with an inventory of identically configured, burned-in standby servers. Force Filtered Cooling All of XXX's custom web servers are equipped with a positive-pressure filtered-air system. Four large fans pull filtered air into each server's protective case, and the components within are cooled by fans that circulate this purified air. This constant introduction of clean air into the case creates a positive pressure environment ensuring that dust and particles remain outside the server. Hot Swap Seagate Drives The drives and drive bays of all XXX servers are constructed from high-grade aluminum and rest in shock-mounted drive cages, which adds to the durability of the hardware. XXXs drives proudly feature the lowest failure rate in the industry. Redundant Hot Swap Power Supplies Each server employs dual-redundant hot swap power supplies. If a power supply were to fail, the server would continue running with power from the alternate supply. Meanwhile, alarms would alert a technician, who would quickly restore redundancy. In the meantime, servers and client sites would experience no downtime. Standby Servers We keep spare servers online of all CPU configurations. If a server were to experience a hardware failure, we would turn a key, grab the handle on the drive, pull it out and insert it into an identical standby CPU. We would then reboot the second machine, and the server would be up and running again in a matter of minutes. Connected to Three Backbones The XXX NOC, located in Baltimore, Maryland, is OnNet with GlobalCenter (GC), Qwest Communications and GTE through three separate bandwidth-on-demand connections that enter Baltimore in our building. GC, a Tier 1 provider with a 13,000-mile fiber optic network and Dense Wave Division Multiplexing (DWDM) technology that provides an enormous 460 gigabytes per second (Gbps) of capacity worldwide, has an Asynchronous Transfer Mode (ATM) fiber node located just a few floors below the XXX NOC. Qwest comes into Baltimore with an OC-12 line and plans to upgrade its connection to an OC-48 in the near future. Qwest also has an ATM fiber node just floors below the XXX NOC. Our Qwest connection enables XXX to offer additional redundancy and better routes to Europe, Latin America and Asia. With our carriers, our router has up to 150,000 possible routes to send each packet of traffic. Furthermore, because of these unique connections, XXX does not need to link to the Internet through an OC3 or T3 Telecom circuit. Instead, independent cables run inside our building directly from the XXX NOC to all three carriers' points of presence. These lines can handle the bandwidth of a T3 or an OC3 with DWDM. Plus, they handle several times the bandwidth of an OC3. Whatever your bandwidth needs may be, XXX has the scalability to meet them. Network Redundancy XXX uses intelligent end-user routing software called Border Gateway Protocol (BGP) between Qwest, GC and AT&T, which use the same protocol. BGP identifies which path is the most efficient for each data packet and then routes the packet to its destination on the fastest path. This increases the speed at which web pages sent from our NOC arrive at their destination. Studies have shown that the most common reason for downtime is circuit failure on Tier 1 provider backbones, the major data highways. To guard against this potential problem, we have three Tier-1 providers. If one experiences problems, we can route traffic down another one. Furthermore, because we are OnNet with GlobalCenter, Qwest and AT&T, we share their digital distribution architecture, which includes private peering network connections to major Internet carriers such as MCI, Sprint, UUNET, EUNET, AT&T, AOL, Best, Erols, @Home, IBM Advantis and others. These private peering arrangements allow XXX to quickly and efficiently exchange packets of data with every major backbone carrier in a one-to-one environment. In addition, GC has high-speed links to eight public exchanges including both MAE East and West and several NAPS. Through these public exchanges, customers have the ability to reach their sites, no matter where they are. Network Reliability Industry analysis reveals that 70% of downtime of over 10 hours with any ISP is caused by telephone circuit failure. With XXX, circuit failure is virtually eliminated. That's because our NOC is in the same building as GlobalCenter, Qwest and AT&T. There is no phone circuit between XXX and these providers. Instead, there is a direct connection between our Cisco 7500 routers and theirs. XXX's providers also have peering connections with other major Tier 1 providers that allow traffic to be switched to alternate backbones should the need arise. Raw Performance Equals Low Latency/High Throughput Too often providers operate their networks at three to four times their responsible capacity. As a result, their corresponding transfer times reach over 300ms. XXXs network daily average is 27% of its capacity, with midday peak spikes reaching only 33% capacity. XXX guarantees that clients will be carried off our network in less than 80ms over a five-minute average at any time of day or night. ha! talk about redundant! resellers, let this be a lesson to you, think before you start cut and pasting a bunch of lies on your website. id like to know HOW you explain this to your clients, when youve made them believe all along that YOU own the equipment!
Posted by Incognito, 09-18-2003, 01:39 PM
I am not an Alabanza fan and would never deal with them. However, in this case, what they are saying has the sound of complete truth. The fact is that if a fire occurs, then almost always the fire department will have all electricity and well as gas cut off to the building. This includes generators. The time they are kept off depends on the particular fire department, the size of the fire, the size of the building and other very subjective factors. In fact, in crowded areas, the fire department may shut off power to neighboring buildings or, as I know in one case, even the whole block.
Posted by Aussie Bob, 09-18-2003, 02:10 PM
You should be consulting directly for them, and enlightening those "dumb uneducated people" with your expert advice.
Posted by marksy, 09-18-2003, 02:47 PM
Not sure if you're missing the sarcasm or I'm missing yours. My comment was in response to this: "This thread is a good indicator of the uneducated market they continue to attract and feed off of by emailing the client databases." We still have some servers at Ala - one prong of a multi-sarcastic remark.
Posted by dr_jade, 09-18-2003, 04:59 PM
Hi all, Two of my sites are hosted with Alabanza resellers and they were down for about 8 hours last night - it really hurt. However, everybody needs to understand that we are dealing with a bunch of computers and some electricity, all of which can and will fail under extreme circumstances. I really dislike Alabanza from a management stand-point, however, as far as a service/reliability stand-point, I have to give them credit where its due. Anyone who has ever worked-in, or operated a data center, can somewhat relate to the Alabanza situation that happened last night. I am sure Alabanza has insurance for things like this so I think if the Resellers who were effected by the outage ask nicely enough, Alabanza will probably cough up a few bucks to make everyone happy. Hope everything stays normal for a while! -jade
Posted by Aussie Bob, 09-18-2003, 05:04 PM
Oops, sorry about that. I did miss the sarcasim in your post.
Posted by centrahost, 09-19-2003, 07:38 AM
This thread is better indicator of how much Alabanza is liked by some here at WHT
Posted by marksy, 09-19-2003, 01:35 PM
FYI, we filled out an SLA Credit Request Form..Here's a blank so you don't have to wait for your acct manager Attached Files credit request form.zip (11.5 KB, 36 views)
Posted by centrahost, 09-19-2003, 01:45 PM
Gotta love Marksy's last post.
Posted by WorkinIt, 09-19-2003, 02:08 PM
im still trying to figure out WHY someone would pay all that money to join alabanza???
Posted by gvard, 09-19-2003, 07:46 PM
Good evening from Greece, Alabanza's SLA states that if for some reason (including power problems) my server is offline, I'll e-mail emergency@alabanza.com. From the moment of the e-mail, they have two (2) hours to fix the error. After the two hours, 5% of my monthly fee will be returned for every 30 minutes of downtime. My server was offline for 6,5 hours. I e-mailed them at 02:00am (greek time) and my server got back online on 08:30am. 6.5 hours minus 2 hours, that's 4,5 hours. That's 9 halftime blocks, 9x5=45% will be returned to me from my monthly fee. *THAT'S* why someone would pay so much money to join Alabanza...
Posted by WorkinIt, 09-19-2003, 08:48 PM
so, youre saying that you are overpaying so that when THEY have downtime you get a refund??? errrr.... ok ps THAT much downtime for whatever reason just isnt acceptable to me. i dont think ive experienced that much downtime all together since 1999!
Posted by OMC, 09-19-2003, 10:41 PM
Well I guess they could have had gerbils in a cage running a treadmill to produce power. Considering the fire dept and emergency services PULLED THE PLUG. That would have been plan Z in their emergency preparedness manual.
Posted by WorkinIt, 09-19-2003, 11:02 PM
another interesting thing... i havent heard one person talk about the inconvenience to their customers or how they plan to compensate their customers.... will you be compensating them or just hoping they didnt notice?
Posted by Carol, 09-21-2003, 04:13 PM
For Alabanza hosts, this situation is discussed in a little greater detail over at the Alahosts.com forums (which are independently run). Most of the discussion is in the private area, but there are a number of posts about it in the public area, including a note from Tom Cunningham, Alabanza's CEO.
Posted by WorkinIt, 09-21-2003, 06:08 PM
ok tom says: Today, our basement is home to major routing equipment for ATT, MCI/Worldcom, FGC, Sprint, Qwest and several others. Anytime you see 'bwi' in a traceroute you are most likely in our basement. ok, so im wondering.... was ATT, MCI, FGC, etc etc down for 10 hours as well? Or did they have a back up plan that worked?
Posted by ceint, 09-20-2004, 08:25 AM
Hello, I'm from Norway and do not have access to http://www.alabanza.com/ with nor our server with them. Does anyone know if it's just fro here or is it a serious network issue over there that effects all Alabanza clients ? PS: I am unable to call them, all lines are busy. Thank you, Thomas
Posted by rackcheck, 09-20-2004, 08:28 AM
Hi, Site won't load for me. I'm in Germany.
Posted by AFMichael, 09-20-2004, 08:30 AM
Doesn't load for me. I'm in Florida, USA.
Posted by anon-e-mouse, 09-20-2004, 09:20 AM
Moved to outages and for the record: Pinging alabanza.com [64.176.204.111] with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Ping statistics for 64.176.204.111: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms
Posted by JenniH, 09-20-2004, 09:22 AM
Yup.... this this the Apollo thing.... see that specific post. There's a fire in the proximity. Power is out. I can't get any ETA from anywhere...
Posted by JenniH, 09-20-2004, 09:24 AM
The question of course... is don't these guys have UPS or similar?
Posted by JenniH, 09-20-2004, 09:26 AM
Anyone have any idea how long it will take to sort all these servers once power is back? Or do they recovery automatically? MORE NEWS And you're not going to like this: "Baltimore Gas and Electric Co spokeswoman Linda Foy said power had been cut to some buildings in the area, and those customers may not get power back before noon today." What is it there now? 1am?
Posted by Aussie Bob, 09-20-2004, 09:35 AM
Same sort of thing happened with the nac fire incident, a year or so back. Power cut, pending the fire people's work completed. They also would have had to evacuate the facility, while the fire folk's did their thing. You'll just have to ride this one out.
Posted by JenniH, 09-20-2004, 09:40 AM
>> You'll just have to ride this one out << The decision is whether to move the sites elsewhere and change the nameservers. That's why I asked about sorting out the servers. We can guess they might get power backing within 4-5 hours (maybe)... but how long to get around the servers and reboot them all? Any input on that? If we shift nameservers now, we will be running elsewhere within 24 hours. If not.... could we still be down then? This is the dilemma.
Posted by JenniH, 09-20-2004, 10:09 AM
Has anyone got a direct telephone number for these guys? The general 0800 number is engaged (surprise suprrise).
Posted by Arsalan, 09-20-2004, 10:10 AM
Since Sept 8 (i beleive), DNS servers are changed with in a few min, so you would be up with in a few hours. Not 24 hours as it used to take..
Posted by tlkern, 09-20-2004, 10:22 AM
Direct Phone number for Alabanza is 410-779-1400. It is currently 10:22 am in Baltimore. http://www.baltimoresun.com/news/tra...-news-breaking Baltimoresun.com staff and wire reports Originally published September 20, 2004, 7:47 AM EDT An underground fire overnight forced some street closures near City Hall and knocked out more than 500 traffic lights downtown, Baltimore fire officials said this morning. The fire was reported shortly after 1 a.m. at Guilford Avenue and Fayette Street, according to city Fire Department spokesman Kevin Cartwright. It blew out four or five manhole covers in the area and firefighters found smoke and flames shooting from one of the manholes. Cartwright said the fire was out, but the flames damaged high-powered electric cables underground, including a cable that feeds the traffic light system. Baltimore Gas and Electric Co. crews were on the scene and managing the electric problems. It was not immediately clear when any of the malfunctioning traffic lights would be back in service. According to city police, Charles Street was closed to Monument Street; Fallsway was closed to Baltimore Street; and Fayette Street was closed between Gay and Charles streets. The cause of the fire was under investigation, Cartwright said. Terri L. Kern http://www.abundanthosting.com
Posted by JenniH, 09-20-2004, 10:25 AM
That number is engaged too.... I would have hoped for a telephone message at least.
Posted by tlkern, 09-20-2004, 10:35 AM
I would suggest you ask your account rep for a cell phone or pager number for him/her in the future in the event that this happens again and you can't get through. Terri http://www.abundanthosting.com
Posted by JenniH, 09-20-2004, 10:39 AM
We don't host directly with Alabana, but via a couple of resellers. Apollo by the way are reporting that services should be up within 90 minutes.
Posted by dk4210, 09-20-2004, 10:48 AM
Hello, we use Alabanza for our hosting too. Does any one know why they can't just use the backup generators? Has the city requested them not to?
Posted by JenniH, 09-20-2004, 11:19 AM
Have they actually got backup generators? I can't see any reason why they would be asked not to, as the fire wasn't in the building itself, but down the road,
Posted by daz_mm, 09-20-2004, 11:23 AM
in the UK here, we've not had email/web all day. Is the underground/transport system out of action? Has anyone actually managed to get into work to fix/restart the servers?
Posted by mdstg, 09-20-2004, 11:30 AM
Baltimoresun.com staff and wire reports Originally published September 20, 2004, 9:50 AM EDT An underground fire overnight forced street closures near City Hall and knocked out traffic lights at some downtown intersections, fire officials said, closing both courthouses for Baltimore City Circuit Court for the day and giving some city employees at least the morning off. The fire was reported about 1 a.m. at Guilford Avenue and Fayette Street, according to city Fire Department spokesman Kevin Cartwright. It blew out four or five manhole covers in the area, and firefighters found smoke and flames shooting from one of the manholes. Cartwright said the fire was out before the morning rush hour, but flames damaged high-powered electric cables underground, including a cable that feeds the traffic light system. Baltimore Gas and Electric Co. crews were on the scene and managing the electric problems. It was unclear when the malfunctioning traffic lights would be back in service. According to city police, Charles Street was closed to Monument Street; Fallsway was closed to Baltimore Street; and Fayette Street was closed between Gay and Charles streets. The cause of the fire was under investigation, Cartwright said. There were no reports of injuries or serious traffic problems. Drivers at intersections with Charles Street, the city's main northbound avenue, that did not have working lights stopped before moving forward to check for oncoming traffic and pedestrians. At other intersections, police officers directed traffic. "It's moving smoothly -- drivers have gotten into a rhythm," said David Brown, spokesman for the city Department of Transportation. BGE spokeswoman Linda Foy said power had been cut to some buildings in the area, and those customers may not get power back before noon today. There was no electricity at City Hall, the Clarence M. Mitchell Jr. Courthouse, the Peabody Institute and other buildings in the area, Foy said. "It's only bright in here because the sun is shining," said Foy, who was at City Hall. She did not know how many BGE customers were affected. Foy also said the cause of the fire was unclear. Crews were working to make the underground area safe so they could take a good look at the damage and determine the extent and cause of the fire. Nonessential city employees who work in the buildings within an area bounded by Monument Street, Fallsway, and Baltimore and Charles streets were initially told by Mayor Martin O'Malley's office to stay home today. Later, they were told to monitor radio and television broadcasts because they might be ordered to report at noon. All other city employees should report as usual. Both Courthouse East and the Clarence M. Mitchell Jr. Courthouse of the Circuit Court for Baltimore City were closed for the day because of the fire and resulting electric problems. Jurors summoned to Circuit Court had the day off because of the power outage at the Mitchell courthouse, said Rita Beyer, a spokeswoman with the court information office.
Posted by JenniH, 09-20-2004, 11:37 AM
I think everything is fine... except they have no power. That's SUPPOSED to be back in the next half hour.... hmm. It's very disappointing that Alabanza can't provide any update, and have not provided any means of communication. That's me being diplomatic by the way. Not exactly impressive, nor is the lack of UPS.
Posted by EndOffice LLC, 09-20-2004, 11:49 AM
Seems like they might be up but their Upstream providers might be down. Their upstream providers appreantly are located in the same building as they are. They do have UPS power and supposedly own a generator (but if it is running who knows?). The building is a few blocks away from city hall (where the fire happend). Now I guess int he worst case some fiber got damaged? Who knows. Anyways here some info on alabanza from the web: Not hosting there, only have a friend who used to work for them a while ago. Not anymore too.
Posted by JenniH, 09-20-2004, 12:10 PM
Well... it looks like we've passed the promised return time. Not looking good. I found another major player who hosts there, OneWorldHosting, and rang them (number in Google cache)... answer machine mailbox full. I guess you can learn a lot about hosts at times like these. Still no joy on the main Alabanza number. Latest from Apollo: Fire in Baltimore Below City Hall has caused Network Issues with accessing our Network. All Systems are fine, we are awaiting a update from Baltimore Electric as to the amount of Damage to the Network Hub Below City Hall. We hope to have this issue resolved. Last edited by JenniH; 09-20-2004 at 12:14 PM.
Posted by JTMNeal, 09-20-2004, 12:18 PM
There is a forum at alahosts.com that has some information not good information but there is some http://www.alahosts.com/phpBB2/viewt...r=asc&start=30
Posted by Eanes03WFU, 09-20-2004, 12:37 PM
I managed to get through to OneWorldHosting earlier today on the phone, and was given the same reason--that the power was out, and that as soon as it was restored, the sites should be made available.
Posted by yomismo, 09-20-2004, 12:39 PM
Hello. We works with Alabanza too in Spain. Some companies works with Alabanza here. Our connections are down too. Is the fire notice true or is an error of Alabanza? Because this is not the first time of a "fire" there !!! Bye ABACOX
Posted by JenniH, 09-20-2004, 01:07 PM
Reading that unnofficial ALA forum is alarming. It sounds like Alabanza are a right load of *£!^"& ! We're an hour past 12 noon, and no sign of power, and some people are claiming 5-7 hours to get the servers restored! Looks like I'm gonna have to sort out a couple of new hosts pronto.
Posted by JenniH, 09-20-2004, 01:23 PM
Sorry, but I've just heard: "Last I've been told, the crews couldn't get deep enough to fix the cables, but were still working on it" This means it could easily be hours, or longer....
Posted by u2web, 09-20-2004, 02:07 PM
http://wjz.com/localstories/local_story_264071211.html Sep 20, 2004 12:50 pm US/Eastern Baltimore, MD (AP) BGE spokeswoman Linda Foy says it will be later this afternoon before power can begin to be restored to customers without power in the downtown area. She said it been hoped that some would begin to get their power back by noon, but the effort is taking longer than first expected. She says crews have been unable to get as deep beneath the street as they need to go to work on the electrical problem.
Posted by kent1688, 09-20-2004, 02:20 PM
Thanks for the info, mdstg. Hope the electricity will be restored soon.
Posted by Eanes03WFU, 09-20-2004, 02:28 PM
Did anyone ever say why the hosting facility hasn't engaged any sort of power backups (generators, UPS, anything...)?
Posted by JTMNeal, 09-20-2004, 02:34 PM
According to the thread at alahosts.com there was no backup generator installed. Instead the datacenter was on redudant power grids. There was to be a generator installed today, and it should be up and running within the next hour, atleast according to that thread.
Posted by Elen47, 09-20-2004, 02:46 PM
This is exactly why we left Alabanza. Almost one year ago to the day they had the same scenario and we lost a large number of clients after being down for almost 30 hours. We have never been down since we decided to partner with another NOC that truly does have backup power and use our own servers with another Administartion system!
Posted by cate1951, 09-20-2004, 02:58 PM
Just spoke on phone with ITX in Baltimore. Servers are running and ok BUT cannot connect to outside due to this fire. They are awaiting approval from the fire marshal to connect. This has been delayed from 1 pm to later, 4 pm. You can check on this info at the Baltimore Sun MD newspaper website with the latest updates.
Posted by davidt, 09-20-2004, 03:24 PM
Sorry for my ignorance, but who or what is "ITX"? DT
Posted by cate1951, 09-20-2004, 03:42 PM
ITX Design Web Hosting, a reseller for Alabanza.
Posted by davidt, 09-20-2004, 03:42 PM
Wait...I think I figured it out..."itxdesign.com" -- correct? (our posts crossed...you posted your answer while I was composing the first edition of this message) Here's the standard Alabanza-speak quote on the ITX "Network" page (archived at web.archive.org): "With these back-up systems in place, we can keep our network up and running indefinitely without relying on external power" hmmmmm......
Posted by cate1951, 09-20-2004, 03:46 PM
For shutting down Palestinian websites recently? Check out articles online, search Alabanza, news on Google. 1 of 3 US companies listed by Dept. of Justice...
Posted by Eanes03WFU, 09-20-2004, 03:52 PM
I host through OneWorldHosting which either shares the hosting facility with Alabanza or is a reseller for Alabanza--I don't know which--but I've used them consistently since 1999 and have never had an outage like this. So I chalk it up to things happening--I won't be changing.
Posted by blacknight, 09-20-2004, 03:52 PM
Not likely. The fire wasn't in Alabanza itself, but nearby.
Posted by cate1951, 09-20-2004, 03:56 PM
This fire shut down the fiber optic cable lines underground that Alabanza accesses. these lines were damaged in this fire. Not a electrical power issue, a communications line shutdown....vulnerability.
Posted by attila, 09-20-2004, 04:09 PM
Some news links http://thewhir.com/marketwatch/ala092004.cfm http://news.netcraft.com/archives/20...a_offline.html
Posted by Dublin2004, 09-20-2004, 04:18 PM
Us too, but this has cost us a lot of money today. We'll be pulling our web and email services in-house ASAP after this. I reckon a lot of other people will be walking too.
Posted by Elen47, 09-20-2004, 04:19 PM
The best thing we ever did was quit using Alabanza. At one point we had a thousand domains hosted. After all the network downtime and fires and excuses and no generators we found someone new and have had 0 downtime the past 16 months. Good luck using Alabanza.
Posted by kent1688, 09-20-2004, 04:27 PM
Elen, do you mind telling me which hosting company are you with now? I am considering switching too. Thanks
Posted by davidt, 09-20-2004, 04:30 PM
Um...are you sure? Alabanza is telling people on the phone that power has been restored and that they're going through the bootup process, which is sequential (as opposed to all at once) and will take hours. I was able to do a direct DNS query of "ns.alabanza.com" like this: $ nslookup alabanza.info ns.alabanza.com Server: ns.alabanza.com Address: 209.239.47.252 Name: alabanza.info Address: 216.110.45.169 I think their fiber might be OK. DT
Posted by JenniH, 09-20-2004, 04:41 PM
Well... OneWorldHosting.Com is back up... but obviously not our sites. At least something is happening. All we have from Alabanza is a pathetic grovelling note on the unofficial forum. Instead of giving an ETA or info to help people, they are on their covering themselves. Says it all really.
Posted by EndOffice LLC, 09-20-2004, 04:42 PM
That also might be because the tertiary nameserver is not in baltimore. So far routes still drop in DC and do not seem to get as far as Baltimore.
Posted by manadesigns, 09-20-2004, 04:43 PM
Just curious, did you have any specific information that said the problem was damage to the fiber and not just power.
Posted by JenniH, 09-20-2004, 04:47 PM
>> This fire shut down the fiber optic cable lines underground that Alabanza accesses. these lines were damaged in this fire. Not a electrical power issue, a communications line shutdown....vulnerability. << In which case, why do all the servers need booting? That's consistent with a power, not a network issue.
Posted by Elen47, 09-20-2004, 04:56 PM
Kent1688, If you have an email address I would send you that info. I don't want to seem like I am advertising on this forum.
Posted by manadesigns, 09-20-2004, 05:04 PM
I've been listening to WBAL.com news Radio in Baltimore and at 5:00 pm EST, they reported that some of the power has been restored but not all. Does anyone know if Alabanza does really have power back. Traceroutes are still not reaching.
Posted by JenniH, 09-20-2004, 05:09 PM
Yes, they have power back (as one of their main resellers is up)... but the re-boot process could take a long time. "NOC staff is bringing servers back up "slowly" according to Chad, until they check the load on the generator, when they are comfortable that they can do it quicker they will." Which in itself makes me wonder if this generator is up to the job!
Posted by JenniH, 09-20-2004, 05:10 PM
Oneworld are reporting "ETA for all services to be restored is 7-8 pm EST this evening." That's presumably for their servers... which I suspect may be given priority for some reason (as their site evidently has). We are in for a wait.
Posted by yomismo, 09-20-2004, 05:11 PM
I know a customer of Alabanza that it is working now. If we visited the web site 209.239.54.144 it is working now. What happen with the others customers like me !!!! BYE
Posted by JenniH, 09-20-2004, 05:23 PM
>> What happen with the others customers like me !!!! << You just have to sit tight (painfully). It's a sequential server by server process... you could be early.... you could be last. Actually, you could only be second to last, because I'm bound to be last!
Posted by kent1688, 09-20-2004, 05:45 PM
Thanks Elen, my email is acer2800@yahoo.com
Posted by Arsalan, 09-20-2004, 06:01 PM
I guess Alabanza dosent host that many sites as they used to. I am amazed to see just 4 pages for such a long outage and 0 comunication from them. A similar outage on theplanet/fastservers/dedicatednow/burst.net/managed would take up at least 10+ pages..
Posted by manadesigns, 09-20-2004, 06:14 PM
Well, at least Alabanza is back up. But if power was the problem then what does this statement mean on their web site, especially the part about "...running indefinitely without relying on external power....": >>Uninterruptible Power System To guard against local power failures, Alabanza has two industrial-grade, three-phase Liebert UPS systems. These act as back-up batteries, maintaining uninterrupted power in case of surges or power outages. With these back-up systems in place, we can keep our network up and running indefinitely without relying on external power. <<
Posted by jackpot, 09-20-2004, 06:26 PM
The servers seems to be getting back online now.....
Posted by hostmoon, 09-20-2004, 06:31 PM
Yep.. one of our servers we host there is back online as of an hour ago. Fortunately for us, most of our clients are in a different NOC dale
Posted by Elen47, 09-20-2004, 06:50 PM
This is really funny.................. Go to Bulkregister.com an Alabanza company that has been down all day and see the notice. Seems like getting the truth from taht company is close to impossible............................... The BulkRegister system is currently unavailable due to unscheduled maintenance from 1745 until 2200 EDT 09/20/2004. During this time, you will be unable to process any domain name registrations, transfers, renewals, or other similar transactions. We regret any inconvenience this outage may cause. Thank you, BulkRegister Support
Posted by Solothurnli, 09-20-2004, 07:18 PM
I got the facts from LinuxWebHost pretty easily via telephone. Guy there said there had been a fire in a communications tunnel in Baltimore, and the whole city was in digital cardiac arrest. Earlier, though, I couldn't even get anyone on the phone, so I'm assuming that a lot of fiber optics smoked as well. Don't envy the people of Baltimore.
Posted by davidt, 09-20-2004, 07:40 PM
It's meaningless, misleading, and obviously incorrect. Alabanza is working on putting some "spin" on the whole issue, and are very offended that people are calling them liars, but they've indeed led people to believe that this sort of thing woudn't happen again (it happened last year also). Their failure to have the generator onsite meant that they couldn't respond to today's problem, and their dedicated server clients (resellers) are complaining bitterly about losing a LOT of customers. The sites that I'm involved with are on an Alabanza-hosted reseller, and we've had it...we're heading for greener pastures. DT
Posted by Joshua, 09-20-2004, 09:58 PM
I can't believe that they don't have a generator... Anyone who thinks that UPS devices are sufficient for keeping a datacenter running when the power goes out is sadly mistaken. UPSes cannot power cooling systems and lots of servers for long. Even BurstNET has a generator & automatic transfer switch in addition to their UPSes, and they're a budget host.
Posted by davidt, 09-20-2004, 10:40 PM
Well, start believing. They admitted today that they were just now installing the generator...during an extended outage. They didn't even wind up using it, because by the time they got it all hooked up, the power company had turned the lights back on. But at least it will be there for the "next time" -- but by then, they'll be missing a lot of their current customers. DT
Posted by Elen47, 09-20-2004, 11:09 PM
Just one more time in a long list of BS from Alabanza. As soon as the power came on you can bet the tightwads running it sent back the generator ( that was supposed to be installed last year during a two day outage ) Any of you folks that want to believe their BS best of luck, because I am willing to bet there will be a nest time.
Posted by Coolraul, 09-21-2004, 02:02 AM
I don't know about having generators or not but.. Have you guys ever managed a datacenter? Seriously. Even a small one? Ok here is how it works. You have a fire, the fire department arrives and they tell you what to do which usually includes but is not limited to shutting everything down. This doesnt mean shut down everything outside and leave the core running. It means shut everything down. You see if you have a generator pumping power and the Firedepartment guys have to use this thing that ocmes out in hoses called "water" it has a tendancy to cause problems when you mix the two. Precautionary steps are to get everything down and off for a couple of reasons. If you happen to have to use water outside and it seeps inside, you really want the systems off too. You have some hope of recovery, if its on, much less. I have no idea what happened there but it rings true to me. Clearly newsworthy report regarding an outage. Sounds like the north east blackout last year. Very few could withstand that even if you stayed up, some communications facilities routing gear died.
Posted by davidt, 09-21-2004, 02:13 AM
Nope...you've got it all wrong. The fire wasn't even near Alabanza's building, so your scenario about firefighters coming inside and giving shutdown orders doesn't apply this time (but it did a year ago, when they actually had a fire in their basement). This all revolved around a "mythical" backup generator, which had never been brought to the building in the first place. This site had NO generators, so when all the various power lines failed or were turned off, they had nothing to fall back on. DT
Posted by yomismo, 09-21-2004, 02:28 AM
Well, the service is up now. It was down over 10 hours. The response of Alabanza about this problem... "There was an unexpected power outage Monday morning. Due to a massive underground explosion in Baltimore city, our building lost power. The battery backup solution was immediately activated with the loss of power. Our secondary power feed, which comes directly from the city grid, was initialized. After the emergency power cutover, the city ordered the immediate shutdown of to the power grid. This grid spans for 26 city blocks. This disabled our redundant power solution. We immediately went to our third solution and brought in our generator. All NOC and Systems Administrator personnel are on-site working to bring all servers online at this time." What do you think about it ? Bye
Posted by BitError, 09-21-2004, 03:30 AM
Nothing says responsibility like operating a datacenter without a generator. They clearly didn't even have a reliable lead on a generator on a trailer, which scale up to the 3/4 of a megawatt level with proper planning. Then again most hosts that keep leads on truck based generators keep them on lead for a generator failure event, not as their only source of power generation second to battery. If Alabanza decided to run on battery alone, they should have at least had 12 hours of runtime configured which is a lot of cabinets but certainly superior to their two electrical grid methodology. In any case I'm sure they learned another hard lesson, whether they will learn from it or not is to be seen.
Posted by JenniH, 09-21-2004, 03:36 AM
What do we think about it Yomismo? I think that: "To guard against local power failures, Alabanza has two industrial-grade, three-phase Liebert UPS systems. These act as back-up batteries, maintaining uninterrupted power in case of surges or power outages. With these back-up systems in place, we can keep our network up and running indefinitely without relying on external power." ... cannot be explained so they choose to ignore it. I really hope that they end up facing serious litigation, as an example to others tempted to place this sort of stuff on their websites when there is clear dependency upon external power. Frankly, it gives the industry a bad smell.
Posted by Elen47, 09-21-2004, 08:24 AM
Sounds almost like making excuses for them....................bottom line Coolraul is that Alabanza lied about having generators .
Posted by BigBison, 09-21-2004, 08:55 AM
Glad to see you've not lost your sense of humor, black though it is! I have to wonder about 'brought in our generator'. Had they actually been ready with it, wanna bet they'd have run out of fuel?
Posted by manadesigns, 09-21-2004, 12:29 PM
Last year when this happened there was a post in the private forum of Alahost from the Alabanza CEO. I don't have access to those private sections because I'm just a lowly 3rd teir reseller but has there been any official post from the CEO this time? If so can someone post it here. Also the "official" quotes from Alabanza mentioned earlier. Can the source of these quotes be posted (if available). Its just nice to read it first hand especially when relaying information to customers. Also, you think they would release an official press release and post it on their web site in the news section but nothing. They even still have the nonsense about "...running indefinitely without relying on external power." on their site. I could understand if the official word was that their servers were never shutdown and they were simply cut off from the outside world from damage to communications circuits or power was lost to switching units outside of their control but that doesn't appear to be the case.
Posted by shanzek, 09-21-2004, 01:55 PM
I wonder if even having a working, on-site generator would have made any difference, as one of the e-mails (although I'm not sure how accurate it was) said that the fire marshall wanted the power completely off. So even with the generators, the fire department potentially could have told them to shut down...
Posted by davidt, 09-21-2004, 02:14 PM
It's not signed by the CEO, but here's the official message sent out by Alabanza (found in the Express Yourself forum at the Alahosts.com site): Dear Valued Alabanza client, Most of you have asked for an honest response to yesterdays events. Our building sits on a redundant power grid that has three power generators. The grid hasn't failed since 1982, which is something like 30 9's of reliability so we took the lead of companies like Qwest and MFS and trusted this system. Yesterday morning, there was an explosion that caused the grid to fail. Providers like AT&T were left without power. Last year we had an issue with the Power being shut down in our building after a fire. It was a building problem, not a power generation problem. We immediately put a plan in place to insure that we had coverage in the unlikely event that it happened again. There was no delay, no hesitancy, no worry about costs; the entire Executive team approved this 3 Phase plan quickly. We installed a dual power feed from the "redundant grid with three power generators" to our data center with a transfer switch between the two feeds. We also decided to purchase a generator. Yesterday morning we took delivery of that generator we had purchased and worked to connect it. It has not yet been permanently installed, but we are working aggressively to make sure it is. Also, we are working aggressively on the build out of NOC #2: We are in final lease negotiations with Montgomery Park www.montgomerypark.com to build out a second data center, scheduled to be operational by Feb. 2005. We realize that no official statement is going to completely overcome yesterdays events. It was a difficult day. We apologize for the great disruption caused to your business on Monday. We will work with each and every client to find an appropriate remedy. The Alabanza Executive Team
Posted by BigBison, 09-21-2004, 05:06 PM
First, please note that the beginning of this thread has this week's dates, from last year's (to the day) fire. That fire would have resulted in a shutdown of a generator, had one existed. This time around, the fact is that they switched to their 'second power grid' which the FD subsequently turned off. That is the point at which a backup generator, for this current problem, would have taken over with no outage for Alabanza or their customers. They do have, and have had, UPSs to carry them through brief outages, or switchovers between the grids they have access to, but these exist to buy a few minutes - until the generator takes over. I have heard of backup generators failing to take over, running out of fuel, blowing up and who knows what else can happen. But I have never heard of an outfit Alabanza's size, regardless of the quality of power to their datacenter, operating without a backup generator. The lesson learned in last year's East-coast power outage (and one would think Alabanza's last fire) was that redundant grids and backup generators aren't enough, now DCs regularly tout their 'refueling contract' and on-site fuel storage capabilities. Does it really take a year after deciding to get a generator to get one installed? What were they thinking not having one (or two) all along anyway?
Posted by manadesigns, 09-21-2004, 11:44 PM
Thanks for the post davidt. >>Providers like AT&T were left without power.<< If the pipe providers were left without power, did they have back up generators in place to keep the switching up. If not, I guess the generator is a mute point. But I suspect they were left without power and their backup generators kicked in and thus a major communications hub located in the same building (as the Alabanza site describes it) continued to operate without interuption.
Posted by geniusj, 09-22-2004, 03:24 PM
Latest letter -- Last edited by Odd Fact; 09-23-2004 at 05:02 PM.
Posted by shpilkus, 09-22-2004, 05:22 PM
Might I suggest for privacy's sake (since this is a public forum) that you edit the above post to remove Mr. Cunningham's phone number? It's the right thing to do since he did not post this message here, and it was sent to a private group of people (Alabanza-based hosts).
Posted by davidt, 09-22-2004, 05:37 PM
Yes, I agree...remove his telephone number *and* his email address, please. Now...on to the content of the message. Spin, spin, spin. His use of the word "generator" is so loose it's not funny. Unless I'm totally confused, the existing generators he's talking about (not the new emergency one) are all conventional power generating stations -- not emergency power generators. If an emergency generator is not physically located AT your NOC, it simply doesn't count, and he's been "counting" those offsite sources for years, as evidenced by the misleading and incorrect verbage on the Alabanza network description page. Heck, given his definition, my house is probably on four or five "generators," in that we've got a nuclear plant outside of Phoenix, multiple natural gas plants, and hydroelectric. He does confess to having delayed the acquisition of a TRUE generator (in the sense of the word that everyone expects when associated with a NOC), but the lack of a REAL emergency backup generator was simply inexcusable....period. He deserves to lose a LOT of business. DT
Posted by shpilkus, 09-22-2004, 07:52 PM
Playing devil's advocate: The downtime came to around 11-12 hours for us. To me, this is not nearly enough of an issue to warrant discarding a 4.5-year relationship we've had with them. Bad? You bet. Will we lose business? Almost certainly. But I seem to remember reading about much worse situations on this very board at other NOCs. A famous incident involving a host with a C and an I in their name comes to mind - I believe some sites were down for 3 weeks. Nobody can be perfect all the time. I agree that the lack of emergency generators on site is inexcusable, but I do also understand the reasoning, having been to the facility and touring its infrastructure on numerous occasions. It really is a unique building with unique connections to power & data. And should the grid fail again within the next 22 years, we now have a permanent generator. Which won't guard against every contingency either...
Posted by Joshua, 09-22-2004, 08:37 PM
Craig - It's sad to see that you guys were affected by the Alabanza issue - I hosted with you around 3 years ago (before moving to dedicated), and you had the best support and uptime out of any of the hosts I've tried. Hopefully the downtime won't be too hard on you - I know that it's really luck of the draw as to who gets the clients who care a lot about the downtime, and those who are able to brush it off and stick with you because of the service you've provided in the past. -Josh
Posted by davidt, 09-22-2004, 08:45 PM
And yet the reps from Alabanza keep making excuses about that, rather than just coming out and saying "we were wrong....we should have had a backup generator all along." Reminds me a bit about a certain "Commander in Thief" who has *yet* to admit ever making a mistake...but I digress....back to Alabanza... The yearly "fire drills" aren't what are going to drive me away from them as a reseller....it's the outdated versions of the software in their packages (Open Webmail 1.81 for example, and "Smartmail" -- an ancient procmail add-on, instead of a real mail list package), their cluelessness about some email issues (bounce and catchall quirks), their phony network status page (Alabanza.info), their high prices vs. stingy disk and transfer quotas, etc. True, nobody's perfect, but someone just has to be a bit better than all that. DT
Posted by shpilkus, 09-22-2004, 09:09 PM
Isn't that what Tom Cunningham just said??? Whoa boy. Leave your partisan and misguided political leanings outta this. [quote]The yearly "fire drills" aren't what are going to drive me away from them as a reseller....it's the outdated versions of the software in their packages (Open Webmail 1.81 for example, and "Smartmail" -- an ancient procmail add-on, instead of a real mail list package), [/quote[That's be "SmartList", and it's being updated with ListXpress if you pay attention to the mailings. also being addressed, the new Mail Manager will take care of these issues and others [quote] their phony network status page (Alabanza.info),[quote]we're in agreement that this should be corrected Depends on how you view things - the price includes the automation, monitoring and sys-admin level services that you just don't get with a $139 server... Anyway, do what you need to do for your business. Obviously at this time tempers are hot and emotions are high. But also remember that often times bad business decisions are made when tempers are hot and emotions are high.... Good luck to you whatever you decide.
Posted by shpilkus, 09-22-2004, 09:40 PM
Josh: Long time, no speak! Thanks for posting. Yes, right now it's difficult for some (us included) to see the larger picture based on Monday's extraordinary events. So far the comments we've gotten from our clients range from 'thank goodness no one was hurt' to 'thanks for keeping us informed during this time'. Only one client so far has told us they're definitely leaving because of Monday. And he was a pain in the a$$ anyway. 8-) We'll be in Baltimore next week for Alabanza's user conference, where you can be sure Monday will be a hot topic. I know it will make me feel better to see the new generator in person! Take care!
Posted by manadesigns, 09-22-2004, 09:58 PM
It still hasn't been made clear whether or not the AT &T and Qwest communications systems remained up through this whole thing. It appears from Tom's letter that they also relied on the same redundant power grid which would lead me to believe that even if Alabanza's servers stayed up, they wouldn't have been available. Also a quick google search on redundant power grid (not generator) reveals that many NOCs in other parts of the country rely on this method of redundant power. I'm not trying to defend this method but the search was an education for me. Lastly, my ISP with Alabanza claims 99% uptime guarantee. The outage was inconvenient but they still haven't broken any promises. 365*24*.01 = 87 hours. I guess I should consider myself lucky that the outages don't amount to anything close to that.
Posted by davidt, 09-23-2004, 12:34 AM
I see that your company is based in Florida...ground zero of the theft to which I alluded. OK...I'll leave your Governor's little brother alone. It's nice to hear that they're finally going to do something about all those outdated packages on the servers (yes, I know it's "SmartList" -- I was having a premature senior moment). No, I don't get "the mailings" because we no longer have a dedicated server with Alabanza...we used to, but we didn't need quite that much space or bandwidth (we're a nonprofit that hosts some other nonprofits, so we're not a hosting company). We downgraded to being customers of another Alahost, OneOnOneInternet. They've not informed us about any of the planned upgrades you mentioned. The last newsletter that OoO posted in our CP's is dated April 8, 2004, and only addresses things like upgrades to Open SSL, PHP, etc. Good night, DT
Posted by shpilkus, 09-23-2004, 08:04 AM
That is a valid concern, but there are other peering arrangements besides the connections in the basement. Things would still have been hurting that's for sure. This was posted by the MIS manager of Alabanza over at the Alahosts board:So it looks like they're already thinking in that direction.
Posted by BigBison, 09-23-2004, 08:10 AM
There's been some good information posted in another thread, about fire suppression and datacenters, if anyone's interested. It pertains more to the situation a year ago. http://www.webhostingtalk.com/showth...34#post2499934
Posted by shpilkus, 09-23-2004, 08:12 AM
Since you just can't leave it alone without getting in a snide remark, here goes.... It sounds to me like you're disappointed in the outcome of the last election. Imagine how you're going to feel in a few months. Try not to take it too hard - it's not going to be close this time. You'll have to come up with a new title for President Bush, maybe "winner by a landslide". Has a nice ring to it. But hey, don't worry too much, you can always vote for Hillary in 2008. Oh and Jeb is the younger brother, GW is older. So it should be "leave your Governor's big brother alone". Just FYI.
Posted by BigBison, 09-23-2004, 08:28 AM
Please stop it now, both of you, before you get the thread locked. Take it to the lounge and snipe back and forth all you want. Until those threads are locked.
Posted by arfarf, 09-23-2004, 12:40 PM
Jason, Can you please for the sake of privacy REMOVE the phone number and email address of the letter you uploaded. Gosh, it's bad enough that you posted the letter which was sent to Alabanza's clients but then you post this information which if abused could disrupt my ability to contact him. Try to think of how troubled you'd be if YOUR personal information were posted. That would be horrible.
Posted by geniusj, 09-23-2004, 02:32 PM
I'm working on it, however I cannot edit posts that were made >15 minutes ago according to the error message I'm getting. I'm requesting the help of the administrator here instead.
Posted by Soms, 09-24-2004, 05:20 AM
AT&T, MCI, Sprint, Qwest, Level(3), Savvis suffered no downtime at all, they may ask Alabanza to kill the power but they arn't going to go to a provider which powers the goverment offices, schools and even hospitals with data and voice capacity to turn off a backup system, unless it is possible to reroute it another way (across another path of the ring) however BWI to too close to have many other routes able to handle all of the capacity it would need.
Posted by jsw6, 09-24-2004, 09:14 AM
The whole point of a SONET APS protected path is to guarantee the availability of an alternate path for each protected circuit in the event of a failure on the SONET ring impacting a circuit's primary path.
Posted by Soms, 09-24-2004, 03:59 PM
Right, but there isn't much of a ring to build between IAD and BWI for example, look at how close they are together...now something like LGA or BOS to IAD would have a bit more to go on. Edit: Forgot to also mention that just because they may even have a full capacity redundent ring in the area, still won't excuse the fact of a hospital which may just get a bonded T1 losing VoIP capability, or the wifi network it powers so that police cars can run checks while on the road without calling in.
Posted by zoza69, 09-25-2004, 05:43 PM
I did read the response of Thomas Cunnigham, CEO of Alabanza about Alabanza'z 9 hours outage with interest. Have you heard of a device called "Diesel Generator" Mr. Cunnigham? You are so cheap that you deprive your users from the basics. Your excuses are lame.
Posted by sirius, 09-25-2004, 07:57 PM
At some point, you're going to grow up and recognise that your little tyrades (in most every web related forum so far) are old and unprofessional. If you don't like the host, move on, without spamming the boards. Best of luck. Sirius
Posted by arfarf, 09-26-2004, 09:58 AM
Gee, you make some really good points (I say that sarcastically). You obviously haven't heard of redundant power grids. And for good reason this was the option he chose. While it failed and a generator would have saved the day clients have received a multitude of opologies for this admitted mistake. He never asked to be excused for this decision, he took responsibility. This term appears to be foreign to your vocabulary as you're mixing responsibility up with excuses. If you're a client of Alabanza, I don't fault your frustration. If you're not, you're a person with a lot of time on their hands to bother with this forum. If that's the case, please go out and take some time to find a productive hobby, you'll be a lot happier. And I do want you to be happy.