CI Host - Chicago Down [MERGED] - Knowledgebase

Portal Home > Knowledgebase > Articles Database > CI Host - Chicago Down [MERGED]

CI Host - Chicago Down [MERGED]

Posted by dotRoot, 10-02-2007, 07:46 PM
CI Host's Chicago Datacenter (CDC-3) is down. Their network has been down since early, early this morning. See https://hostingsupport.com/networkstatus/ for updates. They keep changing the ETA times. If its just a simple router upstream on their SONET Ring, there is still no excuse for it to be down this long. Also, where are their redundant connections?
Posted by FastServ, 10-02-2007, 09:02 PM
I recommend you do a bit of searching on these boards for CIhost, and any other provider you wish to partner with in the future.
Posted by dotRoot, 10-02-2007, 10:32 PM
I already know. I just keep a colo machine there. Have for years. Its no big deal for me, but for others I'm sure its hell.
Posted by moosh28, 10-02-2007, 11:42 PM
have you called them for status?
Posted by dotRoot, 10-03-2007, 12:41 AM
Yes. They say its a router. I find it interesting that the entire datacenter went down just after the first. And that its taken this long to fix a router and that they'd have such a bad single point of failure in the first place. I am assuming thay CI Host didn't pay the bill with Equinix. I'm waiting for my colocated machine to come up so that I can pull my own MRTGs, get credit as I find a replacement. I know that you get what you pay for, but its just a server for a couple of personal projects. I never thought that this would happen. Ah well. Live and learn I guess.
Posted by moosh28, 10-03-2007, 08:09 AM
CI isnt with Equinix. They actually have their own facility in the city. Just wondering, did they ever come back up?
Posted by Patrick, 10-03-2007, 08:46 AM
According to their network status page, it's still ongoing... the new ETA is 08:30 CST.
Posted by dotRoot, 10-03-2007, 10:42 AM
Absolutely correct, however, their network transits through Equinix. Just came back up. Time to rsync to somewhere else!
Posted by avizzle, 10-03-2007, 04:23 PM
I heard they had a break in at the datacenter, can anyone confirm?
Posted by Nkrapf, 10-04-2007, 10:41 PM
Yes they had a break in No one from CIHOST will return our calls or emails. I had to call chicago police department to confirm there was a break in. Story is wishy washy but they did file a police report and had photos and finger prints taken. we lost $15k worth of equipment
Posted by dotRoot, 10-05-2007, 01:45 AM
Last time this happened, I read that they didn't reimburse anyone either. Seems the security there is outstanding, isn't it?
Posted by RyanD, 10-05-2007, 08:56 AM
can you provide proof of that?
Posted by pzil0cyb3, 10-07-2007, 11:46 PM
I did a web search and found this thread. Awesome. I noticed my server (appx $4k) , colocated at the Chicago CI Host facility had been offline since early last week. Likewise, I was getting various excuses.. "There was a power outage".. "There is a router problem".. and ETAs that were all drastically incorrect. Finally I received a call in the afternoon on Thursday I believe, from Matthew Rappaport at CI Host. He told me there was a breakin and theft, there was a known weakness in the wall that the intruders used to gain access. An employee was alerted and when he showed up he was tasered. He believes it was an inside job but he doesn't know for sure. I was also given a detective's name and number as well as a case number -- Detective James Swick, 312-744-8263 Case #HN-623601 The detective basically told me that this CI Host facility is horribly insecure, he also thinks it's an inside job (all of the cameras/recording equipment were taken) Matthew at CI Host offered to setup a temporary server for me, not even remoetly in comparison to the computing power of the server I lost. I was a customer for about 4 months. I am consulting a lawyer and am completely prepared to take legal action against CI Host as they are obviously in gross negligence which is a violation of the contract.
Posted by unixbox, 10-09-2007, 03:42 AM
Can you let us know what your lawyer says? This is not the first time this has happened. It kind of feels like a theme...something is not right here. The detective said it seamed like an inside job.
Posted by RyanD, 10-09-2007, 10:34 AM
wow this is insane....
Posted by marcus9000, 10-09-2007, 10:46 AM
I've been a customer out there for a little over a year. When did a break-in happen before?
Posted by Nkrapf, 10-09-2007, 11:25 AM
For those of you that have lost equipment and cihost has not contacted you here is the info: Police Report Number for this incident is HN 623601. The detective assigned to this matter is Detective James Swick, Area Three, Property Crimes, of the Chicago Police Department. Detective Swick may be reached at (312) 744-8263. you may also want to contact cihosts legal department : James M. Eckels jamese@cihost.com CIHOST provided us a inventory sheet to fill out and fax in to them for the police department. Nkrapf
Posted by HNLV, 10-10-2007, 12:27 PM
Isn't the equipment in locked cabinets?
Posted by spaethco, 10-10-2007, 09:05 PM
If you're going to break down a wall to gain access, cabinet locks aren't going to slow you down.
Posted by drumman24, 10-10-2007, 10:18 PM
Just terrible.
Posted by RyanD, 10-10-2007, 11:46 PM
couldn't have been much of a wall. Anyone here been to the Chicago CI Host facility ? I'd be interested in seeing some pictures.
Posted by johringer, 10-11-2007, 08:11 AM
I was there briefly a year or so ago to pickup a client's hardware, and needless to say, I wasn't impressed. Though I didn't snap any pics, I do remember a lot of loopholes in security, mainly in their sign in and access measures to the floor.
Posted by Ryan - Limestone, 10-11-2007, 02:20 PM
What a horrible situation!!! Has anyone gotten any updates, compensation, anything from cihost at all?
Posted by jimijon, 10-11-2007, 09:23 PM
I had ALL of my servers last year. It was a major disaster and cost me a lot of customers. And, I had the majority of my data backed up between servers and not off-site. Now, I had two of my three servers stolen. Luckily the third server was left and am able to limp along as I try to weather this substantial financial hit again. It has been a disaster. I can also say it has to be an inside job. It as the exact same M.O. as last time just a different, non-reinforced wall. Like to know of who else got ripped off. -j-
Posted by RyanD, 10-12-2007, 12:23 AM
it's amazing that this isn't getting more press as this is the *SECOND* break-in they've had.
Posted by fastdeploy, 10-12-2007, 02:11 AM
Wait a second. You're saying you had all of your servers stolen out of CI Host last year and yet you still put additional servers in there with them? Please tell me it's not so.
Posted by RWH, 10-12-2007, 03:48 AM
Did they have 24 hour coverage at this place?
Posted by RyanD, 10-12-2007, 06:24 AM
apparently not, looks like "an employee was alerted and when he arrived he was tased" seems like no on-site staff
Posted by johringer, 10-12-2007, 08:09 AM
If you're going to carry a gun into the datacenter for a hoax like this, and are not a cop, why a tazer? I mean it seems out of the ordinary.....
Posted by David, 10-12-2007, 12:36 PM
Hm, what concerns me more is all of the 0 post wonders posting about it.. but still nothing from anyone else. Someone give the detective a call. Still, if it's the truth: Damn, that's got to suck. I hadn't really considered security from that 'perspective' before. The last instance that set me off was when Katrina ripped through New Orleans & there were a few DCs there with m16s on hand to prevent problems. Re: Jdubz, Inside job, like someone mentioned before. Can't kill ex-coworkers. It's just not kosher, this isn't the postal industry! At any rate, sorry to hear about your guys' loss. I can't wait til the boxes flood the market though.. cheap unformatted ones here we come! Last edited by David; 10-12-2007 at 12:39 PM.
Posted by bummer6666, 10-12-2007, 01:04 PM
LoL it would be interesting to see something like that happen in our building. Those guys would probably get shot before they could reach the exit door. I remember there was a customer who forgot to show the property pass to the building security when leaving the building with his computer and got tackled&handcuffed outside by plain cloths cop
Posted by johringer, 10-12-2007, 04:58 PM
But that's also CBOT....I would expect that type of security there, just like 350.
Posted by unixbox, 10-15-2007, 01:19 AM
After talking with our lawyers we have found a clause in the contract that will allow us to have CI-Host pay for the equipment Hardware/Software and downtime. Here is what you signed with the contract: "INDEMNIFICATIONS AND LIABILITY. a. Customer shall defend, hold harmless and indemnify C I Host and its affiliates, officers, partners, agents and employees from all claims, demands, actions, damages, judgments, expenses, costs (including reasonable attorneys fees), and liabilities arising out of Customers access to or use of the premises or the space. b. C I Host assumes no liability for damage to, or loss of, the equipment or Customer property resulting from any cause whatsoever, except as a result of C I Host's gross negligence or willful misconduct. c. C I Host shall not be liable to Customer for any indirect, incidental, consequential, exemplary, reliance or special damages, including damages for lost profits, regardless of the form of action, whether in contract, indemnity, warranty, strict liability, or tort, including negligence of any kind with regard to any conduct under this Agreement." The underlined is what will save us. Seeing that this is the second time this year this same incident happened they are negligent. If you would like to join our class action please email me. info at unixbox dot com We had a lease with Dell for about 15k and can not get anyone at CI-Host to call us back to help us with our stolen equipment. It is a shame because the colo is in a great spot with great pricing. It is just shady that the Management allowed this to happen twice.
Posted by FastServ, 10-15-2007, 07:52 AM
It's shady for it to happen even once... I'd call hosting there at all gross negligence after that.
Posted by ross22a, 10-15-2007, 01:32 PM
Unixbox: Do you have a different email address I can use? I'm trying to contact you and that address gives a user unknown bounce.
Posted by unixbox, 10-17-2007, 02:16 AM
My apologies not sure what I was thinking. info at unixbox dot ws not com brain malfunction.
Posted by MinusOneBit, 10-17-2007, 02:19 AM
This thread might be one of the better examples of the "you get only what you pay for and nothing more" doctrine in action when it comes to hosting. That said, lets just say that I am not too surprised that this happened and I hope you get your stuff back and/or reimbursed. Last edited by MinusOneBit; 10-17-2007 at 02:29 AM.
Posted by ub3r, 10-17-2007, 04:46 AM
I just called the detective, and there was in fact a break in. I'm not sure how valid the other claims are, but there was a break in. How someone could get up to the 3rd floor that ci host is on, and break down a wall. Was this via a window? Does CI Host own the whole floor, or just a suite on that floor? I googled the address, and found that it's a fairly public building, even housing a restaurant. I guess it's possible that it's not monitored by security guards 24/7, but why would cihost even be there if there wasn't telco-house grade amenities? When did they move in there?
Posted by AllyJ, 10-18-2007, 03:02 PM
Apparently this has happened 3 times before. I added copies of the reports if this will help you all. Seems like extreme gross negligence to me. Sorry I can't link directly so you will have to add the tt yourself. hxxp://img81.imageshack.us/img81/4881/report1part1vx1.jpg hxxp://img81.imageshack.us/img81/8002/report1part2kw8.jpg hxxp://img222.imageshack.us/img222/4590/report2part1bm4.jpg hxxp://img86.imageshack.us/img86/8008/report2part2jo8.jpg hxxp://img86.imageshack.us/img86/6286/report3part1zx1.jpg hxxp://img81.imageshack.us/img81/3007/report3part2wo8.jpg Considering the number of times they let this happen I wouldn't be suprised if they were doing it for insurance. I bet whoever their insurance is has to be on their asses. Last edited by AllyJ; 10-18-2007 at 03:08 PM. Reason: Misspelled
Posted by 01globalnet, 10-18-2007, 03:33 PM
I remember CI host has been stolen again in the past!! wow....
Posted by TotalChoice - Bill, 10-18-2007, 05:18 PM
CI Host, Is my server down? Yep sure is, its down at the pawn shop
Posted by Ryan - Limestone, 10-18-2007, 05:21 PM
Hahaha that quote made my day!
Posted by pzil0cyb3, 10-18-2007, 05:26 PM
I just emailed you unixbox and am certainly interested in joining the class action.. I don't have anything new to report other than what I see on here..
Posted by bithost(NET), 10-18-2007, 05:37 PM
Stolen equipment?! Are you serious?!?! /me goes over to Network Outages section. Bailey
Posted by KarlZimmer, 10-18-2007, 05:56 PM
I pretty much agree with what was posted previously. There have been at least TWO other incidents of equipment being stolen from CI Host and simply seeing the facility should be enough to tell you that there are security issues. I would honestly think that those suing may have an issue for being negligent themselves, entrusting their equipment to CI Host in the first place. There is a clear history of lax security and of items being stolen from that location. I honestly don't see why anyone would colocate any equipment at a CI Host facility, especially if that equipment isn't fully insured against theft.
Posted by AllyJ, 10-18-2007, 06:20 PM
If anyone still has equipment there I wouldn't worry. They are willing to pay someone 6.25 an hour if they can supply their own gun to perform building security and the may even bump your pay up to $7 an hour if highly qualified. I am sure this will attract great job applicants. My horrible linkage doesn't work but you can go to career builder and check out Keyword "CI Host" and Location "Chicago" to find their job listing. Seriously though if this is the fourth time this has happened they either need to move shop to a more secure facility or get some highly qualified armed guards. It sounds like they don't even take this seriously. Who really owns a handgun and wants a full time job making minimum wage not to mention the fact that Chicago has no concealed carry licenses unless you are a cop or in the armed forces.
Posted by bithost(NET), 10-18-2007, 06:23 PM
Remember, anyone can get a copy of a police report thanks to Open Records laws so I would recommend that anyone affected by this call the police investigator and find out how to submit an Open Records request. Then your lawyer is sure to have as complete a set of documents as possible. Also ask the investigator for case numbers of all previous theft reports at that facility. Then submit Open Records requests for copies of those cases as well. Wonder how long it is going to be before Chris Faulkner's Mommy the Lawyer shows up and starts dishing out the threats. Bailey
Posted by bithost(NET), 10-18-2007, 06:30 PM
ROFL, my gun alone is worth a helluva lot more than $6.25/hour. I don't know what they're smoking, but carrying a gun and performing a job as armed personnel in any setting is a very highly-demanding, high-liability and high-risk job (simply by nature of that noisy thing on your hip). It's not the kind of thing that a person takes lightly, no matter how gravy-train it is. You are still a one-(wo)man deadly force, and that is a sobering concept. $6.25/hour and BYOG. That's like asking my cat to work as a french fry cook at McDonald's and paying him 25¢/hour. Bailey
Posted by AllyJ, 10-18-2007, 07:07 PM
I guess I could have included those all here for everyone. But here they are now. HL554438 HL629835 HM613627
Posted by AllyJ, 10-18-2007, 07:11 PM
Oh and FYI so its easy to find the previous police reports are copied on post 39 on the 3rd page and the police report numbers are on page 4 in post 47.
Posted by anon-e-mouse, 10-19-2007, 04:01 AM
Those not affected by the outage, please don't post in here.
Posted by dotRoot, 10-19-2007, 08:17 AM
I'm sure it won't be long now. He's got to do damage control and if it takes sueing someone for defamatory remarks before they sue him for negligence, then so be it. It'll just be yet another month for CI Host. === Before the misinformation gets out of hand, I believe the break-in for Chicago (CDC-03) has only happened one other time. And it does sound similar. They backed a truck into the wall there and took the stuff. Sounds like they did the same thing again, doesn't it?
Posted by StuartLittle, 10-19-2007, 01:01 PM
Man that is crazy I remember CI Host in Chicago being robbed before. Nice work on the police reports it confirms I will never put my stuff there. I wonder how many more times they can have a robbery there. Does their contract have fine print about your equipment most likely winding up stolen as theft seems to be the norm there.
Posted by AllyJ, 10-19-2007, 02:03 PM
Well we now have 4 confirmed police reports to show this has happened at least 4 times. And being that they are located on the 3rd floor in their building I would assume they went through more planning than backing a truck through a wall to get in. If thats how it happened I wouldn't be so upset but more in marvel at how someone got a truck up 3 stories. The building is in a crappy area but when I placed my stuff there it seemed like it would be semi difficult to get into and I was assured it was secure with 24/7 monitoring. From the look of the building you would never expect it to be there. Its kind of like that old SNL parody commercial where you protect you luxury car by making it appear as it were a beater on the outside. Not to mention I believe I recall a few cameras there and they should have some sort of an alarm system. I wonder if they turned the footage of the break in over to the police or if they kept it to themselves to hide their identities. Last edited by AllyJ; 10-19-2007 at 02:09 PM. Reason: Spelling
Posted by RyanD, 10-19-2007, 07:29 PM
It was noted that the security recording equipment was also stolen which is why it was believed to be an "inside" job.
Posted by pzil0cyb3, 10-21-2007, 03:20 PM
Everything in the last two posts coincides with what I think and what the detective told me (that all of the surveillance equipment was taken).. Assuming this was really just a group of ocean's-11 type robbers (hehe) what attracted them to this nondescript building in a shady part of chicago? Likewise they assured me the place was secure and they had security measures in place.. And I can't imagine how much time it would have taken for them to carry all that equipment down the stairs.. blah
Posted by Nkrapf, 10-24-2007, 03:22 PM
If there is already a lawsuit for this please contact us so we can join it. We lost 3 high end dell servers and a lot of customers. Not only did this happen but they destroyed 2 of our servers when they shut down the NY data center... so in 30 days they took out 5 of our servers. They also charged our credit cards for renewals EVEN THOUGH ITS CLEARLY MARKED on the contract not to auto renew and to contact us for renewal first and would not give us a imediate refund "James will get back to you in a week or so..." James will not return any calls. Everyone eles gives us the run around on everything. Nick Krapf 630-878-0602 www.bloodservers.com nick@bloodservers.com Instant mesanger names (not to be used for email): MSN: nick_krapf@hotmail.com (not email) Yahoo: krapf_net AIM: krapfnet ICQ: 18065448 gmail / Jaber: bloodservers@gmail.com
Posted by unixbox, 10-25-2007, 06:51 PM
We are going to have a meeting 11/01 and I would like to invite those that were effected by this outage. please email me admin at unixbox dot ws
Posted by Nkrapf, 10-25-2007, 09:10 PM
i sent an email to participate in the meeting. Nick
Posted by Devil Inside, 10-25-2007, 11:48 PM
Yep. I just got back from the chicago DC to get what we thought was a downed server. Only to FINALLY be told about the robbery and that the server was stolen.
Posted by Devil Inside, 10-25-2007, 11:53 PM
Ditto on the lawsuit. Also, what about the building security cameras? I literally just got home from the DC and JUST found out about the robbery while there. There are security cameras outside the elevators and more in the hallway run by adjacent offices...not CI host.
Posted by AllyJ, 10-26-2007, 05:11 PM
Did you see anything to indicate a break in? I am curious to see if a wall was really taken out. Did you see any cameras when you went to grab your stuff? Everything about this and the way they are handling it is just too fishy.
Posted by AllyJ, 10-26-2007, 05:26 PM
Sorry I replied to your other post as well but your mentioning here of other cameras in the building answers one of the questions I asked there. Do you know if they were able to grab footage from the other cameras? For stuff to disappear that quick it seems they would have to of prepped for it so assuming it was an inside job it should not be too hard to identify the people going in and out at the time everything was taken.
Posted by Devil Inside, 10-26-2007, 10:20 PM
I hadn't noticed much out of the ordinary. Mainly, while in the entry way of the CI Host offices, I saw that the door to the main hallway was crooked on the hinges. You could easily see through the enlarged crack where the two doors meet. There's only one good quality lock on it - but a few good hits would take it off the hinges I'd imagine. Other locks were either incomplete, or cheap. As for it being an inside job. Well that much seems obvious. Or CI Host security practice just blows. First, one would need to be buzzed in at the door leading to the street. Then the only option is stairs or the elevator. There's a camera facing the elevator doors on the 3rd floor. But does NOT point down the hallway where the CI Host offices are located. My brothers said they saw cameras outside other offices, in addition to CI Host's office. Whether they had view of the hallways is unknown to me as I hadn't noticed them. If a power tool of any kind was used, theres no way someone locked in the main office could hear it. They would have to be in the main entry way of CI Host or damn close to it. With the amount of fans they have blowing around the server room to keep things cool....it's very noisy in there. It was difficult to talk. There's reason to believe that our server was stolen after the break in, and after initial police investigation on scene. Though I'll not give out our reasons for thinking that, publicly.
Posted by unixbox, 10-31-2007, 12:45 AM
Please come to the following location or Call the following # at 9am Thursday Nov 1st to find out about the lawsuit against CiHost. Kalcheim Haber, LLP 134 North LaSalle Street Suite 2100 Chicago, Illinois 60602 (312) 236-9445 Fax (312) 236-9446
Posted by unixbox, 11-01-2007, 12:28 PM
Guys, Everyone meet everyone. We have 3 others that I don't have email addresses for that I will be contacting via phone. Unfortunately, the thing we have in common is we entered into a deal with CiHost (a.k.a. Securing our Business sucks (SOBs).) After meeting with a lawyer today we now know we have a case against the SOBs. The question is how to keep the expense low. The best way is to get everyone aboard and have the lawyer represent us all. The first stage will be to have our lawyer draft a letter of intent. This will let the SOBs know that we are now not individuals they can just blow off any longer. This will also allow the SOBs to do the right thing and settle our loses. What do we need now: Names Addresses Contracts Copies of loses Reciepts of loses Settlement amount Sorry we have to meet like this but hopefully we will prevail better than if we were on our own.
Posted by WII-Aaron, 11-01-2007, 02:34 PM
Why don't you guys just turn this over to your insurance companies? Let the insurance companies go after CI Host. That's what you pay those hefty premiums for.
Posted by unixbox, 11-01-2007, 02:56 PM
Insurance will not cover off site equipment unless you add if on your policy. And when collocating a box in "secure" environment you would think it would not be needed. But then again no one told me this place gets robbed every year. In my opinion this is an inside job that cihost uses to sell our boxes on the black market.
Posted by WII-Aaron, 11-01-2007, 03:24 PM
I'm not sure how to respond to that. I don't know of any legitimate business that doesn't insure off site equipment no matter where it is.... of course I guess I don't know of any legitimate business that would colo with CI Host. Good luck to you. I hope you can recover something from them. Aaron
Posted by unixbox, 11-01-2007, 03:38 PM
So to insure my servers off premise it would of cost an additional $129p/m on my already expensive policy. With the depreciation of server equipment you would of had to have your servers stolen with in the first 7 months of the plan to make since. One would not think this would never happen. Plus one would think the collocation would have insurance or morals to cover a break in. If I had a 100k router with 1Million dollars worth of telco equipment then yea $129 is worth it. But for servers that are less than 20k?
Posted by Dougy, 11-01-2007, 03:48 PM
I'm a Haber too!
Posted by Ryan F, 11-01-2007, 04:06 PM
Are you sure about that? I've never seen a policy that didn't cover all "business-owned-equipment" no matter where it was. Even the most basic, $500 per-year "BOP" (business owner's policy) will cover your servers.
Posted by Nkrapf, 11-01-2007, 04:11 PM
A reporter just contacted me and would like to run a story on this. he will be contacting me latter to get my story. I was hoping CIhost would make good on this before it esclated. In my opinion, once one news company reports on it i would think many more would follow.
Posted by Nkrapf, 11-01-2007, 04:29 PM
incase your looking for a reliable host, These guys have been GREAT in assisting me. [FONT='Arial','sans-serif']Jon ColoCrossing.com jbiloh@colocrossing.com Office: +1 (888) 275-5754[/FONT] [FONT='Arial','sans-serif'][/FONT] [FONT='Arial','sans-serif'][/FONT] [FONT='Arial','sans-serif']Thank you Jon... hope someone here can benifit from the great support provided by ColoCrossing[/FONT]
Posted by rasputin, 11-01-2007, 05:31 PM
Sorry, but you should have insurance on your servers regardless of where they are located if they are that important to you. There is no such thing as too much insurance. I would also be willing to bet that CiHost has a stipulation in their terms that require you to have your own insurance. I know we always have... Never had a problem in over 10 years of business, and yes we have our own hefty liability insurance, but we ask our customers to carry their own insurance as well.. You can never, ever have enough... That said, our liability policy carries a required professional insurance to indemnify the liability carrier so we can be insured.. That professional insurance next to payroll and our bw is our single largest expense.. I'd be asking for two things first. 1. Copy of the police report 2. Proof of CiHost's Professional and Liability Insurances. Send the police report and open a claim to CiHost's carrier, and to your own, assuming one exists. Ultimately, you'll get farther off taking this up with your insurance carrier first... If you did not or do not have one, then I'd say the only recourse is to rely on CiHost's carriers, assuming too, that they exist. Good luck!
Posted by drgigenet, 11-02-2007, 06:33 PM
Most co-location agreements I've seen require the customer to carry at least 1MM coverage for their equipment.
Posted by pzil0cyb3, 11-03-2007, 12:08 AM
CI Host's colocation agreement doesn't mention anything about requiring insurance
Posted by plumsauce, 11-03-2007, 07:54 AM
Even if you are required to carry insurance, it does not excuse the other party from liability for negligence.
Posted by MartHUK, 11-03-2007, 10:37 AM
The news is spreading - it's just hit The Reg.
Posted by tinkertim, 11-03-2007, 12:32 PM
About to hit the front page of Slashdot, its red in the fire hose now, linking here and to the article on the reg.
Posted by peruviantalk, 11-03-2007, 08:12 PM
I am sorry for the people who has lost their biz/servers over this.
Posted by tinkertim, 11-03-2007, 11:54 PM
I sincerely doubt that anyone would repeatedly taze and beat their own employee for the sake of grabbing 20 servers. While their problems with disclosure are now obvious, I'm quite certain that nothing _that_ sinister is going on.
Posted by unixbox, 11-04-2007, 12:35 AM
How funny is this? "C I Host is a large web target. Due to this, we have a sophisticated in-house security staff that keeps C I Host secure from The Wild-Wild-West Internet. We are able to do the same for you: allow us to put our talent to work for you."
Posted by jaredbeckham, 11-04-2007, 12:40 AM
I agree with you tinkertim, but I think the employees at the Chicago data center may very well be taking advantage of the situation and the absence of the parent company.. I'd take a thump on the head and get tazed for a cut of $50k+.. It's just a hunch. I worked at a place once before, long ago, where a couple of employees pulled a stunt similar to this, but we weren't a hosting company or in the tech field even.....
Posted by tsj5j, 11-04-2007, 08:16 AM
Is 20 servers even a confirmed figure? I don't see how high the chance can be that so many have suffered loses in this thread with only 20 servers lost. I would expect many who lost servers not to know of WHT.
Posted by pzil0cyb3, 11-04-2007, 02:38 PM
I haven't seen the latest police report so I don't know how much was actually taken, I've only seen the older reports. I wasn't familiar with WHT.. I kept doing web searches while trying to find ANY info about it, then one day I get a hit, so I signed up and I've been coming here ever since.
Posted by cmsoko, 11-04-2007, 02:45 PM
Let's just say between overhearing cihost employees discussion the number of servers stolen and talking to another target of the theft this last break-in...It's dozens upon dozens of servers stolen and you can do the math on the costs associated. the latest theft cost one customer alone $20k!! Unfortunately I'm a sucker still paying at cihost. I have steel cable locking my servers to the racks and every "power outage" or "network outage" I have to run downtown and see if my server is still there. I'm joining the lawsuit I read about higher up. Hopefully that'll gain some momentum and become as big as it should be. I'm in chicago and want to joing the support group for prior cihost suckers!
Posted by onejdev, 11-04-2007, 04:12 PM
I sincerely wish that I had found this much sooner than I did. My co-sysadmin forwarded me the story from slashdot today and needless to say I am now making every effort within my means to expedite the moving of my equipment from one DC to another. Prior to today, I was in the process of switching data centers to the Texas data center due to constant outages at the Chicago data center. I see now that the early oct outage was just not a problem for me, and I am grateful that my equipment was not taken from the premises. Disturbingly I just encountered that the person that was dealing with my transfer to a new host, their email is no longer a valid email address at CI Host. I have again contacted sales to see what sort of service I can get, but wouldn't be surprised if it took half a week to receive a response out of someone. Last person I spoke to at the TX data center (during early oct outage) didn't really have good things to say about the Chicago data center. In fact, when I commented that CI Chicago "must be the bastard child of the CI Family" he chuckled and said "no comment" In fact, this same tech was the person who mentioned the shutting down of the Newark data center and how they created temp servers for customers while servers were shipped out. The very next morning I was on the phone having my server moved to the TX Data Center after I exchanged emails with a CI rep. and explained the situation. Bottom line of this, I am back to square one with moving between data centers. It would appear that I am now going to have to re-explain the situation to whomever contacts me back, though I am sure that I need to say no more than "I don't feel that CI Host can guarantee the security of my hardware within the walls of the Chicago Data Center and therefore I demand that CI moves me to a more secure location"
Posted by cmsoko, 11-04-2007, 04:25 PM
feel free to contact me if you need some assistance. I have found a few helpful people at cihost. That said most are either non-responsive or leave for obvious reasons. that all said there is now 24/7 staff onsite. weekdays for certain and possibly weekends also. a security guard is back on nights as well. Not sure if that helps but usually they wait a while before breaking in after a recent one. You should have a few months of security while you search for a new host.
Posted by onejdev, 11-04-2007, 04:27 PM
for some reason I can't send you a PM or view your profile to send you a mail message, but yes, I would love to be put in touch with someone who will at least answer my emails and help me get this expedited. please email me at jdevaney at gmail dot com. Thanks!
Posted by Sirozed, 11-04-2007, 08:44 PM
I have a server there and had one for almost 3 years. What I can say is that I have been there when it first opened. I recently canceled my account as well. They had none of the security CI host claimed at the start. No working cameras, no security guards and the doors were wide open. The data center was a dump, no air filtration at first and they had to bring in portable AC units to keep it cool for a while. There was crap laying around everywhere. With open racks it would be easy to turn off or unplug anything while in there. Also once you are in there the person who works there goes outside the data center into the office down the hall. Any one could let you in the building and you could easily get in the Cihost suite with the right story. My ID was never checked to get into the data center until the last time I went about 5 months ago. They also have a wet sprinkler system in the data center, so if there were a fire say goodbye to the equipment. None of the colo servers are bolted in unless you pay for a secure rack. I have pictures inside the data center but until I remove my equipment I wont post them.
Posted by johan_hammy, 11-05-2007, 12:57 PM
Only if you pay extra to have it in a locked cabinet. When I was there, most servers were just in racks, not cabinets.
Posted by Speedy08, 11-05-2007, 01:09 PM
I'm not surprised. My company used to be in CI Host and it was terrible. We had a private cage and STILL had laptops and a server stolen. We moved down the street to Anet. Little more expensive, but our gear stays put.
Posted by johan_hammy, 11-05-2007, 01:12 PM
Illinois's minimum wage is higher than $7/hour, IIRC. Who uses handguns anyway? A shotgun is much more of a deterrent than any handgun, visually.
Posted by AllyJ, 11-05-2007, 01:52 PM
Hopefully the attention this is getting will help in getting CI Host to pony up for the responsibility as everywhere seems to mention it being an inside job. In addition to the register it also made digg albeit lowkey but I also attached slashdot which was mentioned earlier. hxxp://digg.com/security/Data_Center_Robbed_for_the_Fourth_Time_in_Two_Years hxxp://it.slashdot.org/it/07/11/03/2054208.shtml Its amazing they haven't even updated there webpage of all their false security claims at CDC-03 Last edited by AllyJ; 11-05-2007 at 01:53 PM. Reason: addition
Posted by radioinsomnia, 11-05-2007, 03:35 PM
Yes, Illinois minimum wage is $7.50, with 25-cent increases slated for each of the next three years. I can't link to the US Department of Labor page with minimum wage listings for each state, but it's the first Google hit for "illinois minimum wage". I'm going to guess that CI Host doesn't have an HR department worth its salt, either.
Posted by pmabraham, 11-05-2007, 05:37 PM
Greetings: See http://www.dslreports.com/shownews/C...nd-Again-89142 Thank you.
Posted by johan_hammy, 11-05-2007, 06:10 PM
I thought it was $7.50, but I didn't wnt to say for certain if I was wrong. I saw a job posting on Monster for $12.50 - $15.00/hour.
Posted by AllyJ, 11-05-2007, 06:30 PM
The job has since been pulled off of career builder and I am guessing it is not because they found a qualified candidate but rather the fact hiring for that job at that rate was probably illegal. The job posting that was on careerbuilder for a couple of weeks offered an annual salary of 13,000 to 15,000 for the full time job which when you take into consideration a 2080 hour work year it is less than minimum wage. If they want to pay employees so little they could probably start a convicted felon work rehabilitation program and get state and federal refunds for hiring the felons thus having the end pay rate be below minimum wage. My guess is the felons would even be more trustworthy. Last edited by AllyJ; 11-05-2007 at 06:32 PM. Reason: Addition
Posted by KarlZimmer, 11-05-2007, 07:35 PM
OK, I'm rewriting this whole post in a matter so CIHost can't really sue me about it.... If you saw that any facility was robbed 4 times in two years, a facility which held customer equipment, wouldn't you begin thinking it could be an inside job, in that it was actually sponsored by the company in question? With annual contracts, the customer would be stuck in the contract, even if they didn't have equipment there. If the customer was using resources to the point of not being profitable, it would be better to have that customer gone. Then the equipment being stolen could be sold for a profit and I am sure there could be insurance claims made for lost business, and at least repairs, etc. where some profit could be made as well. I'm not saying this is what happened here, but I can't see how you couldn't think something like that might be happening. Last edited by KarlZimmer; 11-05-2007 at 07:40 PM.
Posted by Kevin2001, 11-05-2007, 08:03 PM
If and when you can, please post the pictures. Could be very educational.
Posted by chopsmidi, 11-05-2007, 08:31 PM
It's possible, but I'm going to have to invoke Hanlon's Razor: never attribute to malice that which can be adequately explained by stupidity. I was a customer of CIHost for several years, and moved (to your company actually) after some of my servers were stolen from me by another CI Host colo customer that just walked out with them (they had him on camera taking them). Their security is thoroughly crap. If someone walks out of the front door, you can simply go in and go up the elevator to the third floor. Anyone who has been to CIHost Chicago can determine that breaking in the data center is only slightly more complicated than throwing a brick through the window. It's painfully insecure.
Posted by KarlZimmer, 11-05-2007, 08:40 PM
The main reason that made me bring it up is their previous shady business practices, suing those who say bad things about them, even if they're giving an honest review, their licensing lawsuit with Microsoft, etc.
Posted by tsj5j, 11-05-2007, 09:04 PM
I would believe that if they were in violation of the contract (due to negligence), the customer can sue them for losses and break the annual contract.
Posted by KarlZimmer, 11-05-2007, 09:30 PM
Though a vast majority of clients are not going to have the time and/or resources to deal with the issue through the proper legal channels.
Posted by ChrissyF, 11-06-2007, 01:38 AM
To all those affected I sincerely apologize. I am a recently ex CI host employee and can give my word that Mr. Christopher Faulkner was responsible for the thefts. I have worked closely with him and he was not as quiet as he should have been considering his willingness to tell people about the thefts then fire them. From what I was originally told of the operation Chris had setup was that they would have certain servers marked, the majority of which belonged to us and a few customer servers that were either backed up or had no information on them. It wasnt until this last theft that I realized majority of the servers did not belong to us and none of the customers actually knew about this. I do not know who he orchestrated this with but I do know it happened more than the 4 times listed. I believe the total count was 6 times that this was pulled off and I believe police reports were filed all 6 times to prove it. I assure you the Chicago datacenter employees were not to blame and that the first few times he scheduled were with no one there and as he got greedy he thought he would need employees there a couple of times to avoid the accusations of this being a scam. I was always assured no one would be hurt but he wanted the thefts to appear real so the manager on duty or on call was not aware what would happen that night. We rarely staffed 24/7 but our phones forward to a cell phone to make you think we did. I remember how he gloated on how a fake pellet gun was used on Jason and that he thought it was real and the police report was perfect. But due to the fact he heard that Jason put up a fight he decided that a taser should be used the next time should an employee respond to the outage. I know there is one missing police report between the time the fake gun was used on Jason and when the taser was used on a different Jason. In addition all the thefts were planned when our eBay bought security system was not on or because we didnt pay the security bill and Chris had the equipment stolen to avoid any questions of why it was not on. I urge you all to take great caution with this company as plans for CDC-04 were not far behind as Chris is an egomaniac person who believes he can do anything. I know I will be looked at as a scumbag since I was aware of what was happening but please believe I had no idea your data was being stolen. Oh and Chris if you dont wind up in jail over this be very careful of who you fire from now on!
Posted by chopsmidi, 11-06-2007, 02:56 AM
Chrissy, If this is true, have you gone to the police?
Posted by AllyJ, 11-06-2007, 03:09 AM
A lot of good this does us if it is true. If it is then I say be a man/woman and go to the police if you really know what you say you know. If anything happens from this I am sure it wouldn't be hard to trace back to you and even with what you stated you are an accomplice to insurance fraud. If you come forward with your facts you might have some slack cut.
Posted by jaredbeckham, 11-06-2007, 04:12 AM
Chrissy if this is true PLEASE contact the police.. Detective James Swick, 312-744-8263 Case #HN-623601
Posted by RyanD, 11-06-2007, 04:48 AM
... those are some pretty serious statements. I think they should be made to the police rather than on a public forum. Knowing Chris' litigious past I'd be careful what I say about / around him
Posted by AllyJ, 11-06-2007, 01:52 PM
Well CI host has finally admitted there was a break in 4 times but still no word about 6. Still smells of BS to me. thewhir.com/features/110607_C_I_Host_Responds_to_Robbery_Reports.cfm
Posted by C I Host Team, 11-06-2007, 03:09 PM
To all concerned parties, C I Host is concerned about the misinformation and speculation surrounding the recent robbery at our Chicago Data Center. C I Host has experienced an unfortunate robbery at our Chicago Data Center which has affected our clients and our company greatly. While we did delay the information to both the victims and the public we apologize and recognize our notifications should have been more prompt. Our first priority was securing the Data Center, gathering the facts and notifying the clients that were victims in the incident. As stated in the articles, this is the 4th time our Chicago Data Center has been a target in criminal activities and we assure both our clients and the public we have taken appropriate steps to increase security in every event. This is an opportunity to reevaluate all of our security and improve it to the best possible levels and we welcome any suggestions you may have. Please understand that the improvements we have made and will continue to make will not be released for security purposes. C I Host continues to be committed to our clients and their needs during this unfortunate time and would appreciate that the speculation of the events and our actions be directed to us so that we may clarify and state the facts. Should you have any questions, comments or suggestions please direct your inquiry to James Eckels. James Eckels jamese@cihost.com
Posted by KarlZimmer, 11-06-2007, 03:26 PM
Sorry, this line is hilarious, "One of the biggest mistakes is that people are talking about four robberies. A robbery means than property has been seized through violence or intimidation. C I Host has technically only been robbed twice in two years. The other two were break-ins where things were stolen, but not robberies." The fact that the other two incidents were simply thefts, since no one even showed up to be intimidated, not robbery just makes it SO much better. I understand now... :-) Oh, and then trying to pass the buck to the building, even though it is CI Host who selected the building KNOWING the location and the security measures in place... Oh, and James, good job waiting until the 4th issue to make changes. I'm sure all your customers feel so much better now that you're reacting so promptly... Also, Chrissy, you should definitely go directly to the police and/or seek legal council immediately. Last edited by KarlZimmer; 11-06-2007 at 03:34 PM.
Posted by RyanD, 11-06-2007, 03:44 PM
So, Have you hired armed guards?
Posted by C I Host Team, 11-06-2007, 04:07 PM
We appreciate your inquiry. As asked in our post, please direct all inquiries to James Eckels at jamese@cihost.com. Please understand and appreciate that we will not publicly discuss the improvements made or future improvements to come for security purposes.
Posted by KarlZimmer, 11-06-2007, 04:26 PM
If they were actually good security measures everyone would be happy to hear of the change, and you can easily give vague enough updates that it wouldn't offer any assistance to those planning thefts, such news would actually deter such attempts. That leads me to believe the measures taken are less than adequate.
Posted by C I Host Team, 11-06-2007, 04:45 PM
Karl, Thank you for your response. Given the situation we feel it is in both our clients and our best interest to not disclose any changes in our security at this time. We would hope that since you are in the same industry you could appreciate that. Please feel free to direct your inquiry to James Eckels jamese@cihost.com.
Posted by KarlZimmer, 11-06-2007, 04:54 PM
Well, we publicly disclose the security measures we have in place and offer tours to everyone who asks, customers or not, so not, it doesn't really make much sense. It just seems like an odd stance for someone who says, "We've got nothing to hide." As I said above, showing publicly that you've increased security is going to discourage future incidents, but it seems you have no interest in preventing such incidents.
Posted by C I Host Team, 11-06-2007, 05:12 PM
Karl, While you are entitled to your opinion, please try to not make an assumption that we "have no interest in preventing such incidents". Given the sensitivity, the severity and the fact it has been repeated we do not feel it is wise to disclose our security measures at this time. We are 100% committed to the prevention of such incidents. If you feel so strongly about needing the information, then we invite you to contact James Eckels. Last edited by C I Host Team; 11-06-2007 at 05:15 PM.
Posted by pzil0cyb3, 11-06-2007, 06:49 PM
I am severely disappointed by the handling of the situation by CI Hosts's upper management. There was no reason to lie about what happened and to delay the truth for 3-4 days. I can't imagine how the news could be spun into something less negative. I am also disappointed with the lack of information or status updates provided by CI Host. It's been a month and I still haven't heard much of anything. You would think myself and other customers with losses would be kept up to date from CI Host directly instead of having to read things in the news. I'm just disappointed all around, especially for a hosting company that prides itself on having "state of the art data centers", "world-class web hosting services", and being the "#1 Web Hosting Company." As I've stated before, I used to be a proud supporter of CI Host, despite the plethora of naysayers on the web, but I'm one of them now.
Posted by brommel, 11-07-2007, 07:05 PM
CI host isn't a good webhosting firm, look at the reviews, I checked out their website and they claim to have armed 24/7 security, so where was the security if they had a break in? seems odd? If they didn't have security and claim it on their website and my servers were stolen I would sue, also I called the city to see if they were even a valid business, and they don't have a business license in the city, so they seem very shady to me. Also from research, CI host sued the BBB, so think about that....
Posted by brommel, 11-07-2007, 07:06 PM
Yes they did have a break in, it's all true.
Posted by brommel, 11-07-2007, 07:11 PM
I doubt a Chicago cop really cares about some servers being stolen, the datacenter is right near Cabrini green where open drug markets flourish, I really can see a Chicago cop taking time to look for some stolen servers, hell the cops probably don't even know what a server is.
Posted by chicagogringo, 11-08-2007, 12:04 AM
A very close relative of mine was on duty when the place was robbed back in Sept-Oct of '05. There was no security. (There never was) The thugs waited outside the hallway and simply waited for my guy on duty to take a leak, at which point they pistol whipped him, tied him to a chair and took off with around $40k worth of servers and other network gear, including his own personal laptop, Xbox and mp3 player. The company fired my relative, refused to pay his workmans comp (although that has been since settled in court) and still refuses to reimburse for personal lost property. There are other multiple lawsuits pending for other various issues, including unpaid overtime and shady taxation procedures. The "reinforced wall" they're talking about is a thin sheet of sheetmetal my relative put up shortly after his experience, before they fired him. Their claims of "24 hour armed security", at least until 2 years ago (and apparently, ever since) is totally bogus. The office was located on the 3rd floor; there was a keypad access on the first floor, but anyone with an office in the building had access, (all you had to do was wait outsie for someone to walk out) and then it opened up to the elevator, which had unlimited access to any floor. Basically wide open.
Posted by nelsonite, 11-08-2007, 07:37 AM
Why am I not surprised. CI Host screwed me over the whole time that I was with them. Being the gullible guy that I sometimes can be, I went ahead and bought a handful of dedicated servers from them only to be hit with constant and consistent downtimes, different stories from tech support staff about the reasons for outages and seriously questionable business practices. There's nothing in the world that can make a webhost justify lying to their customers. None. If you get robbed, which in itself should sound alarm bells, then at least have the decency and respect for your customers to be honest with them. Unless, as some have posited, it's an inside job. And here are a few more links to add to the set already posted by my fellow zero-posters about the unenviable outpouring of love for cihost all across the web! cyberiapc.com/forums/index.php?showtopic=10656 echoreply.us/tech/2007/11/04/ci-host-chicago-robbed-at-gunpoint-again
Posted by brommel, 11-08-2007, 09:55 AM
Well, from the surface it would seem that CI host would be on the hook for gross negligence, after this incident I wouldn't host with them anywhere. They sound like a crappy firm, I am glad you're relative wasn't too hurt. That is terrible.
Posted by brommel, 11-08-2007, 10:28 AM
misinformation and speculation hahaha, well there is the freedom of information act, which is black and white...facts are facts.
Posted by WebGod, 11-09-2007, 09:05 AM
Can someone recommend a Chicago Metro area host, where I can co-locate on a full rack for under $500/month currently I pay $300/mnth for a full rack with CI Host (1yr pre-paid contract) contract began in 2004 I'd love to be able to switch, but can find anything in my price range. <<< post recommendations here >>> Thankfully, I had no hardware stolen, but I lost $4,700 in business revenue. I was actually @ the data center one night before the break-in and the morning after. What a mess. Last edited by SoftWareRevue; 11-09-2007 at 10:38 AM.
Posted by FeliXdk, 11-09-2007, 09:17 AM
There's a thread here started by another C I Host client who needs a new provider. Although the person only needs 1U, the thread seems to have a few interesting recommandations for your needs too (don't know if their pricing matches your requirements though).
Posted by RyanD, 11-09-2007, 10:25 AM
I don't know of anyone that will match that pricing as that is absolutely insanely low. I would recomend you go check out Steadfast. Last edited by SoftWareRevue; 11-09-2007 at 10:40 AM.
Posted by KarlZimmer, 11-09-2007, 01:07 PM
Wow, what was included at that $300 a month pricing? I can't think of anyone in Chicago that could come close to that pricing, though that is then a perfect example of "you get what you pay for."
Posted by brommel, 11-09-2007, 01:30 PM
There is no price traction in hosting anymore, fixed costs, rising energy, only the big boys will be left soon, Microsoft is actually opening a massive hosting center in Chicago or north burbs soon, nobody will be able to compete with them, they will crush alot of small hosting firms. I wouldn't want to own a co-lo facility, in 2-5 years many co-lo's will be out of business, CI host probably included in that group.
Posted by brommel, 11-09-2007, 01:34 PM
Microsoft (MSFT) has confirmed its plans for a major data center in Northlake, Illinois, which trickled out through the real estate industry earlier this week. Microsoft senior director of data center services Michael Manos said the project will set a new standard for the company's data centers. Manos told the Sun-Times that the Northlake site, which is 14 miles from Chicago, was chosen for its excellent fiber connectivity. He also said climate was a factor, indicating that Microsoft plans to use free cooling techniques that use outside air temperatures to improve the efficiency of the facility's cooling systems. Free cooling can save power by reducing usage of compressors in chilling systems. There are some discrepancies about the size of the facility. The Sun-Times reports that it will be a 550,000 square foot data center, which is about the same size as the announced plans for Microsoft's new data center in Dublin. That would mean Microsoft has slightly increased the size of its facilities, as the data centers in Quincy, Wash. and San Antonio have 470,000 square foot footprints for their first phase. But the web site and marketing materials for the Northlake Data Center list the size as 430,000 square feet. The project is a greenfield (ground-up) build, as can be seen from the live webcam of the construction site, meaning the design can be tweaked to accommodate the additional space. According to the spec sheet, the facility has a 40 megawatt power feed, with the option of expanding to 60 megawatts. The plans by the developers, Ascent and the Koman Group, call for eight 50,000 square foot pods of raised floor space in a two-story structure. Plans call for 21-foot ceilings, which would allow raised floors of up to four feet. The specs call for a pre-action sprinkler system and VESDA fire protection. It's important to note that these are the developer's specs, and Microsoft will be able customize for its own design. Good luck to the competition in Illinois, you can't compete because Microsoft can offer volume and discount the crap to the point nobody will be able to make a profit due to rising higher fixed costs, and energy, and bandwidth prices coming down.
Posted by David, 11-09-2007, 01:36 PM
Brommel, Er, unlikely. But nice try. The market isn't all based on price. Anyone can push volumes of sales by dropping price.. CIHost proved that. But can they retain quality, that's what matters. The answer? A resounding no.
Posted by brommel, 11-09-2007, 01:39 PM
Thats where Microsoft and bigger firms can win on scale of efficiency, they use blades, CI Host uses crap they make in shop, no standardization, frankly most companies can't afford the efficiency needed to be profitable now in the hosting industry. Now with VMWare why would anyone even buy a server and put into a datacenter, all you need to do is have a VM instance with you're data on a Blade server in a large datacenter. Hosting firms are going to go the way of the mom and pop ISP circa 1995ish. It's a mature field now, and expectations are much much higher for excellence. Downtime is not even acceptable.
Posted by KarlZimmer, 11-09-2007, 01:46 PM
Ummm, do you know anything about their actual business plan?? Has the San Antonio facility completely destroyed the Texas market? From what I can tell, NO. 1) The demand for data center space over the next several years is MUCH more than 430,000 sq. ft. I know of ONE company that is basically a data center broker that has brokered over 1 million sq. ft. of data center space over the past two years and indicate demand is just growing. 2) Microsoft, if they are even in the colocation business, though it has been shown most of their data centers are built for their own purposes, their own sites, their own internal systems, etc. they are going for a much different market than most people here on WHT and they are definitely going for a different market than us, Gigenet, Server Central, etc. Their market is solely Fortune 1000 companies looking for LARGE amounts of data center space, space that is larger than the total amount of space any of the companies I listed even have in Chicago... It is not worth Microsoft's time to compete for individual dedicated servers, or 10 cabinet deals. 3) It isn't as if they are doing anything different. DuPont Fabros has been building massive facilities for years, including one in Chicago that is 550,000 sq. ft., yet all the smaller providers are still doing fine. I guess in short. 1) There is enough demand to go around. 2) This business has many varied market segments, all looking for something different and you're never going to find one provider that is right for everyone. 3) This happens ALL the time and has never been a big deal previously.
Posted by KarlZimmer, 11-09-2007, 01:52 PM
1) How are blades any more efficient? The Supermicro systems we have been using, with high efficiency power supplies, are cheaper and use less power than any of the blade systems we have seen marketed and/or tested by customers or colleagues. They also don't stick you with proprietary motherboards, switch modules, etc. 2) The "VMWare" thing is already being done by dozens of smaller companies and other similar products, such as Virtuozzo and Xen are even more prevalent. I agree, virtualization is a large part of the future here, but I don't see how that gives the big guys a leg up. 3) ANY downtime is already unacceptable with any large clients and has been the case for years. What is the difference/change there? 4) How did this thread turn from CIHost to the future of the web hosting industry?? Seems a bit off topic... You have a reason for taking this thread off topic??
Posted by brommel, 11-09-2007, 01:55 PM
I am glad you're business acumen is as good as you're technical, these massive datacenters are just coming online, lets see where it goes in 2-5 years. Electrical costs are going to get higher, the profit margins in a DC are already thin, many Hosting firms from my guess will start having cash flow problems, I am betting many are already.. I am sorry to bust on anyone's parade, but look at it from an econmics standpoint. Where will the price traction come from? As an investor, I wouldn't invest a dime in a new co-lo or a webhosting firm. The smallest amount of production a company can achieve while still taking full advantage of economies of scale with regards to supplies and costs. In classical economics, the minimum efficient scale is defined as the lowest production point at which long-run total average costs (LRATC) are minimized. The minimum efficient scale may be expressed as a range of production values, but its relationship to the total market size or demand will determine how many competitors can effectively operate in the market. If the minimum efficient scale is relatively small compared to total market size (a good example would be computer software), many companies can exist in the same space. In other industries - such as telecom and basic materials - the minimum efficiency scale is quite large due to the high ratio of fixed costs to variable costs. In these types of industries, only a few major players will tend to dominate the space. This will start happening in the Hosting world.
Posted by brommel, 11-09-2007, 02:03 PM
How are blades any more efficient? Where the blade server wins out is on compactness, greater integration of server and network components, and lower power consumption. In a typical 1U rack-mounted server, about 30 percent of the power use of the entire machine was from the CPUs, with memory accounting for 11 percent, PCI buses for 3 percent, the backplane for 4 percent, the disk drives for 6 percent, 2 percent for standby components, and the remaining 44 percent for power and cooling components such as power supplies and fans. In a blade server, that 44 percent is reduced to 10 percent because of the sharing of these components. This gives the blade server a tremendous advantage when it comes to electricity consumption and heat dissipation. An individual blade can cost 25 percent less than a configured 1U server, is 33 percent more efficient when it comes to power use, and takes up half the floor space. The virtualization and internalization of the network in the blade approach also cuts cabling costs by as much as 86 percent, you can quickly see where the advantage is in a 500,000 sq foot Datacenter.
Posted by brommel, 11-09-2007, 02:06 PM
You can't think like a tech geek, have to think about the business as a gestalt, as a whole, and how all the small parts come together. If you want to make you're DC more price efficient, I can consult with you and we can work out some terms.
Posted by KarlZimmer, 11-09-2007, 02:20 PM
Are you just a blade marketing drone?? 1) Space does not matter, you're going to reach the limits of what you can cool efficiently, etc. well before space becomes an issue. 2) Yes, the blade CAN be cheaper, if you're getting ripped off on your 1u server purchases. I have not seen any blade configuration come close to what I spend on 1u systems. The dual Clovertwon E5335 blade from Dell, with 2GB of RAM and a single 73GB drive is $3500, I buy those servers for less than $2000 and they can hold more hard drives. 3) From the testing I have done, with using the high efficiency power supplies, etc. the power usage of the system is 5-10% less than that of an equivalent Dell blade configuration. 4) They cut cabling costs in exchange to adding costs for the blade chassis itself, along with the proprietary switch modules, etc. That definitely costs more than the whole $100 I spend on cabling, including cable management, etc. for a full cabinet of 1u servers. Yes, that price also includes the cost of the man hours in running the cables. See, look, I use real world examples, not just numbers given by a sales/marketing department... Honestly, if ANYONE is running systems where 44% of the electricity is used by the power supply and/or for fans/cooling inside the chassis you need serious help. I'm sure those systems exist, but that is not even close to the norm, at least in my facilities. Also, you really believe they have a redundant power supply configuration for blades that is more than 90% efficient??
Posted by KarlZimmer, 11-09-2007, 02:23 PM
Based on the "knowledge" you have been showing in this thread, take a hike... I'm done with this aspect of this thread, it has already gone way too far off-topic. If you want to continue the discussion feel free to IM me or PM me, all my contact info is in my signature or in my account profile.
Posted by brommel, 11-09-2007, 03:00 PM
If you have so much more knowledge than me, why not price match against CI host if it's so affordable, oh you can't because you would be out of business. You never answered my question about price traction, because you know the margins are so narrow, You take a hike, in 2-5 years I am sure you're little itty bitty firm will join the rest in the out of business pool. Right now Microsoft deals with bigger clients, but they will scale down, and crush you along with others in same size, as they capitalize on small and med business hosting needs, you have an ego, and won't face up to the reality of it.
Posted by KarlZimmer, 11-09-2007, 03:30 PM
Wow, ok, show me ANYONE that can price match that and turn a profit... You're using a non-sustainable business model, as has been proven in this thread, as the basis for your point...
Posted by moosh28, 11-09-2007, 03:41 PM
What makes you think Microsoft wants to toil in colocation services for the masses?
Posted by moosh28, 11-09-2007, 03:47 PM
Wouldn't call Karl or many of the players in the market "itty bitty". Plus, I fail to see where or how it makes sense for Microsoft to get in the colo game.
Posted by pzil0cyb3, 11-09-2007, 07:18 PM
Likewise I don't think Microsoft is getting into the colo game.. it's probably moreso to compete/catch up with Google.
Posted by FeliXdk, 11-10-2007, 01:19 AM
I agree, since when should the Microsoft (or Google for that matter) data center projects be considered a threat to the co-location industry? These companies are focused on supporting their own services, which has been proven by their previous data center projects. Regarding blades vs. 1U, I also haven't seen any price advantage on turning to blades versus the 1U configurations we are using. You are obviously also off track on this matter.
Posted by tsj5j, 11-10-2007, 01:32 AM
It would be stupid to price match a failing company, because that will just push you down the same route. You're not using your brains at all, just writing out of pure spite.
Posted by tsj5j, 11-10-2007, 01:35 AM
I can forsee it affecting in colocation, but not in the near future. If they can really offer more competitive pricing, colocation companies might move there and provide colo space there. It's like how Microsoft (Windows) often neglects its end users, but OEMs are there to facilitate distribution on new machines.
Posted by brommel, 11-10-2007, 02:23 AM
yeah, Microsoft probably won't get too much into colocation, they probably see the low ROI, and won't touch it, but other big players exist that will eventually put the squeeze on the smaller firms. There are hardly any publicly traded web hosting firms, so that goes to show you how small most of the firms are.
Posted by tinkertim, 11-10-2007, 03:40 AM
There can be a significant price advantage using blades, in both density and power savings. The blades that I'm using on 2 clusters are completely fan-less, cooling is managed by the blade housing. 7U of space gets me 10 or 12 blades, depending upon the configuration that I'm using. That mans more paying customers per 42U. Power is also a major issue in some places, with blades I pull less watts with greater density. They aren't for everyone, of course, but there is a savings to be obtained by using them should they fit your particular need.
Posted by KarlZimmer, 11-10-2007, 10:45 AM
What data center is allowing you to fill full cabinets with blades to take advantage of the space savings?? I don't know of ANY facility built to the 700 watts per sq. ft. needed to take full advantage of blade servers and I have doubts as to whether or not that is even physically possible with standard air cooling. As for power usage, what are you comparing the power usage to? As I stated above, in my testing, the high efficiency SuperMicro systems were actually slightly more power efficient than a blade configuration, though it is true that the only blades I have seen/tested are the Dell and HP models.
Posted by tinkertim, 11-10-2007, 11:10 AM
We're in a facility in Scarborough, Canada. We have quite a few racks there, so we pretty much get to do whatever we want within reason and logic. The owner of the place is very involved in its operation, we don't get many odd looks unless we're drawing a DoS. The blades that Dell and IBM sells are gorgeous, stealthy looking, but not very energy efficient. Our's are based on an Intel/Asus MB with built to order PSU's. I've worked, several years with specialized clusters using fan-less SBC's on segmented PCI backplanes, believe me, central cooling is more than efficient, in some cases it can be better. As I said, they aren't for everyone. My point in posting was simply to indicate that some blades (if engineered properly) do provide a significant energy (and cost) savings.
Posted by Nkrapf, 11-10-2007, 03:48 PM
This is who we are moving to [FONT='Arial','sans-serif']Jon Biloh ColoCrossing.com jbiloh@colocrossing.com Office: +1 (888) 275-5754[/FONT]
Posted by pzil0cyb3, 11-15-2007, 03:49 AM
A new article.. chicagoreader,com/features/stories/hottype/071115/
Posted by unixbox, 11-15-2007, 03:53 AM
We have many more to come. We have 3 in the Dallas area. We are also getting TV coverage later next week. We will make sure everyone knows about what they are getting by using cihost Last edited by unixbox; 11-15-2007 at 03:57 AM.
Posted by tinkertim, 11-15-2007, 06:16 AM
HOLY COW! The walls are plaster?!?! The other articles that I read lead me to believe that the thieves got through a concrete wall, because the articles talked about an 'exterior wall' being breached. This is even worse than I thought. That's a good read, it brings some interesting points into clarity: Even AFTER the robbery, someone (the reporter) was able to get in without even showing ID (so much for biometric hand scanners) It is NOT at _ALL_ a secure facility. You can get through most drywall partition walls with just a _running_start_! No saw needed! I thought that the facility itself might be inherently secure if the human factor improved a bit, obviously not the case. I think CI Host might win the Brain Dead Of The Year Award for this.
Posted by unixbox, 11-18-2007, 07:16 AM
Illinois Security Services(ISS) was supplying 24/7 on site security but CiHost decided not to pay them. So they pulled out of the center leaving it empty during certain hours. This was interesting to hear from the owner of ISS because if CiHost paid the bill and the center was still robbed ISS would of cover the losses. In my opinion a company that has been around for so long has a lot of growing up to do. Take responsibility for your actions and care about your reputation not just $$$. 14.4 million in sales and they cant refund loyal customers for losses screams financial problems. I would not be surprised to see the headlines CiHost files Chapter. But that is just my opinion.
Posted by tinkertim, 11-18-2007, 07:51 AM
Can you post a link with the source of that information? I don't doubt it at all, but would love to confirm it prior to relaying it word of mouth. I know a few people who were shut down due to that fiasco. Again, I'm not at all surprised to hear that.
Posted by unixbox, 11-18-2007, 01:35 PM
Have them email me as it would would be great if they joined our lawsuit. info at unixbox dot ws I need the following information: Copy of the contract they signed if any. Some of us dont even have a signed contract. What was stolen and its value... What was lost due to the downtime or robbery.
Posted by brommel, 11-18-2007, 07:10 PM
14 million in sales, that data is from 2002, but CI host is a private company and they can say they make 100 million a year, I am betting that the real figure is closer to around 1.2 million today. After fixed cost they probably are just barely making pay roll and keeping the lights on. They closed a Datacenter in Newark, that doesn't sound like something a firm making tall cash would do.
Posted by hivelocitygm, 11-20-2007, 04:55 PM
CIhost addresses the robbery. http://www.thewhir.com/features/1106...ry_Reports.cfm
Posted by KarlZimmer, 11-20-2007, 07:58 PM
They MUST be making more than $1.2 million in revenue a year with the number of facilities they have, etc. We bring in significantly more than that and I'd have to say with one location, etc. that we're smaller than CI Host. Though they do charge a TINY amount for what they offer...
Posted by onejdev, 11-29-2007, 02:04 AM
This whole thing is absolutely nuts. Since finding out that there had been a string of robberies at the Chicago data center, I have tried vigorously to have my server moved to the Dallas data center. I had worked with a woman at CI to get this done, and towards the end of the month of October I hadn't heard back from her and tried to email her again to get a status update. Imagine my surprise when I found out the email bounced back as an unknown user. After sending an email to the sales staff expressing my concerns yet again, I was told my email was forwarded to the COO of CI Host. This was all on november 5th. Not surprisingly I haven't heard back from CI host. Below is the email that I sent out to CI host today. It was addressed to the following people: sales@cihost.com, abuse@cihost.com, billing@cihost.com, allanb@cihost.com (COO), lindap@cihost.com (person who I dealt with first, but just bounces now), rockyp@cihost.com (handled my inquiry on Oct 4, before I knew about the robbery), customerservice@cihost.com, collections@cihost.com, jamese@cihost.com (responsible for giving credits according to a previous email I have from CI) I sent this at 1:44pm EST and I have not received a response, other than an automated response from the billing department informing me of their new corporate address which is as follows (for those of you who may need it). Lets see if I can get a response out of them in less than 30 days. I'm not holding my breath. Last edited by the_pm; 05-11-2008 at 09:33 PM.
Posted by ub3r, 11-29-2007, 04:52 AM
I would recommend you switch to a completely different, non-cihost datacenter. Unless, do you have any dependencies on cihosts' ip addresses? You would have to switch all those, probably even if you were switching to a different cihost datacenter.
Posted by vpncast, 11-29-2007, 09:24 AM
im sorry to stick my nose in to none of my business but why do I always hear about C I Host being robbed, down, everything but good.
Posted by onejdev, 11-29-2007, 12:00 PM
Believe me, I have researched other data centers and have gotten many quotes to get myself out of there. The problem here is 2 fold. 1) I can't even get a response out of them via phone or email, so even if I did get a contract somewhere else, contacting them to cancel would probably prove to be a pain in the ***. I would probably end up having to fly out there to take possession of my equipment, though I would do it without a stun gun or a pistol. 2) The only dependencies I have on CI Host are their attractively low prices. The other data centers I have looked at start at 2-3 times the price of CI. My server is nothing mission critical, and if it goes offline, I don't lose thousands of dollars, so I can almost put up with having it hosted in another of their data centers as long as the uptime isn't an issue and its not prone to burglary. My budget for hosting doesn't allow me to pay any more than what I am paying now. Like I said, colocated hosting is a luxury, not a necessity to me.
Posted by KarlZimmer, 11-29-2007, 12:41 PM
If it is your equipment, you can remove it. Find a provider in Chicago willing to work with you, send them say the signed contract, proving you have service, along with a list of systems, receipts would be preferred, and signed authorization for those people to take the systems, and I'm sure they could find someone to go and pick the systems up. I'm sure there would be a fee, but it would likely be easier than you needing to go to Chicago. Yes, the thing that makes you stuck is you're not willing/able to spend anymore. At the pricing CI Host charges, the level of service they're giving you is about what you can expect.
Posted by onejdev, 11-29-2007, 01:05 PM
Good point Karl. I actually hadn't thought of asking a provider in Chicago (should I stay in the Chicago area) to pickup the equipment for me. Ideally, I would be able to find a host closer to home, Boston, but thusfar it hasn't been so successful. I have contacted one host that has come close to matching the price (which I could actually live with the price increase) but I used to work for them, and I know the inner workings of that place and how things are handled. In that same respect, it would be a customer service nightmare. Also, they are prone to the same type of break in as well. Its an unfortunate truth, and it bears repeating, but you get what you pay for. I pay a small price yearly, and I get just that back in return for customer service, small if any. And believe me, I am more than willing to pay to have the server hosted somewhere else, its just my current financial situation doesn't figure in having to pay double for something I have paid a set price for over the past 2 years.