bluemilecloud.com down

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > bluemilecloud.com down

Posted by halfgeeks, 02-06-2012, 01:01 PM
Both of my VPSs are down, as well as their main site, and phone system. They sent out a maintenance notification last week but that was for the 4th, and said it would only be 5 minutes. I would really just like confirmation that the issue is being worked on and they did not just disappear off the face of the earth. Seems kind of silly to not have a backup phone line that is separate from their network. It looks really bad when I can't get anybody on the phone. Hope they are back soon. Tracing route to bluemilecloud.com [68.68.19.53] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 10.1.10.1 2 39 ms 25 ms 29 ms 96.140.84.1 3 170 ms 51 ms 11 ms te-9-5-ur01.normal.il.chicago.comcast.net [68.85.179.41] 4 24 ms 23 ms 23 ms te-0-2-0-3-ar01.area4.il.chicago.comcast.net [68.85.177.198] 5 31 ms 25 ms 23 ms pos-3-11-0-0-cr01.350ecermak.il.ibone.comcast.net [68.86.90.13] 6 28 ms 27 ms 27 ms pos-1-5-0-0-cr01.chicago.il.ibone.comcast.net [68.86.88.37] 7 38 ms 27 ms 33 ms xe-8-2-0.edge1.Chicago2.Level3.net [4.71.248.65] 8 31 ms 24 ms 29 ms vlan52.ebr2.Chicago2.Level3.net [4.69.138.190] 9 23 ms 23 ms 25 ms ae-5-5.ebr2.Chicago1.Level3.net [4.69.140.193] 10 30 ms 30 ms 30 ms ae-1-10.bar1.Cincinnati1.Level3.net [4.69.136.213] 11 38 ms 36 ms 43 ms BLUEMILE-IN.bar1.Cincinnati1.Level3.net [4.59.40.2] 12 * * * Request timed out. Last edited by halfgeeks; 02-06-2012 at 01:04 PM. Reason: Posted tracert
Posted by FHT-Ben, 02-06-2012, 01:12 PM
We are also down. Looks like the whole network is down.
Posted by zenex5ive, 02-06-2012, 01:45 PM
They are experiencing power issue.
Posted by halfgeeks, 02-06-2012, 01:46 PM
Hey, thanks for the tip. Just wondering how you managed to track down that information? I have had no success making contact with them. -Ian
Posted by zenex5ive, 02-06-2012, 01:49 PM
I spoke to one of their network engineer.
Posted by halfgeeks, 02-06-2012, 02:40 PM
The phone system is now working again, so that is a good sign. Still no servers yet, but they say they have people working on it.
Posted by halfgeeks, 02-06-2012, 05:02 PM
Still down. I noticed at 10am but it could have been earlier that it went down. Now it's 3pm. Not getting any info from support other than "power issues" which seems more like "we don't want to say what the real problem is". I am getting through 2 more hops on the trace route now at least: Tracing route to 68.68.19.53.customer.bluemilenetworks.com [68.68.19.53] over a maximum of 30 hops: 1 2 ms 1 ms 2 ms 10.1.10.1 2 35 ms 21 ms 20 ms 96.140.84.1 3 9 ms 8 ms 9 ms te-9-5-ur01.normal.il.chicago.comcast.net [68.85.179.41] 4 24 ms 23 ms 23 ms te-0-2-0-3-ar01.area4.il.chicago.comcast.net [68.85.177.198] 5 55 ms 104 ms 62 ms pos-3-11-0-0-cr01.350ecermak.il.ibone.comcast.net [68.86.90.13] 6 27 ms 52 ms 50 ms pos-1-5-0-0-cr01.chicago.il.ibone.comcast.net [68.86.88.37] 7 90 ms 24 ms 24 ms xe-8-2-0.edge1.Chicago2.Level3.net [4.71.248.65] 8 33 ms 30 ms 24 ms vlan52.ebr2.Chicago2.Level3.net [4.69.138.190] 9 49 ms 25 ms 28 ms ae-5-5.ebr2.Chicago1.Level3.net [4.69.140.193] 10 34 ms 33 ms 35 ms ae-1-10.bar1.Cincinnati1.Level3.net [4.69.136.213] 11 35 ms 35 ms 32 ms BLUEMILE-IN.bar1.Cincinnati1.Level3.net [4.59.40.2] 12 39 ms 33 ms 47 ms 68.68.24.162.customer.bluemilenetworks.com [68.68.24.162] 13 63 ms 32 ms 41 ms 68.68.24.9.customer.bluemilenetworks.com [68.68.24.9] 14 * * * Request timed out.
Posted by halfgeeks, 02-06-2012, 05:39 PM
Well, my stuff is back online. Their main site is still down though. To my surprise, twitter ended up being pretty informative: https://twitter.com/#!/search/%40BluemileCloud I sure hope they release some sort of statement outlining how this will be prevented in the future.
Posted by bspinner, 02-10-2012, 07:36 PM
any more on this or was there a statement of outage issued? would be sort of surprised if was a power issue instead of some sort of network issue?
Posted by halfgeeks, 02-11-2012, 03:18 PM
Here's what they said: At approximately 10:30 AM EST, the Bluemile Columbus Datacenter experienced a series of electrical faults which disrupted some customers' service. At 10:30 AM EST the Bluemile Columbus Datacenter lost utility power. As designed, the backup power systems kicked in, which included the generator start-up, and provided backup power from the UPS batteries. At this point, technicians discovered the datacenter had suffered a surge from the utility provider which damaged a breaker at the generator and required a repair by a licensed high voltage electrician. While the generator was running, the damaged equipment prevented the site from receiving backup power. Due to the damaged equipment and the length of time required for the electrical team to make the repair, the UPS systems abnormally dissipated causing minor issues. Power was restored at approximately 1:50 PM EST. When technicians moved the Auto Transfer Switch at 3:10 PM EST to normal operating mode the site power dropped for a sub second and all but UPS #3 continued to deliver power, at which point we discovered the events of this morning damaged a string of batteries. As indicated in our prior correspondence, we ordered another set of batteries and had the failed set replaced at 7:13 PM EST. During this replacement we experienced no loss of power and at this time all major systems are restored.
Posted by bspinner, 02-12-2012, 02:39 PM
i stand corrected. power failure extended by cascading events. i guess eliminating single points of failure is hard and/or expensive.
Posted by bspinner, 02-17-2012, 08:58 AM
this sounds off...a surge from their utility feed making it to their generator (ie, risk of backfeeding the utility from the other direction) Last edited by bspinner; 02-17-2012 at 09:03 AM.