TechieMedia down (12+ hours)?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > TechieMedia down (12+ hours)?

Posted by devilsdreams, 05-28-2010, 02:34 AM
techiemedia uses Level(3) Communications bandwidth

Support site as unavailable for most of thursday but came back this evening.

Sites not resolving as of yet (confirmed on other message boards)

This outage seems to be going on 12+ hours now.
Reasons given range from: "a router problem" to "a fiber problem"

My MRTG charts haven't updated all day.

Any news would be be a huge help, the most info I've gotten from a tech at TM regarding when they estimate this problem to be repaired is "unknown at this time"
Posted by devilsdreams, 05-28-2010, 03:13 AM
Sites are resolving again. Problem seems to have been resolved at 3:10am EST (for me at least)
Posted by devilsdreams, 05-28-2010, 05:14 AM
From an email techiemedia just sent out:

Quote:

The problem started at 10am EST when we immediately noticed that our links were down to our second datacenter. We saw link but no activity on our 2 sets of fiber that connect our 5th floor datacenter with our 8th floor datacenter.

We run a layer-3 switch setup with load balanced hot spare connected to our core network between floors to light our second facility. Our immediate thought was that the primary switch failed in some way and the secondary did not take over like it should. This lead to a very long trouble shooting process with our tech even involving the switch vendor who had us update IOS and just about every test they could thing of with no result since the switches were identical in every way we had to be sure. We then decided to put a cold spare switch online to take their place which was a different brand and we had the same results, link and no activity.

The next step was to have the fiber company that handles our dark fiber test everything and also the the building meet me room check everything, they both told us the problem must be with our equipment.

We went back to more trouble shooting and actually brought the primary switch from the second datacenter all the way up into the main datacenter and connected it to the core on short fiber and it was working fine. It just did not work when connected between floors.

We then went back and demanded that the fiber company take a second look. This time they DID find and issue and corrected it and our second datacenter was back online.

This is one of those times we did everything we possibly could, worked tirelessly with all vendors that were involved and the problem was still caused and then prolonged by something we could not control as the hours of the day just seemed to fly by. So, to sum it up we got hit in a soft spot and it hurt.

We have been in constant contact with our customer and many have been told what we just posted by phone and ICQ.