iRacks Service Outage

Portal Home > Knowledgebase > Articles Database > iRacks Service Outage

Posted by james-iracks, 02-13-2004, 08:19 AM
In the last 20 minutes (as of ~04:15 PST), iRacks has received numerous calls from customers whose machines' have lost connectivity; we've confirmed that this is a fiberring problem - fiberring are experiencing a loss of connectivity across their network, and prior to the installation of our new sets of network peering in the intermediate future, fiberring provide all of our connectivity. After consulting with fiberring and redbus, we can confirm that we have been told that our connectivity should come back within the next few minutes (and that this should take no longer than 30 minutes or so). Our apologies for this outage, which is entirely out of our control; rest assured that we are, and have been, working with the providers in question to make sure that this issue is resolved as quickly as possible. Last edited by james-iracks; 02-13-2004 at 08:28 AM.
Posted by james-iracks, 02-13-2004, 08:26 AM
We've just spoken to redbus and fiberring again - fiberring have people on the ground fixing this problem *at the moment*, and it looks like this is a bigger problem for fiberring than originally anticipated; whilst redbus and fiberring both quoted us a figure of 'a few minutes' initially, the fiberring engineer at redbus who we've just spoke to has no idea as to anything but the magnitude of the problem - and in light of the magnitude of the problem, it's looking like this might be somewhat longer in duration than the 'a few minutes' which we were originally quoted. Rest assured that iRacks will be updating this thread as new information becomes available to us. Simon Lorentson is also working on getting this issue resolved; any queries with regard to this may be addressed to him on his mobile number, who will also have the most up to date information on this issue; Simon Lorentson - 07776 196937 Again, our apologies.
Posted by james-iracks, 02-13-2004, 09:11 AM
As of the last 25 minutes or so, we've had limited service restored by fiberring; service is slow, but at least works. We're currently investigating the outage, and are chasing fiberring up as to what caused this (evidently large) outage). Our apologies are extended to all of our customers who were inconvenienced by this outage.
Posted by james-iracks, 02-13-2004, 09:30 AM
As of approx. 0526pst, service to iRacks network via fiberring appears to have been fully restored; we intend to investigate the reason for the fiberring outage, and should have an update for this within a day or so for customers. Again, our apologies and thanks for your patience.
Posted by james-iracks, 02-13-2004, 01:32 PM
We've spoken extensively with fibering, and the problem was tracked down to the Cisco 12000 core routers which they use for the redbus/telecity site. The information we received directly from fibering states that: "Both the Cisco 12000's in TeleCity2 shut down due to a memory overflow. The available memory should normally be sufficient to handle 50% more routes than we currently get announced. The 12000's are setup in failover configuration, but as R2 experienced the same probelems as R1, they both shut down." fiberring had technicians at redbus within 45 minutes of the issue occuring, and the issue was resolved after 90 minutes of downtime; one of their carriers had a similar issue on a peering router. Steps which fibering are carrying out to resolve the issue include: "1. Different IOS version on R2 (completed) 2. Remote console connected via different uplink (in process) 3. Restrict available bandwidth of new customer to 250Mbps (completed) 4. Active monitoring of available memory on 12000 (completed) 5. Add more restrictions in router configs (completed) 6. Request feedback from Cisco (in process) " In the short term, steps being taken by fiberring include: 1. Set up a new - fully separate - network in another location, to cope with the bandwidth growth (in process) 2. Streamline feedback to customers (in process) 3. Setup fault pages at InterXion (in process) Whilst fibering are looking into redundant routers running different hardware to prevent a similar issue from causing a problem again. We feel confident that this issue has been resolved satisfactorily, and that it will not occur again; once more, we extend our apologies to all of the customers inconvenienced by this issue, and hope that all parties involved feel that this issue has been resolved satisfactorily. Any customers still concerned about this issue are welcome to contact us via the normal means.