Sago is down???

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Sago is down???

Posted by sdsolutions, 04-19-2013, 12:29 PM
Anyone out there on Sago?

Everything's down... Our service and their website...
Posted by Antonimo, 04-19-2013, 12:58 PM
Down here too. Everything.
Posted by bear, 04-19-2013, 01:01 PM
Same. Time to get off my butt and move the last boxes off there, I reckon. Been happening every few days now, and not being able to reach them during one of these is moronic.
Posted by Antonimo, 04-19-2013, 01:13 PM
Up for a couple of minutes, down again now.
Posted by sdsolutions, 04-19-2013, 01:15 PM
Tried calling them, they said it'll come up in a few...

Damn! I'm really thinking twice now... Just need an SSAE16 cert!
Posted by jiashanuk, 04-19-2013, 01:24 PM
down for 2-3 hours from UK, their network is not reliable this whole week.

I tried to call them, no one answered the phone.
Posted by avengerx, 04-19-2013, 02:28 PM
I am getting such unstabilities for a month or so. I noticed degradation in stability since end of last year (around november), but it was still acceptable.
I was monitoring my box when it started massive unstability on ~11:40:00 BRT (GMT-3, 16:40:00 CET if I am right). Since then, it is on that down-up-down-up. While out, my machine is still logging in 5-minute steps with mtr.

I doubt all that logging will be useful for sagonet, now the problem is VERY clear.

Well, for those who did not receive it... Sagonet was somewhat aware of the problems:
--------------
Official Report: 4/18/2013

Outage Report and Incident Report:
Incident and customers affected:
At 9:06am Wednesday, April 17, 2013. Sago experienced a throughput capacity deterioration and failure by one of 4 of our Border Gateway Protocol (BGP) bandwidth carriers. Our long haul carrier’s circuit went down and forced the Border Gateway Protocol BGP redundancy to automatically role over the Open Shortest Path First OSPF and Border Gateway Protocol BGP protocols of which are Sago’s core routing protocols. This situation created a random effect on customer which were coming in and being routed from this carrier’s circuit.
Up to 60% of customers felt no impact while the remaining 40% of Sago’s customers experienced up and down results in services until the BGP flapping was manually stopped and the routing tables updated with other 3 BGP active carriers? The Circuit was identified and removed from production to clear the false positive routing advertisements causing spotty service through Wednesday April 17, 2013 10:00am.
At 3:30pm Wednesday, April 17, 2013. Sago experienced a throughput capacity deterioration and failure by one of 3 of our active Border Gateway Protocol (BGP) bandwidth carriers. Our second long haul carrier’s circuit went down and forced the Border Gateway Protocol BGP redundancy to automatically role over the Open Shortest Path First OSPF and Border Gateway Protocol BGP protocols of which are Sago’s core routing protocols. This situation created a random effect on customer, which were coming in, and being routed from this carrier’s circuit.
Up to 80% of customers felt no impact while the remaining 20% of Sago’s customers experienced up and down results in services until the BGP flapping was manually stopped and the routing tables updated with other 2 BGP active carriers? The Circuit was identified and removed from production to clear the false positive routing advertisements causing spotty service through Thursday April 4, 2013 5:15pm.
Incident response and corrections:
Our network engineers were able to isolate the two offending circuits in the routing systems that were causing the up and down results in services. Human intervention from Sago’s IP engineering team was required to remove the offending circuit. Sago has acquired a 2 new 10Gig long haul circuits April 18, 2013 to be installed by Tuesday, April 23, 2013. This will be added and replace the two long haul carriers that created the issues experienced Wednesday April 17, 2013 If you are experiencing any issues you feel are related to this service interruption please report the abuse to Sago’s Network Operations Center (NOC) immediately with reference to this report and any information which supports this abuse
Thank you,
Jonathan E. Cox
Chief Operations Officer
Sago Networks, LLC.

Main Phone (866) 366-3640
4465 West Gandy Blvd. • Suite 800 • Tampa, FL 33611
---------------------------
Posted by J0o0j, 04-19-2013, 03:08 PM
its back now
Posted by J0o0j, 04-19-2013, 03:33 PM
it will be down agen
Posted by avengerx, 04-19-2013, 03:46 PM
I suppose we can only assume it is back for good once we receive any notice from one sago net representative. I think I will open a ticket to ask them to notify me when the system is back in once piece.

of course, I will try to open the ticket once I catch the servers up again.
Posted by bcinc, 04-19-2013, 03:56 PM
We've been down so often lately that I've turned down our monitoring, since I would otherwise be getting almost constant alerts. One box left there that I've been waiting to retire but am going to have to move now since they are just continually down and without any explanation.
Posted by jiashanuk, 04-19-2013, 03:56 PM
on their facebook and twitter, they claimed they are under DDOS attack
Posted by J0o0j, 04-19-2013, 03:56 PM
In you any information arrived hope News
Posted by ath1234, 04-19-2013, 04:40 PM
Today is pretty much a loss for us and our 300+ customers. Has anybody heard anything from Sago? I keep being told they're 'working on it', but no official word of what's happening.
Posted by avengerx, 04-19-2013, 04:48 PM
According to website at dns.measurement-factory.com/surveys/openresolvers/ASN-reports/latest .html, sago net has 47 IP addresses affected by DNS amplification attack vulnerability, an almost-nothing near the vast number on my country's TELESC, leading the list with more than 3.000 compromised hosts.

But probably the 47 on sago net are worthy at least 90% of the trouble all the three thousand hosts on TELESC could do on their technology-limited, capped bandwidth.

It is just a possibility though, and known potential trouble on the network.
Posted by J0o0j, 04-19-2013, 05:03 PM
it work now
Posted by ath1234, 04-19-2013, 05:10 PM
Spoke with a tech at the NOC, was told they are having trouble with their core routers and every engineer is working on it. This is crazy.
Posted by J0o0j, 04-19-2013, 05:15 PM
Is there a global news regarding this issue ؟
Posted by Antonimo, 04-19-2013, 05:16 PM
Almost 6 hours down now and no official word about it... shame....
Posted by ath1234, 04-19-2013, 05:23 PM
Has to be a routing issue. I'm pinging our equipment (in Atlanta) and their website (in Tampa), and it's been that as soon as one comes up, the other goes down. Our stuff would be up in ATL for a while, then the Tampa IP would start responding to ping and ours would promptly go down, then Tampa would go down and ours would come back up a minute later. Godaddy was hosed for a day due to routing table issues a while back.
Posted by J0o0j, 04-19-2013, 05:28 PM
But we need permission from the company
Posted by J0o0j, 04-19-2013, 05:45 PM
it work now
Posted by ath1234, 04-19-2013, 05:47 PM
First time today I've seen both Atlanta and Tampa facilities online at the same time for more than a couple minutes... maybe they're getting a handle on it? Not too confident to be honest, which is why we're getting an outside internet connection directly to our rack for backup purposes.
Posted by J0o0j, 04-19-2013, 05:56 PM
Heck of a company does not respect its customers
Posted by Antonimo, 04-19-2013, 05:58 PM
down again
Posted by jiashanuk, 04-19-2013, 05:58 PM
down again!!!
Posted by J0o0j, 04-19-2013, 06:02 PM
also me
Posted by ath1234, 04-19-2013, 06:04 PM
We have just been told that they have found and are working to replace a failed part of their core routing system. Looks like ATL is up but Tampa is back down.
Posted by J0o0j, 04-19-2013, 06:06 PM
twitte - @SagoNetworks

Posted by J0o0j, 04-19-2013, 06:09 PM

Quote:

Originally Posted by ath1234

We have just been told that they have found and are working to replace a failed part of their core routing system. Looks like ATL is up but Tampa is back down.

How long does it take to work on reform
Posted by ath1234, 04-19-2013, 06:10 PM

Quote:

Originally Posted by J0o0j

How long does it take to work on reform

What we were told is that they just found the problem recently, and are now working on repairing/replacing something.
Posted by ath1234, 04-19-2013, 06:13 PM
I don't understand why they can't do a better job of updating people. We've been sending hourly updates to our customers since this morning.
Posted by J0o0j, 04-19-2013, 06:16 PM

Quote:

Originally Posted by ath1234

I don't understand why they can't do a better job of updating people. We've been sending hourly updates to our customers since this morning.

I'm so far did not I receive notice of the company
Posted by ath1234, 04-19-2013, 06:17 PM

Quote:

Originally Posted by ath1234

I don't understand why they can't do a better job of updating people. We've been sending hourly updates to our customers since this morning.

I have had to call them repeatedly to get answers.
Posted by J0o0j, 04-19-2013, 06:19 PM
I'm from Kuwait
Posted by avengerx, 04-19-2013, 06:45 PM

Quote:

Originally Posted by jiashanuk

down for 2-3 hours from UK, their network is not reliable this whole week.

I tried to call them, no one answered the phone.

Friends, sago net has not really left for side at all, they are giving some feedbacks but nowadays we have lots of communication channels and every of these have several ways of communication and so on...

Well, we get some people aid with very updated phonecall information but sago net seems to be pretty active on their facebook (don't know twitter) profile at facebook.com / SAGONETWORKS?fref=ts

So I suggest everyone who owns a facebook profile to follow this one in hopes to get official updates.

By myself, I opened a ticket (when I found the portal up) and asked them to give feedback once the server is down. Every minute the ticket has without an answer is counting as downtime, officially, now. So I suggest you to do the same, so we both are notified via the ticket system once its alright, and we can have records of downtime.

But it's strange the speech bouncing between DDoS attacks and faulty routers. Its been days since I noticed loss from the first hops on sago net, and at the same time I am keeping an open ticket about these problems with them.
Posted by jiashanuk, 04-19-2013, 07:34 PM
up for a few minutes, down again!!!
Posted by akses, 04-19-2013, 07:53 PM
Come on sago.. Some of us actually have businesses to run off your servers. Adwords is running and if you don't inform your customers that your servers are down, I'm losing a **** ton of money on empty clicks.

At least send a notice to your customers that something is wrong! Oh, I forgot, everything you guys do is based on on your main servers. So when your servers do down, everything does down, including phone lines, customer support portal, everything, it's like you've cut yourself off from the rest of the world.

Get your micky mouse operation together!

Your servers have been down most of the day.. I've never seen a server down for the good portion of the day.
Posted by ath1234, 04-19-2013, 08:00 PM

Quote:

Originally Posted by akses

Come on sago.. Some of us actually have businesses to run off your servers. Adwords is running and if you don't inform your customers that your servers are down, I'm losing a **** ton of money on empty clicks.

At least send a notice to your customers that something is wrong! Oh, I forgot, everything you guys do is based on on your main servers. So when your servers do down, everything does down, including phone lines, customer support portal, everything, it's like you've cut yourself off from the rest of the world.

Get your micky mouse operation together!

Your servers have been down most of the day.. I've never seen a server down for the good portion of the day.

Their servers actually haven't been down, it's been connectivity due to router issues. Still sucks but the servers are not the issue.
Posted by Antonimo, 04-19-2013, 08:01 PM
Too bad they use facebook as their primary source of communication, some people, like me, do not have/want a facebook account and everyone knows fb is a closed garden. Who would be the genius who thought of using a closed garden as a feedback tool.... especially being a hosting/Intenet company, that is simply moronic... I am moving out of sago as soon as I can, that's a fact....
Posted by jiashanuk, 04-19-2013, 08:06 PM
do the same.
Posted by J0o0j, 04-19-2013, 08:07 PM
me to

I am moving out of sago as soon
Posted by akses, 04-19-2013, 08:13 PM
Sago,

Facebook and Twitter as communication tools? Do I look like a hipster to you?!?!

I'm trying to run a business on your servers. Do you think I have time to use facebook or twitter?

How about sending an email blast to your customer base instead!
Posted by avengerx, 04-19-2013, 08:20 PM

Quote:

Originally Posted by Antonimo

Too bad they use facebook as their primary source of communication, some people, like me, do not have/want a facebook account and everyone knows fb is a closed garden. Who would be the genius who thought of using a closed garden as a feedback tool.... especially being a hosting/Intenet company, that is simply moronic... I am moving out of sago as soon as I can, that's a fact....

antonimo, I would create a facebook account right now just to read this:
facebook.com/ dejan.protich?hc_location=timeline

turned me uneasy... to what conclusion can people get?

p.s.: I wonder how many posts are needed before I complete 5 posts!
Posted by ath1234, 04-20-2013, 01:28 PM
Here's the official outage report:

Incident and customers affected:
At 10:28am Friday, April 19, 2013. Sago experienced a throughput capacity deterioration and failure in general network services affecting Georgia and Florida customers. Sagos IP engineering team identified a failing core router in our Tampa, FL facility. The failing equipment was spawning network loops in the core routing tables which caused intermittent service to no service for customer in Sagos Southeast Network. The core routing failed over to our backup core router in the Tampa, FL Datacenter. Physical Media was required to be manually moved over to the backup core to support the redistribution of the network. Sagos IP engineering team and outside vendors was able to complete physical media move and stabilize the operations by 5:03pm. There were reports of isolated issues up till around 8:10pm Friday, April 19, 2013.

Incident response and corrections:
Our network engineers were able to isolate the offending routing system that was causing the up and down results in services and migrate off of the faulty equipment. Human intervention from Sagos IP engineering team was required to remove the offending equipment. Sago has acquired a replacement core router to be installed by Tuesday April 23, 2013 between 12:01am and 5:00am.