[MERGED] Sagonet/Allmanaged Down

Portal Home > Knowledgebase > Articles Database > [MERGED] Sagonet/Allmanaged Down

Posted by HD Fanatic, 11-11-2005, 04:27 PM
I can't reach my server at the Sago datacenter. Sagonet and allmanaged.com doesn't work for me either. I tested those domains with dnsstuff with the same problem. Anyone know what's wrong?
Posted by gimme5, 11-11-2005, 04:33 PM
Same problem here for a very short time. Everything is now back up.
Posted by Marty, 11-11-2005, 04:35 PM
Yea, it was less than 6 minutes.
Posted by RyanD, 11-12-2005, 03:52 PM
this is getting to be a regular occurence...
Posted by HD Fanatic, 11-12-2005, 08:42 PM
They did say that additional uplinks will be added soon. mci/uunet, savvis, or cogent. Only time will tell but they definitely need more carriers. Level3 is not enough.
Posted by sprintserve, 11-13-2005, 02:04 AM
This was a scheduled upgrade. I received this email on Tue/Wed: Dear Valued Customer, The following maintenance has been scheduled. Sago Networks router code maintenance notification - Tampa, Fl facility. Dates: November 13th 2005 Times: 12:00AM to 6:00AM Maintenance: Core router code upgrades and BGP peer maintenance. Impact: Major. Customers will experience several instances of network unavailability in durations of 5 to 15 minutes per occurrence. Sago Networks maintenance ID: COP-63705-228 If you have any questions, feel free to call Sago Networks Network Operations Center at 866-510-4000 and refer to the above maintenance ID. Regards, IP Engineering Sago Networks
Posted by sprintserve, 11-13-2005, 02:06 AM
They have more than Level 3, and yes, new carriers is soon, but I can't say who as I am sworn to secrecy under a death pact
Posted by Marty, 11-13-2005, 02:45 AM
mmmm, Yea, I got that email as well, however this thread started by reporting an outage on Nov. 11 not Nov 13.
Posted by sprintserve, 11-13-2005, 02:48 AM
Ah. Oh yes. We did have a 5 minute issue a couple of days back. miss the dates on the opening, and only saw the date on the last post.
Posted by kneadingu, 11-13-2005, 06:32 AM
can you reach them now? I can't
Posted by james007, 11-13-2005, 07:18 AM
For me, down two, more than 20 servers.
Posted by Marty, 11-13-2005, 07:21 AM
I am down now as well, and what I am seeing is that my bandwidth is flipping back and forth between Savvis and Williams, so it is obvious that they are adding Savvis to the mix, but I do wish this would get fixed soon.
Posted by sprintserve, 11-13-2005, 07:24 AM
Yup down and flipping. One of my admins just called and they said the "maintanance" will take 1 more hour. So it's going to take 1 1/2 more hours than scheduled. Hopefully, that estimate is accurate.
Posted by sprintserve, 11-13-2005, 07:25 AM
Yup Savvis it is. It is only meant to go fully live in a couple of days, I guess they just got ahead of schedule.
Posted by swijaya0101, 11-13-2005, 08:09 AM
I am still unable to access sagonet site + my servers from Australia.
Posted by Peter Avey, 11-13-2005, 08:17 AM
seems to be abig problem with the datacentre.. im sorry to hear it boys
Posted by Marty, 11-13-2005, 08:24 AM
5 minutes to go and still things don't look too good.
Posted by TrevorN, 11-13-2005, 08:40 AM
I guess i will be the first one to say it's up and has been up for me since 6:52 this morning.
Posted by swijaya0101, 11-13-2005, 08:52 AM
Trevor, My partner is in Texas and he told me that he can access sagonet + servers without any problem. Unforunately, I am still unable to access the servers from Australia.
Posted by jokaroo, 11-13-2005, 09:15 AM
Access to all servers on Sago in Canada are not working. We've experienced problems since 2AM till now. I hope there is downtime compensation because this is taking forever.
Posted by sprintserve, 11-13-2005, 09:15 AM
They are still flipping quite a bit.
Posted by WCSWEB, 11-13-2005, 09:16 AM
They are still down 2 hours beyond their scheduled maintenance.
Posted by TrevorN, 11-13-2005, 09:20 AM
From what you guys are saying it sounds as though i'm lucky, since i still have complete access to mine.
Posted by Marty, 11-13-2005, 09:35 AM
Still up and down here.
Posted by swijaya0101, 11-13-2005, 09:42 AM
yeah, up and down from here as well.
Posted by RyanD, 11-13-2005, 09:56 AM
*sigh* again still flopping like a fish out of water
Posted by RyanD, 11-13-2005, 10:07 AM
now it's just up with 50% loss
Posted by jokaroo, 11-13-2005, 10:10 AM
Looks like I will be removing this server as well. This downtime is pathetic. Unless they offer complete compensation, I'm a goner.
Posted by RyanD, 11-13-2005, 10:13 AM
I've got ~20 servers there and it had been very reliable for the last year - year and a half. Lately the network has turned to trash. I've had more downtime in the last month than I have in he previous 2 years.
Posted by Xzyte, 11-13-2005, 10:24 AM
same here, 50% loss on 4.79.144.0 coming in over Level 3.
Posted by sprintserve, 11-13-2005, 10:34 AM
It's mostly due to Level 3's sudden dictatorial instincts. Adding a new provider is the right step, but the implementation leaves much to be desired.
Posted by TrevorN, 11-13-2005, 10:40 AM
I'm having about 27% packetloss on 4.79.172.2 using level3 and 65.77.96.150 on wiltel Last edited by TrevorN; 11-13-2005 at 10:47 AM.
Posted by kneadingu, 11-13-2005, 10:48 AM
I'm still seeing as much as 75% packet loss and it damn near 1000 EST.
Posted by jokaroo, 11-13-2005, 10:52 AM
They lost my business. I already purchased another server at a different provider.
Posted by sprintserve, 11-13-2005, 11:12 AM
sometimes I wish it was so simple
Posted by RyanD, 11-13-2005, 11:14 AM
*sigh* unhappy customers = unhappy me and unhappy me = business going elsewhere
Posted by phactor, 11-13-2005, 11:33 AM
any new on this? I'm still having a lot of PL. Anyone called them?
Posted by sprintserve, 11-13-2005, 11:33 AM
Ya. Our admins call every hour. They have no information to provide (or at least those picking up the phones) beyond the fact that they have no ETA.
Posted by phactor, 11-13-2005, 11:40 AM
great, hope they give us a good explanation!
Posted by TrevorN, 11-13-2005, 11:51 AM
My connection is beggining to go in and out now.
Posted by RyanD, 11-13-2005, 12:02 PM
1 1 ms 1 ms 1 ms 192.168.0.1 2 6 ms 9 ms 7 ms 73.60.136.1 3 6 ms 8 ms 7 ms 68.86.109.225 4 6 ms 7 ms 22 ms 68.86.106.222 5 7 ms 6 ms 9 ms 68.86.106.226 6 9 ms 7 ms 12 ms 68.86.106.142 7 7 ms 9 ms 13 ms 68.86.106.146 8 8 ms 8 ms 9 ms 68.86.106.150 9 8 ms 7 ms 9 ms 68.86.106.154 10 9 ms 7 ms 9 ms 68.86.106.158 11 12.124.64.73 reports: Destination host unreachable. NICE, they've totally fallen off the map as far as comcast is concerned
Posted by SeeServers, 11-13-2005, 12:04 PM
Same here, fallen off the map. * bellsouth and RoadRunner
Posted by phactor, 11-13-2005, 12:06 PM
why they dont come here to say something?
Posted by RyanD, 11-13-2005, 12:08 PM
still "We have no eta at this time" I asked, well realistically what does that mean... Today, Tomorrow, Next Week, 6 months from now? To which they responded "I would think sometime within the next hour" This isn't very comforting. To this point I've been a pretty happy customer of Sago (and a reseller too!) the recent bouts of network outages are doing serious harm to my business and this is the final straw... Time to start shopping
Posted by oziris, 11-13-2005, 12:19 PM
Amazing...my server is in the process of moving to a new home. Started the process last night. ANd then this. Now I have some sites on the new server and some nowhere cause Sagonet is down once again. Just confirms that I am making good decision.
Posted by miracle, 11-13-2005, 12:23 PM
still down to me. I waited for a very long time.
Posted by RyanD, 11-13-2005, 12:23 PM
This DOES NOT mean please PM me with offers.
Posted by SeeServers, 11-13-2005, 12:28 PM
just report them, and then WHT is a better place..
Posted by Sohan, 11-13-2005, 12:28 PM
Are they having network problems? The site is loading very slowly, and it keeps terminating the conection.
Posted by WCSWEB, 11-13-2005, 12:34 PM
Can't believe it 9.5 hours down! And as usual no response and no updates. I wonder where is Matt and Jason?
Posted by miracle, 11-13-2005, 12:46 PM
yeah, we need update. I believed what they said, "Customers will experience several instances of network unavailability in durations of 5 to 15 minutes per occurrence." so I didn't notice my users. but my server is down for many hours. when will everything be ok?
Posted by animelab, 11-13-2005, 12:47 PM
Yikes! Wonder what happened.
Posted by sprintserve, 11-13-2005, 12:48 PM
They now say that the adding of the new provider has caused internal routing problems and they are still working on it (but no ETA)
Posted by miracle, 11-13-2005, 12:58 PM
something wrong after reach "4.79.144.2 unknown.level3.net" here, "Router did not respond".
Posted by dreaz, 11-13-2005, 01:01 PM
Great, still down, now they won't even reply e-mails. It was supposed to be 6 hours of occasional 15 minutes downtimes, now we'll be looking at 6 hours of complete downtime. Another great move by sago
Posted by Xzyte, 11-13-2005, 01:03 PM
same here. IP packets are being lost past network "Level 3 Communications, Inc. LVLT-ORG-4-8" at hop 12. The route has changed so the next network at hop 13 is unknown.
Posted by TrevorN, 11-13-2005, 01:05 PM
If you are unable to reach the network most likely e-mail wouldn't work. But yes, I not able to access stuff anymore.
Posted by sprintserve, 11-13-2005, 01:05 PM
Your emails most likely aren't going to be reaching them
Posted by dreaz, 11-13-2005, 01:08 PM
They actually are, they replied to me 1 hour ago and it was still down. But still, they could at least inform customers about what's going on during this kind of downtime.
Posted by Vaya, 11-13-2005, 01:10 PM
This is a complete joke now. I checked my server at 10:00 GMT and it was down, but since downtime was expected anyway I went out with the assumption it would be sorted within the given timeframe. It's now 17:10 and I've got in to find all my sites and Sagonet.com are unreachable.
Posted by Chuggles, 11-13-2005, 01:10 PM
They really need an off-network location where they can post updates and keep clients informed. No ETA is not good...that could be all day.
Posted by TrevorN, 11-13-2005, 01:12 PM
I am getting like 30% packetloss now.
Posted by bestprobe, 11-13-2005, 01:13 PM
Same here... My Server has down for 1 and half hour already
Posted by miracle, 11-13-2005, 01:18 PM
now it's "65.77.96.150, unknown.wcg.net", same problem.
Posted by WCSWEB, 11-13-2005, 01:19 PM
Looks like part of their network is coming up. I can access one of our servers.
Posted by sprintserve, 11-13-2005, 01:30 PM
25% of our servers have popped back into service. definitely better than nothing.
Posted by Xzyte, 11-13-2005, 01:33 PM
Still the same form here, nothing will go past 65.110.32.241 | unknown.sagonet.net. What I think is most disappointing is that nobody of Sago even bothers to post here. This doesn't look very professional at all.
Posted by miracle, 11-13-2005, 01:35 PM
"65.110.32.241, unknown.sagonet.net, Router did not respond" now.
Posted by kneadingu, 11-13-2005, 01:41 PM
lol ... first it was eight characters and now it's ten.
Posted by Gemhdar, 11-13-2005, 01:43 PM
Brief Update: The planned network upgrade has gone obviously over the allotted timeframe we had slated for it to be done in. However our engineers are working with the manufacturer of the products in question to get the resolution done as soon as possible. Matt will be on the thread soon to give a more in depth update, however please be assured we are working to resolve this as soon as possible.
Posted by RyanD, 11-13-2005, 01:48 PM
well atleast it's not ENTIRELY down anymore, I've got 80% packet loss... I guess 20% is better than nothing
Posted by jokaroo, 11-13-2005, 01:54 PM
100% packet loss here. Unreal... Does anyone know if they have downtime compensation?
Posted by dreaz, 11-13-2005, 01:58 PM
All my servers are up now, finally. Hope it stays like this.
Posted by Xzyte, 11-13-2005, 02:00 PM
Mine are back up form here with 50-80% packet lost over Level 3
Posted by TrevorN, 11-13-2005, 02:02 PM
I'm still hanging out with 30% over level 3.
Posted by RyanD, 11-13-2005, 02:07 PM
PL is down to 40% for the majority of our boxes... still unacceptable
Posted by Justin, 11-13-2005, 02:11 PM
I'm seeing 45-50% loss and higher at bursts (up to 60 even) to their main website even. Hopefully they will have this sorted out very soon.
Posted by emills01, 11-13-2005, 02:12 PM
Glad I'm not the only one seeing problems the last few weeks with intermittent outages. Every time I submitted a support ticket about it they told me everything was fine and to check and see if it was on my end...not good.
Posted by WCSWEB, 11-13-2005, 02:32 PM
11 hours and counting... At least is not Monday morning!!! I will like to hear about this one!!
Posted by sagonet, 11-13-2005, 02:33 PM
Hello Everyone, Just a quick update as very specific details will follow. The upgrade this morning was a mass-software upgrade in preparation for some large additions to our network and to maintain some general housekeeping. The upgrade path was pretty clear cut and, as usual, verified by a third party. The initial upgrade was successful, but around 3am certain routing loops causes unusually high processor utilization. This issue result in our IP Engineering department's decision to back out of the upgrade. The process of downgrading the software on our core switches was impeded by a version mismatch between previously existing firware on external routing cards and the management software on the switch itself. This left us in a limbo between two versions and an inability to reimplement the original code. We are currently waiting on a software patch to be written by the manufacturer to enable to either go full forward or fall back on the old version. Right now, the most obvious symptom of this condition is packet loss. We are working with the manufacture, who is onsite, to resolve this. A full exmplanation will be posted shortly. More details will be distributed as they become available. We apologize for this incident.
Posted by Justin, 11-13-2005, 02:36 PM
Will there be any SLA compensation for this outage? Hopefully this is resolved completely soon (presuming there is no eta yet? Is there any backup plan in place of an older core to fall back to besides the existing one? EDIT: just had yet another complete drop and now seeing 70% PL again Last edited by Justin; 11-13-2005 at 02:40 PM.
Posted by Compworld, 11-13-2005, 02:37 PM
Any idea on when things should be back to normal?
Posted by Def, 11-13-2005, 02:55 PM
It's always something with them.
Posted by codeguru, 11-13-2005, 03:01 PM
For me --- sagonet.com ping statistics --- 6 packets transmitted, 0 received, 100% packet loss, time 5087ms
Posted by RyanD, 11-13-2005, 03:01 PM
lately thats been true... aren't there any fail-over or hot-spare equipment in place that can be turned up to eliminate this ?
Posted by codeguru, 11-13-2005, 03:08 PM
why is that there is no precuation measures or replacement measures when such a disaster occurs ???
Posted by kneadingu, 11-13-2005, 03:10 PM
People don't plan to fail they fail to plan ...
Posted by Compworld, 11-13-2005, 03:12 PM
Surpised they do not have a secondary host backup to switch to. My servers at other hosts have this. Not sure why Sago didn't consider any backup plans. :\
Posted by RyanD, 11-13-2005, 03:32 PM
Sago's site states a 99.95% Uptime Guarantee that would put it at 262.8 minutes of downtime per year we've more than exceeded that today.
Posted by Justin, 11-13-2005, 03:37 PM
Hopefully they will be honoring this for everyone that requests an SLA credit unlike some providers who fail to do so.
Posted by HD Fanatic, 11-13-2005, 03:38 PM
I wish Sago would tell us that they are adding more carriers instead of saying router maintenance. So, Savvis is officially part of the mix? My server is very slow atm Last edited by HD Fanatic; 11-13-2005 at 03:52 PM.
Posted by WCSWEB, 11-13-2005, 03:48 PM
Our servers there have been down since 2:30AM at times they will come up with a high % of packet loss. This is exactly what happened last time they did a router upgrade. I really wish that they will plan out their upgrades and have some sort of BACK UP!!!!!!!!! in place. Last edited by WCSWEB; 11-13-2005 at 03:54 PM.
Posted by Chuggles, 11-13-2005, 03:49 PM
Um...my downtime is a lot longer than 2 hours. Consider yourself lucky if you only had two hours. I personally would not be this irritated over 2 hours.
Posted by Vaya, 11-13-2005, 03:53 PM
I'll second that. My server's been totally inaccessible for a good 5 hours and has been unusably slow since then. I'm still getting up to 70% packet loss at times.
Posted by HD Fanatic, 11-13-2005, 03:54 PM
Actually there was 2 instances of downtime over 2 hours and 1 instance over over 1 hour.
Posted by codeguru, 11-13-2005, 04:15 PM
--- sagonet.com ping statistics --- 11 packets transmitted, 4 received, 63% packet loss, time 15646ms rtt min/avg/max/mdev = 41.074/44.509/53.944/5.452 ms, pipe 2
Posted by Compworld, 11-13-2005, 04:20 PM
I couldn't care less about the credit. I just want my server to work! Is that too much to ask for?
Posted by Chuggles, 11-13-2005, 04:38 PM
Yes, it seems like today that is too much to ask for. I am concerned because today is Sunday - the day when tech support is hard to come by. In the this thread Sago said someone is on site. In a thread on their forums, they said they had someone on the phone. So is the person who can fix this at the data center, or on the phone. Compworld - some people have clients requesting refunds. So yeah, Sago should compensate the client or reseller and the reseller compensate their client.
Posted by HD Fanatic, 11-13-2005, 04:40 PM
Server is down again there really needs to be some compensation of some sort. Sago really disappointed me this time after being a client of theirs for over 9 months
Posted by RyanD, 11-13-2005, 04:43 PM
yup totally out again, hopefully this is them applying the "patch" and restarting the routing gear.
Posted by Justin, 11-13-2005, 04:47 PM
Been with them myself for over a year also. Seriously they need to make good on their SLA on this though, it's getting quite ridiculous at this point. Just noticed major loss of around 50-70% right before the last drop so who knows at this point what is going on. EDIT: seeing no route at all at present (bgp down more than likely), so it is looking like a router reboot (maybe they're cycling all upstream networking equipment?), hopefully what they're doing solves this finally. Last edited by Justin; 11-13-2005 at 04:51 PM.
Posted by tmwes, 11-13-2005, 04:53 PM
I have a dedicated server with Sago via Beachcomber, who unbeknownst to me stopped offering Sago servers quite a while ago. I'm probably going to take Beachcomber's advice when they present us with options of other networks to switch to. What a major pain, though...
Posted by swijaya0101, 11-13-2005, 04:58 PM
100% packet loss for the last 5 hours.
Posted by SeeServers, 11-13-2005, 05:00 PM
I don't know about 100%for 5 hours, it has been up for a fe seconds every now and then. Sagonet - Uptime today 0.0007% ...... arugh
Posted by RyanD, 11-13-2005, 05:04 PM
well we're back up and PL is no better
Posted by swijaya0101, 11-13-2005, 05:17 PM
I've just received an email. ETA the next 2-3 hours (sales reply). So, I believe it will most likely be running as per normal by Monday morning.
Posted by emmons, 11-13-2005, 05:21 PM
Ping statistics for 63.246.145.180: Packets: Sent = 271, Received = 144, Lost = 127 (47% loss), Approximate round trip times in milli-seconds: Minimum = 59ms, Maximum = 101ms, Average = 68ms 47% loss in the middle of the afternoon. Unacceptable. *sigh*
Posted by HD Fanatic, 11-13-2005, 05:27 PM
of all days, why did they choose a Sunday to perform this maintenance? I hope this is covered by the SLA, this is really unacceptable.
Posted by Justin, 11-13-2005, 05:34 PM
No sooner did things come back than dropped yet again: hopefully this yoyo routine will stop soon enough.
Posted by Xzyte, 11-13-2005, 05:35 PM
There we go agin, 100% packet loss at 4.79.144.2
Posted by Brian@HostXpro, 11-13-2005, 05:36 PM
I am very disappointed with the service we have been having with Sago Networks latly. We are now hosting over 100 websites with a reseller with Sago Network. We used Sago Network in the past for a dedicated server but was forced to change to a new company due to the reason we were issuing support tickets almost daily for issues. For dedicated service, we have choosen GigeServers. I will bet we havn't issued 5 tickets since we moved to them over a year ago. The DDOS protection filtering they offer is amazing and havn't seen anything like it. Hope our webpages are back to normal soon. Concerned Sago Networks Customer.
Posted by Justin, 11-13-2005, 05:39 PM
Seeing a lovely level3 routing loop now in tampa: Seems like they're once more trying to route away from level3 but in doing so causing an almost endless loop (took a trace awhile to find the route properly even). Here's to sagonet getting this sorted asap
Posted by HD Fanatic, 11-13-2005, 05:41 PM
so they're going to use savvis as their primary connection instead of level3 now? I'm seeing 75% packet loss
Posted by Justin, 11-13-2005, 05:59 PM
Which transit is being used at this point would really be irrelevant due to the fact that this heavy loss is actually being caused by their core router's failed upgrade (and the subsequent issue of their being unable to revert to their original settings). In any case hopefully they get this software issue resolved finally. There really isn't much point behind us continuing to speculate though as it isn't going to get their issue resolved any faster or slower for that matter. Hopefully one of their reps will post again soon with an update on what exactly is going on causing these intermittent complete drops, as some of them result it total bgp reset(s).
Posted by r2tech, 11-13-2005, 06:12 PM
Hi, Not sure if this is already been discussed. If so, please point me in the right direction....BUT My server at Sagonet seems to be down. The server is actually with Beachcomber, but hosted at Sagonet. Beachcomber are telling me its a problem at the NOC. An upgrade gone wrong. Anyone know any more info on this? Don't know for sure, but seems to be high packet loss. Cheers
Posted by yourmp3, 11-13-2005, 06:24 PM
Hi ... I just bought a dedicated server with allmanaged and apart for a few problems at installaion everything went smooth from the start. Now all of a sudden I get slowliness accessing my website and also 50% packet lost when I try to ping my domain. Also, I noticed Sagonet.Com and Allmanaged.Com websites seem slow too. When I tried tracerouting it stopped around Sagonet. Is there a problem with sagonet DC as of the past day or maybe it's just my ISP or something? Thanks - Yuval
Posted by SoftWareRevue, 11-13-2005, 06:30 PM
/more merging.
Posted by hostxpro, 11-13-2005, 06:31 PM
I agree with what you have said Brian, its not nice when you dont even receive a notification email informing you of what is going on. Also I submitted a ticket around 8hrs ago asking what was going on and I havnt even received one reply (it was sent from a non sago hosted email @). Very dissapointing AllManaged/Sago - you have let yourself down ~ Matt
Posted by PhMatt, 11-13-2005, 06:46 PM
I have to admit, I haven't gone through this entire thread yet, but wanted to let everyone know, everything is being done to correct the problems with the router upgrades, the next ETA is expected in approximately 30 minutes. I was taking care of some personal matters today, but was kept informed of things that being the upgrades did not go accordingly, and Foundry was onsite working with out network admins on the upgrade issues. All clients were notified early last week by e-mailing through the billing system as to the upgrades, and posted on our network status page as well. Though this was certainly a disappointing upgrade, the necessity of it was there, and needed to be done. We'll have things resolved as quickly as possible, and when I have further updates, I or Adam will post them online as quickly as possible. Thank you,
Posted by swijaya0101, 11-13-2005, 06:50 PM
yes, email was sent last week. however, it is quite dissapointing for such company like sagonet that you dont have a backup plan when things go wrong.
Posted by gmunday, 11-13-2005, 06:55 PM
Well i guess u dont need to hear about another unhappy customer. I have had family problems the past 2 weeks, and havent been about much, im not too happy about my customers calling me on my cell, complaining about their sites not working, as i have not been about, i did not have any idea about the planned maintanence this weekend. I actually get my sagonet server from a resller, and i am honestly thinking about cancelling. It looks like my box has been down for 9.5 hours now. **Sighs** Thankfully i do have other servers elsewere. Not everyone does.
Posted by Chuggles, 11-13-2005, 06:57 PM
The server (for me) have been unreachable (or at a crawl) for 12 hours. When they come up I'm going to be so happy it's up I might throw a party. See, unlike Matt I had to babysit email all day. I could not take care of the personal things I was supposed to do today. So my question I suppose is what is being done to prevent this from happening in the future? Because this situation will not be acceptable - if I decide to stay that is. And no, I don't expect an answer now (work on the servers dang it!) but I do expect someone from Sago (after the servers are up and everyone is fresh eyed and bushy tailed) to go through this thread and address every concern in it.
Posted by cunni, 11-13-2005, 07:13 PM
ditto, Im not best pleased either... I have been with sago for around 1 year and 4 months, and this is a real shame... My server is up and down like a yoyo at the minute, and when it is up its as slow as a 128k uplink. Im not liking how this has gone down at all. But problems do occur, and thats the way its gone down. Just please keep us "The customer" updated matt. Thanks.
Posted by Tinozaure, 11-13-2005, 07:18 PM
Ok, Sagonet is definitely taking us for loosy *** holes. My private ADSL line had less downtime than SAGONET core routers in the last 6 months. They speak about Gigabits of lines but they are not able to have redundancy when upgrading a router. They upgrade a router with help from Foundry team and it gives us a down time of more than 6 hours. Who are the guys that think than this is professional ? Do they really think at Sago that we will continue to pay for that level (3) of service !!! I AM REALLY PISSED OFF !!!! I WILL SERIOUSLY CONSIDER CANCELING ALL MY SERVERS AND GOING ANYWHERE ELSE THAN THEM !
Posted by nexcess.net, 11-13-2005, 07:23 PM
I think you meant "lousy" Tinozaure . I'm now seeing 0% packet loss from some locations. So it looks like things are normalizing. Still getting 30+% from others.
Posted by web_hosting4u, 11-13-2005, 07:26 PM
This has been bad, and I finally returned to the home office to see what all the news was, and on Sago's site, not much about it, except from really ticked off clients. I have been trying to keep up with this outage on cell phone all day, and really cannot believe what is going on. Luckily it is Sunday and I am led to believe that all my clients are not using or checking their websites. I have only received a few calls and a bunch of IM's... (can't get e-mail - ~snicker~). Anyway, this is truly sad...until today, I thought I had it good with Sago (AllManaged). After today, I am not so sure!
Posted by pietjepuk, 11-13-2005, 07:27 PM
This is intolerable, I will switch over to a new host.
Posted by kebirhost, 11-13-2005, 07:33 PM
Hello Everybody, I am a dedicated reseller of AllManaged for over than 10 months. All hosting companies have some wrongs and disadvantages. But the most important thing is the percantage of the advantages. SAGO is a big and reputable firm. We cannot say SAGO is a bad hosting firm by refering this outage. I am sure that they are working on the issue and it will be solved within 1-2 hours. If we want to work with SAGO, we should thrust and try to support in hard days. Yes i know we are customers but we shouldn't forget SAGO's advantages and good parts. Thanks, Melih Sarıoğlu www.kebirhost.net
Posted by RyanD, 11-13-2005, 07:36 PM
Whats this inside information you have that says 1-2 hours? this has been ongoing for 12+ hours.
Posted by miracle, 11-13-2005, 07:40 PM
still very high packet loss here, and extremely slow speed.
Posted by web_hosting4u, 11-13-2005, 07:42 PM
The ping that makes it looks great....then I get Timed Out! I keep cheering when I get 4 pings in a row...sad..... Stopped my pings....50% packet loss....(and not just 4)
Posted by diginux, 11-13-2005, 07:44 PM
If anyone here knows a comparable service to SAGO that offers comparable servers, I think it would be smart of them to offer a special deal to current SAGO users to switch over.
Posted by kebirhost, 11-13-2005, 07:45 PM
Hello, I do not have any inside information. But as you see all people say too many things about the issue without any inside information. Today is sunday and some time a disaster may be occur. You cannot prevent this. I do not say SAGO has no wrong, they have wrongs but i want to thrust and support them in this kind of hard day. I am sure they will improve their communications with their customers after this outage because this event will learn somethings. Thanks, Melih Sarıoğlu
Posted by Justin, 11-13-2005, 07:46 PM
kebirhost, While I do see your point of view you also need to take into account that such outages that resellers have no control over make them look bad to their clients. So, unfortunately in this case the reseller is taking the slack that the datacenter is dishing out with no recourse of compensation on the issue (while sago has an SLA I am sure that the sla has an excerpt regarding planned maintenance windows, thus this outage might not be covered by their SLA even though it extended well beyond the window alotted). I've had a server with sago for a bit over a year now and well I won't comment on things that don't belong publicly stated at this point (but regardless it has been a 50/50 experience at times).
Posted by Justin, 11-13-2005, 07:48 PM
diginux, There are many many server hosting providers, just have a look through the offers forum for awhile. Know that if anyone pm's you with spam take it with a grain of salt (and I'd recommend reporting it to the mods also) as that's against wht forum rules.
Posted by Bacus, 11-13-2005, 07:49 PM
Been with Sago for six months, my servers have had the same issues today, but I've been in worse places, and I can say support and service at Sago has been wonderful, we all have issues once in a while, I'm pissed but I am sure they are working hard on fixing this ASAP. Patience.....
Posted by web_hosting4u, 11-13-2005, 07:50 PM
I have been with Sago/All Managed for about 8 months. Really I am with AllManaged, however, Sago is the overall owner. One thing I will say, is AllManaged.com has poor communication with their customers. When I need answers, I call the Toll Free # and speak with someone at SagoNet....same # on both websites!
Posted by pietjepuk, 11-13-2005, 07:52 PM
It's not like they're doing us a favor by keeping their network up, we're paying for it damnit. So how about cutting the ******** about how we need to support Sagonet, do you even consider some are losing quite some money because of this downtime?
Posted by emmons, 11-13-2005, 07:53 PM
Not so reputable any more.
Posted by Chuggles, 11-13-2005, 07:55 PM
If Sago is the type of company I hope they are, they will offer the credit so resellers can funnel the credit down to their clients. It's the right thing to do. If they do not, then that leaves the door open for this to happen again, with no recourse. How can any host explain that to their clients? I am not saying they meant for this to happen and that they did not do everything possible to fix it. I am saying take responsibility for the mistake and do the right thing. It's not about the money it's about respect - that they respect and value their clients enough to do it because they royally screwed up.
Posted by Tinozaure, 11-13-2005, 07:55 PM
In my world, Sunday or Monday, redundancy and offline tests are the key word in a production solution. Communication or not is not the problem. 9+ hours of downtime is the problem. Sagonet never communicate on their network problems. That is a fact, I deal with it.
Posted by ahostinginc, 11-13-2005, 07:56 PM
I was happy to resell Sago servers. All clients are asking about cancellation i will move their servers to another datacenter in Orlando that i resell their products with better service and value. Sorry Sago but clients doesnt want to understand...
Posted by pietjepuk, 11-13-2005, 07:59 PM
Sagonet support is like **** anyways now for quite some time, today's event is a good motivation to make the switch to another host.
Posted by PhMatt, 11-13-2005, 08:01 PM
Latest updates are that Foundry believes to have identified the cause of the software issues, and will be correcting them. Though this certainly has not gone as planned, we will post a full update as to the process, problems, and resolutions that we have undertaken with this upgrade. The matter will be resolved as quickly as possible, technicians have been working on things round the clock, and will continue to do so. This is not something we are taking lightly, and have deployed every available resource to fix things as quickly as possible. Thank you,
Posted by kebirhost, 11-13-2005, 08:01 PM
Hello Emmons, Please just think that you made a mistake and all you did nice things forgetten and all people started to change their thoughts about you in a bad way. What will you do then? You cannot correct your wrong, you go on with your wrong. I am professional and have more than 500 customers. I was affected from this event but i came to this point with the help of Allmanaged. I hope i can explain my point of view. Thanks, Melih Sarioglu
Posted by tmwes, 11-13-2005, 08:02 PM
as I said before, I've been a Sago customer for a little under a year, but a Beachcomber customer for much longer than that. Beachcomber has earned my trust and given me reason to keep my business with them time and time again. Last edited by writespeak; 11-13-2005 at 09:11 PM. Reason: Edited by request
Posted by RyanD, 11-13-2005, 08:03 PM
does this still mean no ETA?
Posted by pietjepuk, 11-13-2005, 08:09 PM
I'm not quite understanding what you're trying to say there, kebirhost. The fact is that even though Sagonet/AllManaged did not mean this to happen, they still have to take responsibility for it. The support this company has been giving me is horrible the least, and I bet many others can say the same. It's their attitude that gotten themselves in this situation the first place.
Posted by pietjepuk, 11-13-2005, 08:12 PM
How about cutting all the ******** there Matt and actually start to answer some questions asked here, in what way will Sagonet/AllManaged customers get compensated?
Posted by web_hosting4u, 11-13-2005, 08:16 PM
I am not trying to defend Sago, but I have one major concern, is slowness actually downtime?
Posted by n00bh0st, 11-13-2005, 08:18 PM
There support has never been good. They take an hour to get to your servers for a simple reboot. They get smart on the phone. They never have an answer for any of your questions. You can see them eating food and doing nothing via their datacenter cameras. Support there is a problem, if it wasn;t for the prices they would ahve no redeeming qualities.
Posted by pietjepuk, 11-13-2005, 08:18 PM
If they want to exclude us from their SLA with that, it says more then enough about them already...........
Posted by kebirhost, 11-13-2005, 08:20 PM
I do not think so. Melih
Posted by Tinozaure, 11-13-2005, 08:21 PM
If you try to start your car and it do not start 5 times on 10 tries. Is your car having a problem or is it normal ?
Posted by RyanD, 11-13-2005, 08:21 PM
the support has been reasonable. However, the quality of thet network has been absolutely awful.
Posted by sprintserve, 11-13-2005, 08:22 PM
The email said 5-15 minutes. Not a full 20 hours (and counting) downtime, 50% packet loss etc.
Posted by web_hosting4u, 11-13-2005, 08:22 PM
If we are looking at it from that angle, then you can say you get what you pay for, however, in most instances, I have been able to get some what reliable support. I will say the allmanaged division is pretty much useless, once I get sago on-line, I have been ok.
Posted by NeonXL, 11-13-2005, 08:24 PM
Sagonet is really hitting rock bottom this time. Although, I do use Sagonet, I won't leave Sagonet. Atleast not yet. I hope Sagonet users get something out of this. Perhaps an upgrade, or some free time. Or something.
Posted by HD Fanatic, 11-13-2005, 08:28 PM
I've suggested Sago to have a network status page hosted OFF-SITE for some time now. This is truly a sad day for Sago I was really a proud supporter until today.
Posted by Justin, 11-13-2005, 08:29 PM
Still seeing anywhere from 30-40% PL at this point. Wondering what is going on with their foundry engineers (they decide to take a brief dinner?). If what Matt says is true why was this not resolved hours ago? Honestly I would think someone qualified to work on a piece of hardware should know the inner and outer workings of the layers it handles. In any case hopefully this is resolved entirely soon. Sharing the same sentiment myself, been with them for about a year and 3-4 months now, hopefully this is going to be the last of these screwed maintenance windows. Last edited by Justin; 11-13-2005 at 08:32 PM.
Posted by holyway, 11-13-2005, 08:30 PM
I was like allmanaged, but ... It's horrible.
Posted by Xzyte, 11-13-2005, 08:31 PM
I received the same email form Beachcomber and will be seriously thinking about changing to a new datacenter. But I have to add that it states at the bottom of the email: So I think you should play by Beachcombers rules and remove the email form you post. Just my 2 cents.
Posted by TrevorN, 11-13-2005, 08:37 PM
The reason you see food in there is because they are working on tickets or other projects, tasks, etc. This way they can eat a meal and continue functioning during the day. I can assure you that they are working on things and not "doing nothing".
Posted by NeonXL, 11-13-2005, 08:37 PM
I'm getting 25% PL, then it jumped to 100%, kicked me off my box, and went down to 75% PL. Now it's at 50% PL It's jumpy. Not good.
Posted by web_hosting4u, 11-13-2005, 08:40 PM
Of course I am following this post by the minute it seems, since I basically cannot get to the SagoNet site more often than not......I have been with many providers, and Sago is the home of my first dedicated server. I am not ready to jump, but I wish we could get some more info. I also agree and off-stie Network Status site would be helpful....I was with another host a yr or so ago as a plain old reseller and they had an off-site forum, just in case! Even came in handy once!
Posted by n00bh0st, 11-13-2005, 08:41 PM
Yeah I am familiar with datacenter work. I've worked in a few of them. The ones I ahve worked at also had a more aggressive stance on support. Poor attitude and lack of getting things done while you sat around and ate pizza was not acceptable. Heads rolled and new employees were hired.
Posted by PhMatt, 11-13-2005, 09:02 PM
Ok, it appears to be a code issue in the Netiron 40G Routers. At this time, new code is being added onto the routers, that will take about 30-40 minutes, which should resolve the problems of the memory leak and CPU overload. Though again better explanations will be given once known, here is what is known at this time. We have dual core routers, during the upgrade, 1 router was taken down, software updated, and traffic passed onto it, while the other router was upgraded, traffic was passing fine on the upgraded router, and the 2nd was brought back online and HSRP reinstated. Again all traffic was functioning properly, and additional work was being performed when the memory leak causing CPU overloads on both routers began, thus the packet loss. Attempts to roll back the software were also unsuccessful and Foundry technicians were deployed as well to assist in the upgrades. It has been identified as a coding issue on the routers themselves, not on our network admins / staff, and Foundry has had to re-write code for the routers. I expect to have another update in approximately 30-45 minutes from the time of this post, and will update then. Thank you,
Posted by NeonXL, 11-13-2005, 09:04 PM
Thanks Matt, thats good to hear.
Posted by Tinozaure, 11-13-2005, 09:11 PM
So I am really impressed to learn that : - SAGO is the only client from Foundry to use NETIRON 40G routers. - Foundry do not load test their patchs on (100K US$) hardware. - "not SAGO's fault so don't think about compensation".
Posted by Born2Kill, 11-13-2005, 09:16 PM
Mat how when you are upgrading software Sago network is not operational ?????? If you have redudant router it should be packet loss ! Wright ?
Posted by NeonXL, 11-13-2005, 09:23 PM
Whow. I just got 0% PL. Does this mean it's over? Edit: nevermind. I got 100% PL now. Scratch that.
Posted by Born2Kill, 11-13-2005, 09:24 PM
Also i have notices some time after you reboot router everything is ok and there is no packet loss ..... After ~60 seconds there is loss .... Two reboots in short time few minutes ago so i can say that foundry tehnicians did not write patch to work Wright ?
Posted by Sekweta, 11-13-2005, 09:24 PM
Over the past several months, Sago issues have caused enough customer complaints, we're ready to start looking around for another facility. If there are any datacenter folks reading this thread (no resellers, we want to talk with the facility owners) that provide colo in the Tampa Bay area, please contact me off-group.
Posted by Born2Kill, 11-13-2005, 09:26 PM
Neon that is after reboot of router ..... He is working about 60 seconds good and then nothing
Posted by NeonXL, 11-13-2005, 09:27 PM
Yes, I realized that. I just saw 0% PL and was like yay! So. Pay no attention to that post.
Posted by Born2Kill, 11-13-2005, 09:30 PM
My question is very simple ? Why sago did not downgrade Foundry software emidiatly after they get info that software is bad ?
Posted by RyanD, 11-13-2005, 09:30 PM
is it that impossible to roll back to a working image while you get foundry's crap together?
Posted by HD Fanatic, 11-13-2005, 09:33 PM
Why is there so much packet loss at the tampa level3 hops? And how will Savvis play a role in Sago after all this mess clears up?
Posted by web_hosting4u, 11-13-2005, 09:35 PM
According to Matt's post above, the could not roll it back...I have seen this with some other equipment that I once worked on, they were totally unable to revert back....talk about a non-back up plan! The only thing I can say is Sago needs their own way of testing rather than trusting ANY third party supplier. You cannot go in on the word of the provider, just cannot be done.
Posted by NeonXL, 11-13-2005, 09:37 PM
Why did they not test it on 1 before they used it on teh entire system? D:
Posted by HD Fanatic, 11-13-2005, 09:42 PM
Sago should ditch Foundry and go with Cisco. Why don't they hav a backup plan in case something goes wrong? A provider this big should have everything outlined before attempting the big upgrade, maintenance.
Posted by RyanD, 11-13-2005, 09:42 PM
how long was the primary running on the new image before you decided to re-image the failover?
Posted by miracle, 11-13-2005, 09:46 PM
75% packet loss here still. Seems almost all the recent network issue due to the buggy Foundry router. I don't get why SAGO switch from CISCO to Foundry. It is not good at all !!!
Posted by RyanD, 11-13-2005, 09:48 PM
imho it's been downhill since they brought in the foundry equipment
Posted by dreaz, 11-13-2005, 09:49 PM
I'm still having packet losses, I'm surprised Sago won't take the time to inform customers what is going on and when it will be fixed. Actually, I'm not.
Posted by Born2Kill, 11-13-2005, 09:50 PM
They are rebooting (upgrading software ?! ) router over and over again and nothing ....
Posted by r2tech, 11-13-2005, 09:56 PM
bah... you know whats missing so often in this industry? accountability. my server is down - as far as my clients are concerned its MY problem. they don't care about who did what at my NOC. the same principal then applies to the next level. my server is down, I blame my NOC. i dont care who did what to what hardware. SAGO - your hardware, you did the upgrade, you take the responsibility.
Posted by web_hosting4u, 11-13-2005, 09:57 PM
While I understand everyones concerns about Sago and their some what lack of customer support / notification, I have to make every best assumption that they are all hands on deck in order to resolve this issue (15 hrs). While I still am not defending them, it is somewhat difficult to update and work at the same time, especially on what I assume is a Sunday staff.....(not that everyone should not be on board by now)..... The worst part for me, is after all those yrs of being a reseller stuck in the mud when outages occurred, I have been feeling great as a dedicated server company, not anymore....I hope to have at least one customer when this is all over with!
Posted by n00bh0st, 11-13-2005, 09:58 PM
Because that would be rational and well thought out
Posted by miracle, 11-13-2005, 09:58 PM
same. I feel I was cheated.
Posted by TrevorN, 11-13-2005, 09:58 PM
You could say the same thing from sagonet level too. they paid for the hardware being told it will work and then there is a software issue. I think it's more complicated then just one person or one action. Most of the time when things go wrong it's a combination of things that create a problem.
Posted by Born2Kill, 11-13-2005, 10:00 PM
slingshot Sceduled maintenance like upgrading Core routers software needs full team in NOC Wright ?
Posted by miracle, 11-13-2005, 10:00 PM
I don't think everyone in SAGO knows how to deal with the core router. they should have people who is not working on the issue and notice their custmers. Last edited by miracle; 11-13-2005 at 10:04 PM.
Posted by n00bh0st, 11-13-2005, 10:02 PM
Yeah but they went whole hog and took the entire network down. They didn't do it in stages or test anything. They did not plan, they did not anticipate, they did not create a backup solution. We are now feeling the flames from our client bases, possibly losing money and damaging business relationships. We are still being affected by their poor judgement, and hardware issues.
Posted by miracle, 11-13-2005, 10:02 PM
that's not my concern. I only concern about the network and service I paid for.
Posted by r2tech, 11-13-2005, 10:04 PM
this very true, and i totally agree but just like i have to take responsibility and refund my clients for this downtime, sago net should be taking similar appropriate actions. i am not saying they need to refund us (althought that would be nice), taking responsibility for this is a start. of course i expect them to give foundry more than just a mouth full, but that is not something that interests me. that is their problem. my problem is my server not being up.
Posted by Justin, 11-13-2005, 10:04 PM
While it is claimed to be "good house keeping" maintenance I don't really understand how that can constitute 15 hours of outage(s)/packetloss. While I understand the situation sagonet is placed in (between a rock and a hard place) sago does still owe it to their loyal customers (majority of us who've toughed it out with them this long) to take accountability for their mistakes be it vendor or not, as sago still is who chose the hardware they did. In any case the PL here is at around 20% so it does appear they are getting somewhere finally on this.
Posted by diginux, 11-13-2005, 10:15 PM
Wow, even better. my.sagonet.com says I have use 170 gigabytes of data transfer today, when this past week I have used 50GB total. Talk about lame. Not only am I not getting screwed out of having access to my server, I am being charged 1/10th of my monthly bandwidth usage. Last edited by diginux; 11-13-2005 at 10:21 PM.
Posted by TrevorN, 11-13-2005, 10:19 PM
If there is an issue of bandwidth calculations just send in a support ticket and state the issue. that type of issue can usually be sorted out.
Posted by PhMatt, 11-13-2005, 10:26 PM
Latest update is that the new software has been loaded onto the routers, and updating are being done which is improving routing / packet loss on an ongoing basis. Though everything is not completed right now, we are on track to resolving things as quickly as possible. I'm not a network admin, nor an admin in any sense, but my understanding is roughly the equivalent of installing a new OS, then applying security patches and updates necessary to have things running properly. The base OS is installed, and the patches being applied to correct any remaining issues. The only ETA I have for completion is as quickly as possible, but the largest portion to things has been done at this time. Thank you,
Posted by TrevorN, 11-13-2005, 10:30 PM
Thank you Matt for the update.
Posted by SeeServers, 11-13-2005, 10:44 PM
I hope it is up by monday moring, but I bet it will still has 20-30% PL when I wake up..
Posted by web_hosting4u, 11-13-2005, 10:45 PM
Well, I suppose you are right, but what is a full team to Sago? I receive replies from the same techs almost all the time. Though Sago seems to be a well named or reputable (previously) company, I do not know how many people work for them. When I was having issues cancelling my AllManaged Reseller Account, I received e-mails from Matt as AllManaged, tonight, he is Sago. Why have two companies? I would hope they would have all hands on, an open bridge, and Foundry on-line....what is a foundry and why would you go from Cisco to Foundry? (Lower cost??)
Posted by Xzyte, 11-13-2005, 10:48 PM
Thank you for the update Matt. Although the packet loss is still high, I really appreciate it that you keep us updated on the issue. This way at least I have some explanations for our customers.
Posted by r2tech, 11-13-2005, 10:52 PM
it is already monday in australia coming up on 5 hours of downtime during business hours....
Posted by web_hosting4u, 11-13-2005, 10:53 PM
I would be willing to bet that all of us pinging, and testing and trying to get e-mail and our clients trying yto get their websites for the first time all day is having ~no~ affect on the work they are doing! ~Hope the sarcasm is noted~
Posted by godboko71, 11-13-2005, 11:05 PM
Hi, This is just for your last bit, in general I have found Foundry a much better Core router. Now I haven't worked in a DC full time in about 6 months now, but the on-site work I still do for my old company Foundry still works hands down better then Cisco's equipment. Anyway I do hope they get this issue fixed soon, if its not up by 9 AM tomorrow I will be forced to switch DC's but that's neither here nor there. To some of the comments on support, I have over all got good support, day time seems to have the best, but slowest support, night time has anywhere from pretty good to bad support, but fairly fast response time. Weekends tend to have the best mix of good support and fastish response times.
Posted by RossH, 11-13-2005, 11:16 PM
Just MHO...but I don't see why people are blaming the foundry equipment. Foundry makes some very solid gear, just depends on how well you can implement it.
Posted by nero-x, 11-13-2005, 11:21 PM
been saying "30-40 minutes" for several hours. i could excuse the downtime, i cant excuse ******** ill be moved before they get this issue resolved
Posted by godboko71, 11-13-2005, 11:22 PM
Your completely right there, in the DC I worked at they loaned us equipment to do testing before doing Major upgrades on Core system. Never fail like this did, but again this was a different DC with different equipment, and different people. About the same size though, so I can't think that's a factor.
Posted by The Manager, 11-13-2005, 11:22 PM
Although I have a VPS through RegisterFly located at sagonet, this problem is larger than just people who are sagonet clients. I have comast here in south FL and this started last night about 8pm and I was completely out until this morning. All day the net has sucked badly, somehow related to sagonet I guess since I have problems connecting to google, 1&1, everywhere.. The Cablemodem has been sync'd up and working since around 9am so it's not comcast entirely anymore, if at all.
Posted by jokaroo, 11-13-2005, 11:31 PM
They still can't be down? It's not possible!? Yup... server is still down! I'm so happy they ripped me off last month and charged me $1000 extra on my bill. It ended up making me change one of my more important servers!! Thank god the important one was moved elsewhere... I only have to complain about 1 box being down now, instead of 2 (one that was $2500 a month!!!).
Posted by HD Fanatic, 11-13-2005, 11:34 PM
What could we expect after this maintenance is finally done with all the issues worked out? Man, this has really put a dent in Sago's reputation.
Posted by web_hosting4u, 11-13-2005, 11:35 PM
I wouldn't call it down, I would call it managing through a pretend dial up connection!
Posted by SeeServers, 11-13-2005, 11:56 PM
A dent, I don't see a dent..... .... .... I see a gaping wide hole.
Posted by davidarenson, 11-13-2005, 11:58 PM
This is causing my business irreparable harm. This is a COMPLETELY unacceptable amount of downtime for what appears to be sheer negligence. This is going to put a lot of people out of business I'm afraid. As soon as this is done everyone should immediately pull their business from Sago, no company does things like this to their valued customer base.
Posted by sprintserve, 11-13-2005, 11:58 PM
That pretty much sums it up. From the sounds of it, they did the patch on production core routers without doing testing or sufficient testing. The issue happened shortly after the patch, and a proper test environment will show that to be the case. Even if they are lazy, they should probably do 1 core router first, run it for a full day and see how it works instead of doing 2, one after another in quick succession. We have pretty much exhausted the goodwill we build over the years in a single day, due to their lack of proper care and dilligence. I been holding back the whole day, but I am pretty much sick to the core right now.
Posted by james007, 11-14-2005, 12:09 AM
Last time, when dedicatednow moved from NAC to own data center, same thing happened, also lack of planning.
Posted by PhMatt, 11-14-2005, 12:09 AM
Still continued tweaking is ongoing, and things are steadily improving, Foundry techs and our network admins are working on resolving packet loss issues non-stop. Though this is frustrating to everyone, it will be resolved, we've done whatever possible prior to the upgrade to ensure things were tested on Foundry's end, then individual core upgrades, and of course then the problems began. Though we're being assured by Foundry that they are doing everything possible, we still stand as unhappy customers ourselves, and can understand everyone's frustration. Once completed though, it is likely as with last year's deployment of the routing gear, that the system will run smoothly and properly from that point on. I'm babbling here, we're all stressed and just doing whatever's necessary to restore things, network admins haven't had breaks in literally over 24 hours, but we are here, we are working on things, and will do so until resolved. Thank you,
Posted by gimme5, 11-14-2005, 12:14 AM
Matt, How are you planning to compensate your customers?
Posted by RyanD, 11-14-2005, 12:27 AM
I'm sure it'll run fine until the next upgrade when it'll place itself right into a dumpster.
Posted by WCSWEB, 11-14-2005, 12:28 AM
The question that I have been asking myself for the past 24 hours of downtime is. Where is the backup? If Foundry showed up why didn't they came in with 2 spare routers to bring things up? that is IF!!! Can't believe a datacenter completely down for 24 hours with no ETA and no resolution. I have yet to see improvement on the servers we have there.
Posted by godboko71, 11-14-2005, 12:31 AM
The people at foundry are great about loaning/renting equipment to there clients if it means they keep an account, so I can't see that Foundry has had any real involvement besides maybe phone support and a patch.
Posted by cknipe, 11-14-2005, 12:35 AM
Exactly. IF Foundry is 'on site' as claimed, I also find it extremely hard to believe that Foundry is unable to fix their own equipment.... There's been no mention of hardware failures (for which they SHOULD have backups), and software... Well, if ANYONE should know this software, I would expect it to be Foundry... -sigh- I'm so tired of this it's not even funny.
Posted by ravegrunt, 11-14-2005, 12:42 AM
The saddest part is, it's probably only begun. I still don't dislike Sago, maybe I fel they've built a good rep with Customer Service (excepting this instance, of course) and I can only imagine they'll give bill credits this round for this. I personally won't leave Sago over this, as they'll probably learn from this mistake. If they don't learn though...
Posted by godboko71, 11-14-2005, 12:55 AM
I would rather a ram upgrade then credit myself but thats just me.
Posted by HD Fanatic, 11-14-2005, 12:56 AM
Or perhaps a free upgrade of cpu/ram/hard drive
Posted by SeeServers, 11-14-2005, 12:57 AM
I now have 100% PL, so it is down again...
Posted by godboko71, 11-14-2005, 12:58 AM
Heck I would almost be happy if they would let me ship them the ram to put in with no lobar costs, they charge to much for theirs.
Posted by kneadingu, 11-14-2005, 12:59 AM
I think so too but it was'nt funny until you pointed it out lol. Nevertheless my trace is still looping around the building lol
Posted by ravegrunt, 11-14-2005, 01:00 AM
Or maybe port increases, to, you know, test the new routers...100mbit for all!
Posted by HD Fanatic, 11-14-2005, 01:02 AM
I wouldn't mind seeing price reductions on current servers too..
Posted by NeonXL, 11-14-2005, 01:02 AM
I'd rather not get a RAM upgrade. This story convinces me not to. http://origis.no-ip.org/sagonet.txt
Posted by godboko71, 11-14-2005, 01:02 AM
Ram and that and I woudn't complain for a year
Posted by ravegrunt, 11-14-2005, 01:03 AM
Anything would be fine, this is frusterating and I could only hope that they would be willing to fix it...
Posted by godboko71, 11-14-2005, 01:04 AM
Not to worried they are generally happy to work with there customers to make them happy.
Posted by kneadingu, 11-14-2005, 01:05 AM
This is/was a preventable disaster. This upgrade should not have been attempted without a tested exit/backup plan. Proper planning would have prevented this and competent personell should have forseen the need to have a tested exit/backup plan.
Posted by godboko71, 11-14-2005, 01:11 AM
Yeah and its been said a million times, I think we get they dropped the ball and we are all upset and some of us are losing money, but all we can do is wait it out. Though I do wonder why they didn't have a backup router at least, or better yet run the upgrade on a test hook it up opps didn't work right revert back to the other. But oh well Hind sight is 20/20.
Posted by RyanD, 11-14-2005, 01:14 AM
Agreed, The first router that was re-imaged obviously wasn't run on very long before someone made the call to image the fail-over device. Once that call was made it was done for. this is just simply unacceptable.
Posted by juju, 11-14-2005, 01:15 AM
Hi, I feel the pain too. I got today about 110 SMS about up and down status. Only this will cost me about 15$ Speed is super slow.. when it works Can I request some money back and what should I do? I have 4 servers there.
Posted by holyway, 11-14-2005, 01:23 AM
Folks! They talk about ETA. Wht is mean?
Posted by RyanD, 11-14-2005, 01:26 AM
Estimated Time of Arrival.... in this case, when the fix gets here
Posted by holyway, 11-14-2005, 01:31 AM
So it's mean, they don't know whenfif issue?
Posted by RyanD, 11-14-2005, 01:33 AM
it means they have no idea when they'll fix it
Posted by swijaya0101, 11-14-2005, 01:35 AM
give them another 24 hours. they have been quite good for me except for this ******** maintenance.
Posted by sprintserve, 11-14-2005, 01:36 AM
Any idea why Hivelocity colocated space at Sago is up and we aren't?
Posted by RyanD, 11-14-2005, 01:42 AM
maybe they were smart and just pulled the faulty foundry gear they were piggybacking on from sago? Only HiVelocity can answer that.
Posted by PeaceNLove, 11-14-2005, 01:55 AM
Reminds me of the Mickey Mouse club....
Posted by ravegrunt, 11-14-2005, 01:57 AM
Mickey Mouse Club? How so?
Posted by PeaceNLove, 11-14-2005, 01:59 AM
Have ya not heard the old saying...meaning not organized...I'd guess you could say.
Posted by HD Fanatic, 11-14-2005, 02:05 AM
Amazing, Hivelocity took the matter into their own hands and fixed the problems. Ther servers are running at 100%.... gotta love this
Posted by NeonXL, 11-14-2005, 02:09 AM
I hope this issue will be up by tomorrow morning. I have servers to run. I guess patience is the key in this one.
Posted by HD Fanatic, 11-14-2005, 02:10 AM
It's already past 24 hours, still getting packet loss and very slow speeds. For god sakes, just take those Foundry routers offline and use another brand.
Posted by holyway, 11-14-2005, 02:12 AM
If we lost our customer's, SAGO lost us I think
Posted by Chuggles, 11-14-2005, 02:24 AM
How did hivelocity fix the problem? More importantly why can't the other resellers do whatever they did? Fixed at 10pm - have to give them props on that one.
Posted by miracle, 11-14-2005, 02:31 AM
can't access our server completely again.
Posted by HD Fanatic, 11-14-2005, 02:31 AM
oh great, i can't even ping my servers now, one step forward, 2 steps back...
Posted by bestprobe, 11-14-2005, 02:35 AM
My sites are online. But really toooo slow. Waiting..............
Posted by NeonXL, 11-14-2005, 02:43 AM
I wonder how much longer this will take. Most of my users have left. Which make me angry, but I have learned something out of this. Always test your upgrades/fixes before you release them to the public. If Sagonet tested, this could have probably been avoided. But, things don't always work out as you hoped.
Posted by patryn20, 11-14-2005, 02:48 AM
Luckily I mainly use these servers for dev purposes. Other than that we just run our email and low-volume sites. Still, this is really annoying. When pinging the Sago servers, I am getting the following packet loss: 30% From The Planet in Dallas 38% From CI Host in Bedford, TX 32% From Rackspace I was considering moving more servers with one of my partners into Sago for money saving purposes, but don't think that will be happening now. I think the extra money is well spent on redundancy in my equipment to counteract lack of redundancy in the providers equipment. I was able to move my "must be up" (my personal) sites over to my servers in CI Host. If you have the money, maintaining a couple of extra servers may be worth it. Just get some really cheap colo or dedicated elsewhere and put a short TTL on your DNS records. I really do have to expect this sort of thing from low cost providers, though. Our colocation space at CI Host had constant power problems until we raised all holy hell about it. I think I will keep a nice, even distribution of servers between at least two datacenters to prevent total failure. Thank god I only recommend client servers be put in Rackspace or The Planet as of late. The extra money is well spent, especially when it isn't my own. ;-) On a side note, back in highschool I worked at an internet provider with multiple carriers into the building. Even there, with only 2 routers and ~1000 customers, we had complete hot spares with known working configurations. Seriously, if a small firm can do it, one churning the revenue of Sagonet can do it. Last edited by patryn20; 11-14-2005 at 03:01 AM.
Posted by sprintserve, 11-14-2005, 03:36 AM
Within US, packet loss is about 30-40%. Outside of US, it's about 80% (i.e. inaccessible)
Posted by miracle, 11-14-2005, 03:41 AM
still packet loss after more than 24 hours!!!
Posted by niyogi, 11-14-2005, 04:03 AM
Hi folks: I'm asking this because for 24 hours, their NOC has been working on some connectivity issues through one of their resellers for us and we cannot get a straight answer about what is going on. Bizarre. Roj
Posted by Born2Kill, 11-14-2005, 04:10 AM
Is there any info ?
Posted by r2tech, 11-14-2005, 04:11 AM
we are getting 100% packet loss now from sydney, australia
Posted by swijaya0101, 11-14-2005, 04:16 AM
100% packet loss from melbourne, australia.
Posted by HD Fanatic, 11-14-2005, 04:19 AM
server is COMPLETELY unreachable, 100% packet loss from New York, USA. Back to square one again, grrrrr This is f***ing great. things got worse as time goes on. You'd have thought they got their act together by now.
Posted by r2tech, 11-14-2005, 04:21 AM
lets just HOPE they are pulling out those damn pesky routers and putting something else in. LETS HOPE!!
Posted by bestprobe, 11-14-2005, 04:37 AM
I am in CHina, server 100% unreachable 20 minutes ago. Becoming worth.......... WHY??
Posted by Born2Kill, 11-14-2005, 04:41 AM
That is just a dream
Posted by HD Fanatic, 11-14-2005, 04:41 AM
Maybe they finally decided to give up on Foundry and go with another brand
Posted by Critters, 11-14-2005, 04:48 AM
8:40am on a Monday morning and the UK working week has started, still 100% packet loss : ( Sagomatt... can you not just replace the routers? I guess that has been thought about and dismissed? I don't think we are going to get much info while they are all busy working, so asking the same questions over and over is a bit futile, however I do expect a full postmortem of the problem this week on the sagonet forums. What have people lost from this downtime? It has upto now been a Sunday, which for me is the least productive day, but still over $100 in advertising (PPC) revenue. Which will rise to $300 if it is not up today. Which is pretty much a months hosting. I have said this on the sagonet forums (which are down now), and I will say it again... Can sagonet please host the forums / service status page on another network so their customers can be informed of downtime progress without having to find and register on a 3rd party forum. Last edited by Critters; 11-14-2005 at 04:51 AM.
Posted by sprintserve, 11-14-2005, 04:52 AM
From what we can see, it's getting worse by the minute. It is now about 60% packet loss within US, and about 90% outside of US.
Posted by Born2Kill, 11-14-2005, 04:52 AM
Hey guys do you know how much that job is ?
Posted by brooky, 11-14-2005, 04:53 AM
I'm moving to another Datacantre: Beachcomber are my hosting company they sen me this email yesterday...
Posted by Born2Kill, 11-14-2005, 05:00 AM
Why are they waiting for patch just downgrade software on routers if they do it on core switches !!!!!!!!!!!!!!!!! They are laying us
Posted by cknipe, 11-14-2005, 05:10 AM
According to what I heard... The upgrade broke firmware that is inside the cards in the firmware. Because the firmware is broken, the earlier versions of the software won't work either (the downgrade). I dont care WHAT they did / did not do.... Or who's fault it is. Someone's head is going to roll over this... I've HAD it.
Posted by cunni, 11-14-2005, 05:20 AM
This is Ridiculous, im sat at my desk (at work), its currently 9am, and yet my server is still unreachable! (UK) i really hope i get access to the box at some point in the next 24 hours, Just so i can backup my data! Depending on how sago treats this situation and how much of a blow sagonet has taken, I dont know about anyone else but when a d/c goes tits up this bad there is going to be really bad repercussions...
Posted by HD Fanatic, 11-14-2005, 05:23 AM
I'll have to go to bed knowing my the network is still down Gosh, things better be back to normal by the time I wake up
Posted by Born2Kill, 11-14-2005, 05:25 AM
Anewday that i thinked when i go in bed lastnight ... Guess what happened when i wake up this morning
Posted by RyanD, 11-14-2005, 05:28 AM
I was seeing only 3%PL to certain portions of the network now it's entirely dropped again. Hopefully that 3% I was seeing was a test and now they're making that change system wide? I'll take 3% over 50% any day. one step closer....
Posted by HD Fanatic, 11-14-2005, 05:30 AM
100% packet loss here in NY, server is unreachable.
Posted by niyogi, 11-14-2005, 05:34 AM
I was wondering where my thread that I posted went - and then found it here. This is really silly - I understand when a network segment upgrade goes bad but the entire network? I'm now waiting for Matt's latest update. Thanks brooky for the information. :-) Roj
Posted by HD Fanatic, 11-14-2005, 05:41 AM
Hmm, sagonet.com and allmanged.com both work but none of my servers are reachable
Posted by Born2Kill, 11-14-2005, 05:43 AM
Same from europe
Posted by Vaya, 11-14-2005, 05:48 AM
It is now 9:50 am in the UK. My server first went down at 10:00am yesterday and has been inaccessible since. 24 hours of downtime is more than 'frustrating', to quote sagomatt. Sagonet has clearly been completely unprepared for anything less than a perfect switch and frankly they have proven themselves utterly incapable. If my server is not back online very shortly I will certainly be spending (wasting) the day trying to arrange a server at a different datacentre. I guess this just goes to show that if something looks like a bargain, it probably isn't.
Posted by Born2Kill, 11-14-2005, 05:49 AM
But i think that they pluged off part of networks where our servers are and then network load is not big
Posted by RyanD, 11-14-2005, 05:54 AM
thats what it looks like large netblocks are entirely unrouteable
Posted by blind, 11-14-2005, 06:01 AM
I used to like AllManaged... until yesterday. It's almost 24hours of my server going up and down every few minutes. Since my business depends on hosting I need to consider the most frustrating thing - looking for and moving to another company :/
Posted by RyanD, 11-14-2005, 06:13 AM
now it's all unrouteable again
Posted by navyseal, 11-14-2005, 06:14 AM
sagonet.com is up..but I am still unable to reach my own server.
Posted by Critters, 11-14-2005, 06:19 AM
Anyone else using PPC programs such as Googles AdSense? Checked your account recently? Last Sunday: Page impressions: 98,097 Clicks: 5,517 CTR: xx% CPM: USDxx Earnings: USD771.28 This Sunday: Page impressions: 14,379 Clicks: 585 CTR: xx% CPM: USDxx Earnings: USD107.72 So that makes me down USD663.56 And now it's Monday, a much more busy day of the week (>USD1,000 last week) My bad for not having redundancy? (having a 2nd datacenter) or is it fair I did not anticipate > 24 hours downtime and the cost to me? Plus the sago server is running some client sites, which will have to be compensated in some way by myself.
Posted by cunni, 11-14-2005, 06:20 AM
Sagonet.com is up and down for me all the time, like every 10 minutes it dies for 10 minutes then its back and then its gone. Its like playing hide and seek, except sago have hidden and we now cant find them. I havn't heard anything on here or on there site in some time now. Like someone said before, its like they have Dead Routed all the Colo and Managed servers from the BackBone and are now trying to get there own working first. This whole thing is shocking... most definatly shocking to say the least...
Posted by Born2Kill, 11-14-2005, 06:21 AM
They have enough time to replace "paperweight" foundry with cisco/juniper for 24 hours ....
Posted by phactor, 11-14-2005, 06:34 AM
this is just ridiculous. More then 24 hours of downtime. I just want my servers up to start moving customers, as its unnacceptable.
Posted by mozza, 11-14-2005, 06:36 AM
I think it's time we started thinking about a class action to recover losses. Anyone know a good class action lawyer in Florida?
Posted by holyway, 11-14-2005, 06:40 AM
Agree with mozza!
Posted by Critters, 11-14-2005, 06:41 AM
Why not wait to see: 1. how long this lasts 2. what compensation is offered before getting lawyers involved. Way to live upto a stereotype
Posted by r2tech, 11-14-2005, 06:41 AM
better yet, anyone know any goons for hire down there?
Posted by cunni, 11-14-2005, 06:42 AM
For the first time in a long time i can now login to my box, im still getting alot of PL though. around 75%
Posted by seusitio, 11-14-2005, 06:43 AM
Now it is done.
Posted by ahostinginc, 11-14-2005, 06:46 AM
How do you know? it is still same...
Posted by phactor, 11-14-2005, 06:46 AM
still down here!
Posted by blind, 11-14-2005, 06:50 AM
My seems to be up but how knows for how long?
Posted by seusitio, 11-14-2005, 06:55 AM
I think that sago died.
Posted by Critters, 11-14-2005, 06:56 AM
nothing here (UK) can't even see the sagonet.com site.
Posted by godboko71, 11-14-2005, 06:56 AM
It was up here for like 30 mins, a few hours ago, I went to try and get some sleep, then my pager and cell went off the hook, have been 100% downtime here ever since. tracert www.eternalgaming.com Tracing route to eternalgaming.com [65.110.37.200] over a maximum of 30 hops: 1 12 ms 12 ms 13 ms ip68-0-104-1.tu.ok.cox.net [68.0.104.1] 2 11 ms 9 ms 9 ms 68.12.9.49 3 11 ms 8 ms 9 ms 68.12.10.245 4 26 ms 18 ms 15 ms 68.12.14.66 5 17 ms 17 ms 15 ms 68.12.14.29 6 19 ms 15 ms 17 ms 68.1.1.56 7 55 ms 62 ms 62 ms fed1bbrc02-pos0201.rd.sd.cox.net [68.1.0.208] 8 * 53 ms 55 ms fed1dsrj02-so010.rd.sd.cox.net [68.1.0.201] 9 69 ms 65 ms 65 ms 12.118.229.13 10 80 ms 66 ms 66 ms gbr2-p100.sd2ca.ip.att.net [12.123.145.58] 11 69 ms 69 ms 69 ms tbr2-p012701.la2ca.ip.att.net [12.122.12.1] 12 65 ms 66 ms 64 ms ggr1-p370.la2ca.ip.att.net [12.123.28.133] 13 56 ms 58 ms 55 ms lsanca3lcx1-pos2-0.wcg.net [64.200.140.89] 14 57 ms 56 ms 56 ms anhmca1wcx2-pos6-1-oc48.wcg.net [64.200.140.161] 15 56 ms 59 ms 58 ms hstntx1wcx2-pos1-0.wcg.net [64.200.240.65] 16 57 ms 57 ms 57 ms hstntx1wcx3-pos9-0-oc48.wcg.net [65.77.93.202] 17 63 ms 62 ms 62 ms btrgla1wce010-pos2-0-wcg.net [64.200.210.54] 18 76 ms 75 ms 76 ms tampfl1wcx1-pos9-0-wcg.net [64.200.210.245] 19 78 ms 77 ms * 65.77.96.150 20 * * * Request timed out. 21 63 ms * 62 ms 65.110.37.200 Trace complete.
Posted by r2tech, 11-14-2005, 06:58 AM
100% up here (sydney australia) EDIT: scratch that, spoke to soon.....100% packet loss now.
Posted by godboko71, 11-14-2005, 06:59 AM
I am wondering, is anyone getting anything from any of there phone numbers? There any new updates? cuz I can't get ahold of any of my contacts.
Posted by blind, 11-14-2005, 06:59 AM
getting a lot of packet losts though
Posted by Born2Kill, 11-14-2005, 06:59 AM
r2tech still 100 up ?
Posted by ahostinginc, 11-14-2005, 07:01 AM
I think there will be an official explanation when its finished so i will wait that to confirm.
Posted by r2tech, 11-14-2005, 07:01 AM
its like a yo yo. get 100 up, then 100 packet loss. RIGHT NOW, its good.
Posted by Born2Kill, 11-14-2005, 07:03 AM
how packet loss may be good ?
Posted by swijaya0101, 11-14-2005, 07:03 AM
finally 0% packet loss ... hopefully it will continue on
Posted by Born2Kill, 11-14-2005, 07:04 AM
i still can not ping anything
Posted by r2tech, 11-14-2005, 07:06 AM
where are you located?
Posted by Born2Kill, 11-14-2005, 07:08 AM
europe but i ping sagonet from managed
Posted by cunni, 11-14-2005, 07:09 AM
My box's are working fine now, and on the looks of things a trace route is bringing back less hops...
Posted by mozza, 11-14-2005, 07:12 AM
Look what happens when you mention class action. You get action - Yippee
Posted by cunni, 11-14-2005, 07:14 AM

Posted by cunni, 11-14-2005, 07:16 AM
totally differnt routes... lol wierd
Posted by cunni, 11-14-2005, 07:19 AM
I think they have sorted it, *fingers crossed* and no offence, but if it is sorted... its about blummin time!
Posted by AbusedCustomer, 11-14-2005, 07:21 AM
I'm customer of sago more then 2 years... ans had some problems with my server time to time. I own few servers at different providers, but have problems only with sago... Looks like they do not have a professional staff in there and make some trainings monthly. I'm going to quit from there - I'm loosing money more then 2 days! Who will back them to me?! Sago? Don't try to make laugh my boots...
Posted by Born2Kill, 11-14-2005, 07:23 AM
I still have problems ... Any one has problems too ?
Posted by InKLy, 11-14-2005, 07:24 AM
it's great from Caen France.. Last edited by InKLy; 11-14-2005 at 07:27 AM.
Posted by cunni, 11-14-2005, 07:24 AM
Ye i also feel the same way, but like i said there isnt much you can do when its down. I have other server else where and i do also feel that this is most unreasonable. We havn't heard from Sago for sometime now...
Posted by sprintserve, 11-14-2005, 07:26 AM
From US, I am now seeing zero packet loss, but from overseas, it's still 60-70%
Posted by Vaya, 11-14-2005, 07:32 AM
0% loss from the UK for me.
Posted by cunni, 11-14-2005, 07:33 AM
Yep ditto for me also.
Posted by worldgoer, 11-14-2005, 07:33 AM
From China,it's still 70-80% packet loss.
Posted by holyway, 11-14-2005, 07:34 AM
0% loss Russian Far East!
Posted by Critters, 11-14-2005, 07:36 AM
All good here (UK) quite fast.
Posted by godboko71, 11-14-2005, 07:40 AM
Well I am setting my alarm for 9 AM and I better see a real answer in my email, on there site, or they are getting yet another angry and tired call. Lucky I go off call in 20 mins so no harm in turning cell and pager off yippie!! Though I do hope it does not completely go down again.
Posted by juangake, 11-14-2005, 07:43 AM
12:38 h. in Spain, 1% packet loss, five mins ago 30%. Things may be improving but we're still praying.
Posted by Born2Kill, 11-14-2005, 07:44 AM
There is 1% packet loss ....... Weird
Posted by miracle, 11-14-2005, 07:48 AM
0% packet loss, China. thank god, after 28 - 30 hours!!!
Posted by gonzalo, 11-14-2005, 07:50 AM
...12:49 h in Spain, rush hour and 12% packet loss... Any words from the sagonet tech support? //Gonzalo
Posted by gonzalo, 11-14-2005, 07:59 AM
...12:58 h (Gmt +1)... 0% packet loss the beginning of the end of the problem? //Gonzalo
Posted by Mr.Muffinman, 11-14-2005, 08:08 AM
7:07 AM (GMT -5:00) Location: New Jersey( Around Newark Int. Airport ) Packet Loss: 25% - 0% Provider: Comcast Hope this helps someone. Thanks
Posted by Xzyte, 11-14-2005, 08:09 AM
0% loss for the last hour from Thailand. Let's hope it stays this way.
Posted by worldgoer, 11-14-2005, 08:10 AM
from china ,still 20% packet loss
Posted by web_hosting4u, 11-14-2005, 08:22 AM
I have 7% lost, but saw my e-mail for the first time in 24 hours!
Posted by Bacus, 11-14-2005, 08:38 AM
100% PL from Germany
Posted by navyseal, 11-14-2005, 08:57 AM
0% loss from India, nice
Posted by swijaya0101, 11-14-2005, 09:07 AM
oh man ... i start receiving 20-50% pl again.
Posted by bestprobe, 11-14-2005, 09:10 AM
Server is Online, but MySQL is failed and could not be restart by WHM
Posted by jokaroo, 11-14-2005, 09:14 AM
We are working now.
Posted by claudioszykman, 11-14-2005, 09:16 AM
Hi guys this is really lame... i am here at Brazil and i am monitoring sagonet from winmtr.soundforge.com and dnsstuff.com pinging and i can see it is better than some hours ago but still has some packet loss seems that they are rebuilding all routes and nullifing unwanted packets, etc so if you cannot access then it can be a condition created from they`re network issues and some cache from our local internet adls providers... the last network issue they had was at march (big issue i mean) and it took 12 hours to be normalized and i lost one customer i want to belive that they will normalize this and stay stable at least until the next year otherwise it will be very bad to their public image... regarding to this i wish i were at florida.. i could try send then some pizzas and see if they play it cool, etc thanks CLaudio
Posted by sami4net, 11-14-2005, 09:18 AM
I am still unable to access sagonet site , iam browse from saudi arabia
Posted by brooky, 11-14-2005, 10:16 AM
I've bought a new server in another location because of this. Moving asap.
Posted by RyanD, 11-14-2005, 10:25 AM
still seeing 25-40% PL on all servers except one that is seeing only 7%. ARGH
Posted by nphase, 11-14-2005, 10:26 AM
me too. this whole weekend of downtime is ridiculous! my clients can't stand it, and I can't stand losing credibility.
Posted by RossH, 11-14-2005, 10:30 AM
Very true, they loaned us equipment multiple times when we were doing testing for network upgrades and their techs are truley top notch. Never used them as an edge router but have used them for multi gigabit load balancing.
Posted by gimme5, 11-14-2005, 10:39 AM
0% packet loss here. Hopefully this is over soon for everyone...
Posted by dreaz, 11-14-2005, 11:14 AM
Still a lot of packet loss from Brazil. It's amazing, this is the fourth time that Sago gives me a fake ETA. I'm out.
Posted by PhMatt, 11-14-2005, 11:15 AM
As of this morning, though improved, there are still some lingering issues of packetloss for people. sagonet.com is a bad choice for testing at the present time, as with so many pings the firewalls will begin rate limiting thinking that it's being attacked. At this time, anyone seeing packet loss, is urged to open a ticket with ping and traceroute results, try to run a traceroute to show where the packets are being dropped, and enough pings to show substantial testing. We are continuing to work on any remaining issues, though things have improved significantly since last evening. Thank you,
Posted by ahostinginc, 11-14-2005, 11:23 AM
Thanks Matt for the update, I can confirm that there is a certain improvements at services much much better than yesterday
Posted by web_hosting4u, 11-14-2005, 11:27 AM
Good Morning SagoNet.....I have a great idea, how about you begin updating your own forums?? You have many customers in your own forums that are looking for a status and there still is not one! You have nothing in Network Status and this entire outage has been tracked in General Questions. I find it outrageous that I have to come to WHT to get a status on my server providers outage!
Posted by Sekweta, 11-14-2005, 11:27 AM
Opening tickets does no good. They go unanswered. Calling Support on the phone does no good-- we just get voice mail. Callilng Billing and asking to be transferred to Support-- we are told that nobody is on premises at this time who can handle routing issues. Did I say that loud enough? It's 10:30 a.m. Monday morning and we've been told NOBODY IS ON SITE who can handle the routing issue.
Posted by Sekweta, 11-14-2005, 11:29 AM
Translation: Those fortunate enough to be traversing Level3 inbound and outbound should be seeing very good performance. Anyone Sago reaches outbound via WilTel (WCG) is seeing heavy packet loss.
Posted by TrevorN, 11-14-2005, 11:37 AM
Thats probably why through this whole thing i've only had maybe 3 hours of total down time.
Posted by dreaz, 11-14-2005, 11:49 AM
Exactly. It's good that Sago is very fast when you need a quick fix, 15 minutes and it's done. But when you have 15+ hours of downtime in 2 days and need things to be fixed it appears that they forget to read the support tickets.
Posted by PhMatt, 11-14-2005, 12:20 PM
Currently we are working on a formal update to things, so that we can mass mail out clients. My inability to reach the sagonet forums kept me from posting last evening, but I'll be updating that shortly as well. Network admins aren't currently in the NOC, but are in the datacenter, but their phone conversations are limited to updating a few key personnel, and working with Foundry techs as continued tweaking is going on. Support tickets are all being replied to, and phone support when large volumes of calls are received do go to a voice mail system. Everyone is doing what they can to work on customer issues, routing, and support. As things subside and improve, support levels drop and quicker responses can be given. Thank you,
Posted by lifeonthego, 11-14-2005, 01:14 PM
Routers are still dropping packets like dealers droppping crack being chased by the cops...This is disastrous for me, even as just a reseller. Seems they must not like fantasy sports websites, as all of the problems as of seem to take up most of my Sundays now.... Hopefully this is over soon enough and I can begin the long process of kissing butt... PK
Posted by Def, 11-14-2005, 01:42 PM
Ever since we moved 3 of our servers to Sago back in March we've had problems. If it's not 36+ hours to replace a harddrive on one server, it's 24+ hours to replace a harddrive on another server, or it's something else. And now this. We're gone.
Posted by Sekweta, 11-14-2005, 02:08 PM
Ummmmm.... how long for a response? Our ticket has been open for hours and no replies at all. Meanwhile our customers are screaming. Hour after hour they are calling, asking when this is going to be over, and like chumps, we have to shrug our shoulders and say "we don't know". We continue to apologize profusely and tell them that we do care about their business, but don't have any answers. Our customers are really losing faith in us. Last edited by Sekweta; 11-14-2005 at 02:11 PM.
Posted by RyanD, 11-14-2005, 02:12 PM
Sure, they're getting answered just like my reboot ticket that was just closed by the tech without even reading it. Just pasted a generic network message in there. The body of the ticket clearly stated "THIS IS NOT RELATED TO THE NETWORK OUTAGE" Then I called to have it reopened and the guy on the phone was rude and kept saying my phone was breaking up... he said he'd reboot it... didn't I called back 40 minutes later and nothing had been done, they said they would reboot it again. 20 minutes later someone actually rebooted it. It's one thing to have scripted responses (good in a crisis) but you have to actually read the ticket before you apply a blanket response.
Posted by dreaz, 11-14-2005, 02:15 PM
I'm on the same boat.
Posted by sprintserve, 11-14-2005, 02:15 PM
ya. happened to me. then the 2nd reboot, called and no one picked up. packet loss is back btw.
Posted by HD Fanatic, 11-14-2005, 02:18 PM
0% packet loss here in NY. Both inbound and outbound are Level3.
Posted by kneadingu, 11-14-2005, 02:19 PM
I've emailed my trace and ping reports for the last hour as requested. However I'm still seeing 25-50% packet loss in your DC.
Posted by brooky, 11-14-2005, 03:38 PM
(can't delete - sorry)
Posted by Sekweta, 11-14-2005, 03:58 PM
37 hours after the upgrade, and we are seeing 20-45% packet loss depending on the destination. A few lucky destinations (primarily Road Runner) seem to be minimally affected, or not affected at all but that is the exception, not the rule. Our customers have torn us so many new a$$holes today, our backsides look like swiss cheeze.
Posted by nphase, 11-14-2005, 04:03 PM
same... i'm losing lots of credibility here as well.
Posted by Vaya, 11-14-2005, 04:03 PM
I'm now getting 7-30% packet loss. As soon as funds allow I'm jumping ship for a better server with a better provider, because a saving of a few quid a month is not a saving when this happens.
Posted by emills01, 11-14-2005, 05:03 PM
4:00 PM here and at least my sites are reachable, but still incredibly slow. While I don't resell my space, I still am losing out on ad revenue. I thought for sure by 4 PM things would be fixed, but this is ridiculous. Time to start shopping.
Posted by dreaz, 11-14-2005, 05:12 PM
Well, its been more than 20 hours with packet loss now. I've been waiting for a ticket reply for 5 hours. Way to go.
Posted by lifeonthego, 11-14-2005, 05:18 PM
I send an email to voice my concerns and am lucky enough to get a reply basically telling me I'm an a$$ for being pi$$ed but I still have nothing online. Seems mail is working fine, but port 80 seems to be F'ed up....and I'm the a$$.....Makes you wonder...Or maybe I'm not an a$$...I just don't know what the problem is....hmmm
Posted by lifeonthego, 11-14-2005, 05:34 PM
Anyone hear anything as of late when this issue may be repaired? Last update I got from Definitive was 'within the hour' at 9:50am this morning...Hey guys, think you missed your projection!
Posted by Sekweta, 11-14-2005, 05:57 PM
Haven't heard a thing, and it's not for lack of repeated updating of our ticket asking for status. We also can't get anyone on the phone who can tell us anything... other than they're "working on it" and it will be fixed "as soon as possible".
Posted by lifeonthego, 11-14-2005, 06:24 PM
After all this, it appears that the server I am on was the last to get its crap together...Oh well lets wait for the roundabout apology and the tech version of what really happened! PK
Posted by godboko71, 11-14-2005, 07:01 PM
Quote from sgoanet.com forums. http://sagonet.com/forums/showpost.p...&postcount=241 Though we are upset about the outage, I want to thank the techs and billing and everyone with being so helpful and nice. Anyway Matt thanks for trying your best to keep us up to date, though someone else who could get to the forums should have passed the update through. Just a note to everyone, stay calm, we all understand losing business, but getting mad at the run people (general support, billing, management) is not going to get you anything, or anywhere. Sago will work with you to make things right just be patient, and work with them not against them. Remember this, the billing, and phone/email techs have NO control over whats happening, believe me if you have ever worked in IT or in a call center you know this, remember that when talking emailing them. If you email support with pings and tracert set the title to something like [Do Not Delete] Ping Tracert... Then include your detailed ping and I would recommend sending 6-10 tracert's this will help them more then just a single tracert. You will also need to include your SSH log-in details so IP engineering can log in. If you send a ticket in that is not network related do something like [Not Network Related] Reboot (or what ever you need done. This will help the phone techs so they can do there job better/right. Anyway good luck to all
Posted by CLKeenan, 11-14-2005, 07:06 PM
Does anyone know if the mail sent to our address was lost in the mail queue? -Chris
Posted by Leperous, 11-14-2005, 07:41 PM
I'm still having 30% packet loss on one of my servers, and 10% on the other. Basically I've been down for almost 48 hours. Awesome.
Posted by CLKeenan, 11-14-2005, 07:46 PM
Yup... ive lost over $500 in revenue
Posted by web_hosting4u, 11-14-2005, 07:58 PM
I am currently experiencing more packet loss than earlier today. If it gets really bad I will do what was said above. Let me ask someone this, other than maybe dnsstuff.com and my own ISP, where does one run traceroutes? The site I use to use, no longer really provides any functional links. Thanks,
Posted by sprintserve, 11-14-2005, 08:17 PM
It depends on the failure reason. Quite a number of emails will be instantly failed due to DNS unable to resolve. If they can resolve and is just having issues connecting to the smtp server, then it will retry for about up to 7 days depending on the sending server configuration. (for example on our servers, it's 3 days)
Posted by Chuggles, 11-14-2005, 09:01 PM
I'm getting 50% packet loss again - this is ridiculous.
Posted by Mr.Muffinman, 11-14-2005, 09:12 PM
Some Packet Loss, That I am still experincing. Time: 8:11PM Eastern, USA Location : New Jersey, USA Internet Provider: Comcast Packet Loss: 0-25% Still! Mood: Sorry for the sagonet coustmers Thanks Mr.MM!
Posted by HD Fanatic, 11-14-2005, 09:21 PM
Things are looking smooth here. I am sticking with Sago, they have been very good to me up until this point. Hopefully, Sago will learn a lot from this disaster. I have faith in them, many upgrades/improvements are coming. Even though I'm very upset over this, they deserve another chance imho. Matt posted this on the sago forums:
Posted by holyway, 11-14-2005, 09:22 PM
I very want that everything became good in SAGO
Posted by kneadingu, 11-14-2005, 09:31 PM
tracert.org or something similiar to that ... I forget the exact URL. I'm showing 0% packet loss and less than 70ms on my bi coastal hops, which is a definite improvement.
Posted by TrevorN, 11-14-2005, 09:33 PM
When i'm trying to do traceroutes, pings, etc from multiple locations i use traceroute.org
Posted by RyanD, 11-14-2005, 11:00 PM
I'm still showing > 50% pl on a good number of our servers... this is not fixed
Posted by holyway, 11-14-2005, 11:04 PM
Where you from? I see your site well
Posted by godboko71, 11-14-2005, 11:07 PM
Where are you from and have you submitted the tracert and pings to sago yet? There IP engineers will look into it if you give them the info they need.
Posted by PhMatt, 11-14-2005, 11:40 PM
We wanted to take this opportunity to update you and fully explain the events that occurred on our network this past weekend. Part of operating a large scale growing living network is constant upgrades. Upgrades on the distribution and access layers of our network are quite common and generally uneventful. Upgrades to our core infrastructure are a major task and occur on a major level every 12-18 months. Our core is a fully meshed 60Gbps (6x10Gbps) backbone that directs our peering traffic out to two local carrier hotels. Before an upgrade is made to our core, the following steps are taken: 1.) 1 month out General design implementation and conference between our IP Engineering department and manufacture hardware and software experts. During this meeting, so-called roll back procedures are developed and median versions are established in the event of an emergency during the upgrade. Additionally, all major changes to our network are officially frozen until after the upgrade. 2.) 20 days out Test implementation using a standby device. For this task, we implement the suggested hardware and/or software changes on a standby switch router and mirror our traffic on this device. Inbound/outbound route and traffic continuity is tested at this point. 3.) 10 days out Second conference with manufacture to finalize any potential version mismatches and finalize manufacturers commitment to our upgrade as they recommend 4.) 1 day out Second mirror test to ensure no conflicts exist in new switch router software. For this upgrade, all of the above tasks were completed. During step #2, Foundry supplied us with a tweaked version of the operating system for both the switch chassis and the individual 10Gbps cards that handle our long haul transport. On this same CD, they supplied us with a revised hardcopy of the current OS on the switches. This was to act as our emergency roll back procedure. The version of software currently running on these NetIron40Gs was implemented by Foundry and required a live patch to operate properly. The events of the upgrade are as follows: 1.) Upgrade began. All traffic was failed to our secondary core infrastructure while the primary devices were removed from production and upgraded. 2.) Upgrade to primary core infrastructure was completed 3.) Traffic was failed 100% to the primary core infrastructure. QoS samples were taken via IronView and SolarWinds. Minimum latency, packet loss, and SNMP thresholds were deemed acceptable. 4.) Secondary core was upgraded. 5.) Traffic was moved 100% to secondary core infrastructure. QoS samples were taken via IronView and SolarWinds. Minimum latency, packet loss, and SNMP thresholds were deemed acceptable. 6.) The above simulated a failure on the network (eg, 100% to primary or secondary infrastructure). The next test performed was to release the network as normally operated, therefore, traffic was balanced based on normal operating preferences to the two core devices, essentially 50/50. QoS samples were taken via IronView and SolarWinds. Minimum latency, packet loss, and SNMP thresholds were deemed acceptable. At this point, the core upgrade was considered completed. A few more non service impacting but high risk housekeeping items were completed, specially the installation and migration to some new fiber transport within our facility to accommodate the opening of our new DC3 in Tampa. 7.) QoS measurements were once again taken, at this time our technicians noticed very moderate packet loss and what was deemed at the time to be a slow memory leak. 8.) Foundry was also monitoring the boxes and had begun working the issue. They initially attributed this to an affiliation issue via the OSPF downstream and HSRP lateral relationship between the core clusters. Because of this, all further maintenance was suspended within the window. 9.) The perceived memory leak worsened dramatically over the next hour. Foundry requested that we connect a standby non production switch via a mirror while failing one core out of production, performing a restart, and seeing if this temporarily stopped the leak. 10.) The reboot was performed, but the situation almost instantly became just as severe. Processor load on the outbound 10Gbps interface line cards was pegged at 99% on the rebooted core, and 90% on the unrebooted core. 11.) Processor load intensified to 99% on all outbound 10Gbps interfaces, >90% on the internal interfaces, and 99% on the switch management modules. 12.) Foundry was given 15 minutes to propose a course of action, but since no satisfactory course of action was forthcoming, the decision was made internally to revert back to the old code distributed on the pre-upgrade CD 13.) After steps 1-10 above were repeated, in reverse, traffic stabilized for less than 10 minutes. Amazingly, the high CPU load continued on previously operational code. 14.) At approximately 11am EST Sunday, on the urging on the manufacture, the routers were reconfigured and reinstalled using old configs and old code, however, the problem continued to manifest itself after a few minutes of operation. 15.) Sago management requested that the old patched software be loaded on the routers. 16.) Steps 1-10 above were commenced to load the old patched OS on the routers. The first to be upgraded failed and we found the ASICs refused the old code completely. Foundry investigated this issue and determined that the software upgrade implemented earlier apparently included a firmware upgrade that WAS NOT mentioned during the preupgrade meetings by Foundry. 17.) Foundry committed to produce code that would recreate the older firmware on the 10Gbps cards, thereby allowing a reload of the old patched software instead of the old software provided by Foundry on CD. 18.) While waiting for the production of this software, a Sago engineer noticed that no memory was allocated to IPv4 routes. Further, no CAM page files were devoted to storing routes or buffering for IPv4. 19.) Foundry investigated this finding and determined that only IPv6 allocations were being made. It was acknowledged that this was a typographical error made during the customer production of our software and existed in both the old and new software versions provided on CD. 20.) A conference call was held, and Foundry determined that given currently available resources, fixing the memory allocation issues was more feasible in a short timeframe. We elected to take their advice. The first software revisions became available at approximately 9PM EST Sunday and were implemented within the hour by following steps 1-10 above. 21.) 3 more updates/fixes became available throughout the night. The last was implemented around 6AM EST Monday. 22.) These each gradually reduced the persistent packetloss issues. 23.) The final packet loss issue involved an improperly computed equation causing an irreversible imbalanced on only 2 single Gbps outbound connection. This caused contained packetloss for outbound routes preferring those connections. Essentially, the switches were computing that connection pipes were larger than 1Gbps and trying to force more traffic out them than was feasibly possible. 24.) This fix was implement between 8PM and 9PM this evening. At this time, all issues are resolved. Any customers still experiencing problems should immediately contact support@sagonet.com as your issue is unrelated to anything above. We sincerely apologize for the obvious problems this has caused you. Our networks performance during this incident is unacceptable and contrary to the normal way our company operates. A relentless investigation into why this occurred will continue to ensure that our customers are never subjected to such an incident in the future. The following changes have already been implemented: 1.) Mandated the presence of a third-party auditor to review any proposed changes to our core infrastructure by either Sago or any of our manufacturers. We rely heavily on input from the experts that make the network gear we use, but that strategy utterly failed us in this incident. A search was begun today for a firm capable of readily assisting us with a network as robust and large as ours. 2.) A full audit by the above mentioned firm by the end of the month of all custom software provided by or implemented by any manufacturer in our network. 3.) A requirement that any manufacturer providing onsite or telephone assistance for a schedule network window include enough personnel to reserve a 24hr shift of engineers that were available during the planning phases of an upgrade. Fatigue was an issue yesterday. 4.) An internal requirement that no more than 50% of senior engineers be involved in any single task within an 8hr period. Again, this is to combat fatigue. 5.) While we have standby devices onsite, we are now requiring any mirrored traffic tests to include all devices at that layer in the network interacting with each other. 6.) A purchase was made today for packet generating hardware capable of simulating our load for testing purposes. Previously, packet generation was done on a scaled down basis to test mostly DDoS related safeguards prior to implementation. This incident, like all our upgrades, was planned in excruciating detail. The upgrade was supposed to be a short, painless operation and it was widely believed by both Sago and Foundry engineers that this could be accomplished without any impact to customer operations. A wide network upgrade window was only scheduled as a formality, given the gravity of the changes being made. We cannot stress enough that everything in our power will be done to ensure there is never a recurrence of this event. Network performance as of late has not been near our companys expectations or capabilities and it is impossible to express the level of commitment our staff has to showing how well our network will perform in the future. As many of you are wondering, there was a reason for this upgrade. While many of these upgrades were slated to be announced later as we DO NOT HAVE DEFINITIVE INSTALLATION TIMEFRAMES (please read the emphasis on this) from our transport carriers, these changes will be implemented beginning December 1st and finished by a projected end of Q1 06. So, as an early announcement, please know that we are implementing many of the following projects: 1.) Local peering for our Tampa datacenter with 17 other carriers, including 3 backbones. 2.) Transport between our Tampa facility and new 100,000 square foot Atlanta facility 3.) Merger of our Tampa and Atlanta Datacenter networks via dark fiber. 4.) Addition of 4 of the following backbones (final announcement will be made later this week): Savvis, Global Crossing, BTN, Telsia, and/or Cogent (for onnet Cogent traffic ONLY). 5.) Establishment of peering with 10+ other providers and 2+ carriers in Atlanta. The following project is currently underway with an unknown timeline: Implementation of a hard line network to New York City 60 Hudson to establish peering and better European transport. Once completed, these changes will give us what we feel is one of the most stable and high performance foot prints of any provider in our competitive spectrum. No amount of announcements or press releases will prove this to you after the events of the last few days, only performance. That will be our primary objective over the coming weeks and months. In the meantime, we can only offer our assurances and continue to update you as to our findings related to this incident. Our extensive apologies can only go so far, so we look forward to the opportunity to prove our abilities to you. If you have any questions, please contact us.
Posted by nero-x, 11-15-2005, 12:03 AM
so because this is "foundrys fault" i guess you arent responsable for your "uptime guarantee" ?
Posted by RyanD, 11-15-2005, 12:03 AM
beyond the appology are there any plans for compensation or is it a too bad so sad situation... ?
Posted by dreaz, 11-15-2005, 12:33 AM
I've been waiting 12 hours. 12 hours. For a reply on my ticket. What have they done? Closed the ticket, status: resolved. Not only they give you fake ETA's (when you go after it, since they will not contact you at all), but they refuse to compensate you, provide the most crappy support you could expect, but they also pretend nothing is happening? [edit] no more packet loss. at last. one less excuse to the lack of support [/edit]
Posted by bjn, 11-15-2005, 12:42 AM
this is the reply that i (and others) have received, when we asked them about it: this was sent sometime this afternoon, while the problem was still ongoing.
Posted by PhMatt, 11-15-2005, 12:49 AM
I believe you were in contact with Adam already, but please submit your request to billing@sagonet.com for any SLA violation. Thank you,
Posted by PhMatt, 11-15-2005, 12:52 AM
No one has said that, but many implied comments were made, such as lack of redundant routing equipment, insufficient testing, why rollbacks weren't done, etc. Releasing a full account of the preparation, upgrade, problems, attempted resolutions, and final fixes was all that could be done as far as explanations were concerned. All too often people tend to make guesses about why people don't do what they would do, without any information as to what was done, and to what lengths people went to get things done.
Posted by sprintserve, 11-15-2005, 01:54 AM
Implied comments were made because of the lack of official information, consistently. So this official explanation is a good start.
Posted by CLKeenan, 11-15-2005, 02:27 AM
I'm still down, even after being told my server was back up. Could anyone confirm that http://www.course-notes.org is down or up? Please post where you are accessing from. Thanks, -Chris
Posted by holyway, 11-15-2005, 02:32 AM
I can tracerout tou you site, but don't see anything in browser.
Posted by HD Fanatic, 11-15-2005, 02:38 AM
your website seems to have a problem, however your blog on your site works. I suppose it's a software issue on your server.
Posted by kneadingu, 11-15-2005, 07:34 AM
Blank page in the browser for me.
Posted by Reseller-Center, 11-15-2005, 07:47 AM
Down for me 2. Blank Page.
Posted by Platypus101, 11-15-2005, 08:40 AM
Is Sagonet/Allmanaged support sleeping??? They said they have to reinstall MySQL on my server 3 hours ago and they did... NOTHING. All sites on my server are down.. 4 hours and counting....
Posted by pietjepuk, 11-15-2005, 08:56 AM
Appears down here, trying to access from The Netherlands.
Posted by Platypus101, 11-15-2005, 10:26 AM
Problem solved thanks to Jason.. Ok, guys if you need anything related to Sagonet/Allmanaged contact Sago-Jay.
Posted by CLKeenan, 11-15-2005, 07:47 PM
My site is still down too.... I'm getting REAL pissed. Like I've just lost $XXX revenue + credibility. -Chris
Posted by Ryan Smith, 11-15-2005, 07:49 PM
Down for me - Philadelphia PA
Posted by pietjepuk, 11-15-2005, 07:49 PM
But pinging works, and it does respond to port 2086 too. It seems some software is messed up on it.
Posted by HD Fanatic, 11-15-2005, 10:02 PM
Dude, as I've said earlier, the blog on course-notes.org works, just the main domain doesn't work, seems like a config. problem... http://blog.course-notes.org/ WORKS http://www.course-notes.org/ DOES NOT WORK
Posted by CLKeenan, 11-16-2005, 01:26 AM
Ok, Im running Mambo CMS for the main site. Could that be a problem? Any ideas? Thanks, -Chris
Posted by sprintserve, 11-16-2005, 04:41 AM
Did you try opening a ticket? In any case, this is totally unrelated to the outage. Please contact their support or start a new ticket seeking help from others in the technical forum.
Posted by miracle, 11-18-2005, 11:02 AM
Great packet loss to Sagonet again. Is there something wrong with their network now?
Posted by phactor, 11-18-2005, 11:33 AM
everything fine here
Posted by nphase, 11-24-2005, 12:50 PM
Are they ever going to release an SLA refund notice?! I am dying here waiting for my provider through sagonet to give me a refund and he keeps pushing me away or not responding to my emails and messages etc, I'm not quite sure how to handle this besides to move servers at the end of the month. 37 hours of downtime and then more random packet loss is not something a growing business can handle.
Posted by web_hosting4u, 11-24-2005, 12:57 PM
Are you actually a customer of SagoNet or AllManaged? Dedicated or Shared? If you are waiting for your host, who may or may not have services through SagoNet or AllManaged you will have to contact the sub in regards to such a request, if you are a customer of SagoNet or AllManaged it was specified in one of their posts at http://forums.sagonet.com to contact the billing dept. directly with server information and so forth for review from the executive board. Hope this helps.
Posted by nphase, 11-24-2005, 01:30 PM
See thats the thing, I am a dedicated customer of what seems to be one of their resellers. the company that hosts me has a box with allmanaged that has it leased to me. They said they are "waiting for a sagonet decision" and that its not up to them to give me a refund. This may be the case, but they are a server provider with spots all over the us.. if you go to their site ( www.server4sale.com ), they guarantee 99.5% uptime, 30min response guarantee, and 24/7 live support with proactive monitoring. all three of these offers has been broken on their end multiple times [especially the 99.5% uptime.] I can't afford that kind of downtime, my clients have lost a lot of money and I've had to dig out of my own pocket to cover losses. There is no way I'm not getting a refund after all that. Funny thing is, i ordered this server to make sure there was no downtime due to load (it's a dual xeon). slingshot, do you offhand know which post it specifically is? that way I could forward it to them and tell them to get moving.
Posted by sprintserve, 11-24-2005, 05:01 PM
Your contract is with server4sale and not with Sago. Your host should be upholding their SLA regardless of Sago's decision.
Posted by nphase, 11-24-2005, 08:28 PM
right, but they keep saying that theyre waiting on Sago's response to give me a refund.
Posted by SoftWareRevue, 11-24-2005, 10:29 PM
Then move on. It is they that have not upheld their SLA to you. Them telling you to "wait . . ." should be a warning of what else they would make you wait for. If they offer an SLA, they offer it. They don't offer it contingent upon it being upheld by their upstream. (Of course, I could be wrong. Maybe they do.) Your posts have nothing to do with this thread. They have to do with your provider. Your provider is not sago. Thread Closed.