WiredTree down

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > WiredTree down

Posted by diesel12, 01-17-2011, 01:06 PM
Noticed our dedi offline at 10:00 AM Mountain, WiredTree.com down and their phones are busy. Any feedback or info is greatly appreciated!
Posted by frki1, 01-17-2011, 01:08 PM
Same for me... What is their phone number?
Posted by AS-Richard, 01-17-2011, 01:08 PM
Down here as well.
Posted by azn_romeo_4u, 01-17-2011, 01:08 PM
Yup yup, can confirm that it's down. At 9am PST
Posted by diesel12, 01-17-2011, 01:09 PM
No luck with phone, but if there's an issue, they're probably barely getting a chance to ascertain what it might be this early on: 866-523-8733. The bad news is it's Monday morning, worst possible timeframe for downtime. The good news is it's a holiday. At least in the US.
Posted by frki1, 01-17-2011, 01:10 PM
Tnx for the phone ill try contact them. Well this is now happening pretty often
Posted by RamboLeer, 01-17-2011, 01:12 PM
Yes, same thing with me... need to start looking elsewhere I guess
Posted by Honey Bear, 01-17-2011, 01:12 PM
Wired Tree is down, our server is down, and I don't even want to think about the amount of money I'm losing with all the downtime this month!
Posted by Zealus.com, 01-17-2011, 01:12 PM
Their Grove panel opens up, but extremely slow
Posted by mrl14, 01-17-2011, 01:13 PM
CoreSite is up...must be related to just their boxes in the DC.
Posted by Kevin K, 01-17-2011, 01:13 PM
Yup here we go again. This is the final staw, the last few months there has been a large number of outages. It's getting to be too much. Time to move on down the road...
Posted by A Grateful Dad, 01-17-2011, 01:14 PM
Yup, down again! This is getting a little long in the tooth now! Time to look elsewhere I am thinking!
Posted by frki1, 01-17-2011, 01:15 PM
Unfortunatly maybe it is time for that...aftert one year, but this is a lot of downtime in last days
Posted by ellinas, 01-17-2011, 01:15 PM
3 of my 4 VPS down. 1 Dedicated down... Time to start shopping.
Posted by JohnDonovan, 01-17-2011, 01:16 PM
My dedicated server is down as-well. Looks to be in the network, based on traceroute. I'm not even getting out of LAX
Posted by stormy, 01-17-2011, 01:16 PM
I have two servers down, plus Grove and the main Wiredtree site are down as well for me...
Posted by frki1, 01-17-2011, 01:17 PM
Must be some network error as we cannot acces grove or WT site also
Posted by hdsrob, 01-17-2011, 01:18 PM
Down here as well. Pingdom alert shows 12.00.03 EST
Posted by Jedito, 01-17-2011, 01:19 PM
5 Dedicated down
Posted by ellinas, 01-17-2011, 01:20 PM
2 of the VPSs just came up now. Stil dedicated down and 1 VPS. ... Later edit... dedicated up again.
Posted by Zealus.com, 01-17-2011, 01:20 PM
My dedi is us
Posted by mrl14, 01-17-2011, 01:21 PM
We are back up
Posted by azn_romeo_4u, 01-17-2011, 01:21 PM
20 minutes and counting. Might as well go renewal my drivers license now since I can't work. Was hoping to put it off for a few more days. I hope my server didn't crash. Edit. Seems to be up now.
Posted by Sam.Serverfreak, 01-17-2011, 01:22 PM
Is back online now.
Posted by JohnDonovan, 01-17-2011, 01:22 PM
I'm back up as-well. 1 dedicated
Posted by DGRing, 01-17-2011, 01:22 PM
Back up now on East coast.
Posted by frki1, 01-17-2011, 01:22 PM
We are up...also WT site
Posted by eva2000, 01-17-2011, 01:23 PM
Yup looks to be down got my site down email alerts from several providers. I checked my email they had a network maintenance email for Jan 19th for a new distribution switch to be added. But it's only the 18th (AEST), so not sure.
Posted by UnderHost, 01-17-2011, 01:23 PM
Back online, 21 minute of downtime.. We are hosting our business website on wiredtree in case we have problem with our networks our CustomerPanel is still online for client but lately the networks isn't stable as it was... time to move before it goes worst. Last edited by UnderHost; 01-17-2011 at 01:29 PM.
Posted by sfoma, 01-17-2011, 01:25 PM
We are backup, down around 20 minutes.
Posted by LiquidR1, 01-17-2011, 01:25 PM
This is just ridiculous. Website down, wiredtree down, not good. We were down a good 6 hours the day after Christmas too, due to "server" issues. I'm betting this time it's a DOS attack.
Posted by DaveDD, 01-17-2011, 01:26 PM
Our ded just came back up, but I think this is the final straw. This is our 4th or 5th outage in the past 12 months. On top of that, we've got Groupon's running today in Philadelphia and Fresno. Great timing!
Posted by sross, 01-17-2011, 01:29 PM
Looks like their DC has been letting them down lately. I trust they'll crack some heads and get it sorted out. I have servers at other hosts and the grass is not always greener to say the least.
Posted by Kevin K, 01-17-2011, 01:29 PM
Seems like this many outages in the last few months that I have noticed. I gave them so many chances, but they used them all up. I will be moving our services out of their business over the next few days.
Posted by diesel12, 01-17-2011, 01:31 PM
Since all lines of communication have been down, a single tweet or update here would go a LONG way to helping us communicate to our clients what is going on. A half hour should be long enough to post a tweet, regardless of having to put out fires, no?
Posted by frki1, 01-17-2011, 01:31 PM
Same situation here, i am thinking also about something new cos my experience says to me when troubles starts this offten it will just get worst. Any suggestions for good VPS, network stability is crucial?
Posted by frki1, 01-17-2011, 01:34 PM
Monstermegs still dont work for me? Is that page on WT also?
Posted by Kevin K, 01-17-2011, 01:35 PM
Yes it is. I think we are going to be moving it to the Gigenet cloud later today or tomorrow.
Posted by frki1, 01-17-2011, 01:36 PM
Wiered, ours page on WT started to work normaly so i thought that everybody is up.
Posted by Jedito, 01-17-2011, 01:37 PM
2 up, 3 still down
Posted by transmit, 01-17-2011, 01:37 PM
Our $800+ dedicated server is still down.
Posted by okihost, 01-17-2011, 01:38 PM
Yes they are still having issues seems dependent on where/who you are trying to access your sites from. Second month with them, third downtime including the long one in December. I had about the same amount of downtime with EV1 over 8 years, seems like moving was the wrong choice even though they got great raves here.
Posted by Kevin K, 01-17-2011, 01:39 PM
I agree, that was the first place I went to check for a report since I had already read their site was down. That is the first thing we did was post a tweet to let customers know that our website was down. Took a whole 20 seconds to post...
Posted by spiritmaker, 01-17-2011, 01:41 PM
Just spoke with them, they had technicians down at the data centre, apparently there were power issues. My sites are backup already.
Posted by BirdBrained, 01-17-2011, 01:41 PM
1 dedicated down, soon to be out.
Posted by diesel12, 01-17-2011, 01:42 PM
Seeing sites come up on West Coast, exactly 40 minutes after going down. I'm almost more disappointed in the complete refusal to communicate via WHT or Twitter a single time to give clients an update. No excuse.
Posted by transmit, 01-17-2011, 01:42 PM
For those of you whose servers came back up, did the uptime get reset? In other words, was it a network issue or did they have to actually power servers back online?
Posted by Kevin K, 01-17-2011, 01:42 PM
I think they need to get those hamsters running a little faster today...
Posted by UnderEstimated, 01-17-2011, 01:42 PM
Its been more then half an hour WiredTree gone down only their site is working now. The servers are down and they said that power issues and they are resolved should come back momentarily. There phones are busy too anyone else have problems rite now?
Posted by frki1, 01-17-2011, 01:43 PM
Yup, no information which is not ok. One twitt would be nice, juts to say what is happening..
Posted by RamboLeer, 01-17-2011, 01:43 PM
Up now but still unacceptable downtimes. Anyone know of any good VPS providers on East Coast?
Posted by BloggerDude, 01-17-2011, 01:44 PM
what is going on?? again this problem?? we had a huge problem not long ago, again all down?? same here, all my sites are down...
Posted by spiritmaker, 01-17-2011, 01:44 PM
Total downtime for my account: 1 hour, 4 minutes, 11 seconds. This is I believe my second downtime in close to two years of service. All in all I always receive top notch support and very quick responses.
Posted by BloggerDude, 01-17-2011, 01:45 PM
OH NOOO !! not again!!!! This is happening VERY VERY OFTEN lately... What is going on now??? Wiredtree is always down, man.. This is impossible... how can we build reliable sites???
Posted by BlogButler, 01-17-2011, 01:47 PM
I got a reply from support saying they are having a power outage in the datacenter. Seems similar to the one on Dec 11.
Posted by mrl14, 01-17-2011, 01:49 PM
finally a twitter update CoreSite power issue again which effected same circuits (and which there was already maintenance to replace).All servers have power restored
Posted by jpeacock, 01-17-2011, 01:50 PM
just logged into my VPS and uptime was still at 50 days. seems like power or network issue then (power from the sound of someone else's post)
Posted by diesel12, 01-17-2011, 01:50 PM
If this is in fact a power outage at datacenter, this would be the THIRD major issue in the past 12 to 18 months. One was during a systems test where the backup generators failed and the second was a month or so ago when power was lost.
Posted by okihost, 01-17-2011, 01:51 PM
Not mine still down
Posted by BloggerDude, 01-17-2011, 01:51 PM
I actually have another VPS with another hosting company, that VPS is secondary. Maybe I will think to make that one primary and the wiredtree's secondary. But the guys are WT are obviously missing something here, it can't be possible that we have this kind of problem so often.
Posted by WiredTree James, 01-17-2011, 01:53 PM
Thank you everyone for your patience. We have been communicating with everyone via phone and support tickets just as fast as possible. We had a single circuit trip at the CoreSite facility this morning about 35-45 minutes ago. While we are waiting to hear back from CoreSite employees on the reason, we have powered up any affected servers and have a group of our admins manually checking the servers one by one to make sure they are all ok and online. As a side note, Zac is currently unavailable on a business trip and while you don't normally hear from me on WHT, I would be happy to answer any questions you may have. If you are still experiencing any issues, please submit a ticket and we will respond to you as quickly as possible. James Webb Customer Service Manager WiredTree
Posted by Kevin K, 01-17-2011, 01:53 PM
Our servers are finally starting to come back up. Time to get to work and get things ready for a move....
Posted by DaveDD, 01-17-2011, 01:54 PM
Yeah...I think this is it for us. We definitely can't afford these outages. What's worse is every time this happens their phone lines are busy and their site is down, so you can't even submit a ticket. I just spoke to GoRack.net. I've been talking to them about getting a redundant solution worked out. They're local for me and they've got some of the best redundancy I've ever seen. Looks like it was a network outage. My server wasn't restarted.
Posted by Jedito, 01-17-2011, 01:54 PM
Hello James Are you aware of the existence of twitter? why nobody updated anything there, why we are still on the dark for almost an hour?
Posted by LiquidR1, 01-17-2011, 01:56 PM
Going on 90 minutes now, our website is still down.
Posted by Cyandin, 01-17-2011, 01:58 PM
Uh oh. I hope everything is back up asap!
Posted by diesel12, 01-17-2011, 02:01 PM
So the same coresite issue that happened a month ago (can't find date as WiredTree removed the blog update from Grove from around December 10th ish?) happened again? A month wasn't long enough for Coresite to make the promised upgrades? If this is the case, this issue can and will occur over and over at any time until the root issue (I believe there was a grounding issue) is fixed.
Posted by okihost, 01-17-2011, 02:01 PM
Mine are back, as someone else said with twitter there is really no excuse to update that if you are already utilizing twitter. It will cut down on your calls/tickets during an outage and help us breath a little better knowing it is being looked into already.
Posted by WiredTree James, 01-17-2011, 02:01 PM
Jedito, I apologize for the lack of twitter update. Zac, was unavailable to jump on twitter for you and we opted to concentrate on bringing everyone online just as quick as possible. We started seeing servers come back online just 15 minutes after the issue at the DC. We will be providing both twitter and thread updates from here on on out. I will continue to communicate with our techs and provide everyone here with any updated news. James
Posted by Kevin K, 01-17-2011, 02:04 PM
So you are telling me that only one person from your company can post on Twitter, give me a break. That should be shared with other staff around your company. Especially if he is on a planned vacation.
Posted by WiredTree James, 01-17-2011, 02:05 PM
I apologize for this. We are now going to be sending out twitter and WHT updates as quickly as possible. You should have seen your server go up very quickly after the outage. If you have not then it is most likely due to an FSCK. Our staff is handling tickets very quickly. I would highly advise sending in a ticket to get the status on your exact server. We have enough staff on hand to take care of everyone. James
Posted by jrandall, 01-17-2011, 02:06 PM
Still down for me. Edit: Well what do you know? The second I switch tabs to refresh it's up! Uptime: 29min 57secs Last edited by jrandall; 01-17-2011 at 02:13 PM.
Posted by BlogButler, 01-17-2011, 02:08 PM
It was on 12/11/2010
Posted by BloggerDude, 01-17-2011, 02:08 PM
My sites are still down!!!! Why do you say it's up?? Internal Server Error.
Posted by WiredTree James, 01-17-2011, 02:08 PM
jrandall, private message me your hostname and I will provide you a status. James
Posted by Jedito, 01-17-2011, 02:10 PM
James, you can't receive PM.
Posted by Jedito, 01-17-2011, 02:10 PM
By the way, none of the servers that were down, were powercycled.
Posted by BlogButler, 01-17-2011, 02:12 PM
James - I can't PM you either.
Posted by racersimage, 01-17-2011, 02:13 PM
My site has been down for over an hour now and the response from support was that there was a power issue at the datacenter. This would have to be the same datacenter that experience the problem last time, I thought they got the problem fixed?
Posted by CreedFeed, 01-17-2011, 02:15 PM
My server still down too. I sent a ticket 10 mins ago.
Posted by transmit, 01-17-2011, 02:20 PM
Interesting! Our server just came back up, and it was power-cycled.
Posted by BloggerDude, 01-17-2011, 02:21 PM
sent ticket 40 minutes ago... my sites are down.. please don't say that everything is up.. because my sites are down and the ticket of 40 mins has no response..
Posted by Jedito, 01-17-2011, 02:24 PM
I just double check, and 2 were powercycled and 2 not, one is still down, so I guess that's fscking, but since I did not received answer to my ticket, I can only assume it.
Posted by WiredTree James, 01-17-2011, 02:29 PM
Updated to premium membership (about time since I read through WHT all the time). I popped you a PM Jedito. James
Posted by diesel12, 01-17-2011, 02:31 PM
So my takeaway from this is that the issue was a repeat of the grounding issue at coresite from a month ago, for which planned maintenance has stil not taken place over a month later. Forget date of initial issue as WiredTree has removed the blog post update in Grove about initial incident. If this is in fact a second occurrence of downtime related to coresite grounding issue, then it can occur over and over until the core issue is fixed. Last edited by diesel12; 01-17-2011 at 02:43 PM.
Posted by BlogButler, 01-17-2011, 02:33 PM
Thanks, I have more than 5 posts but still can't PM. Is there a time restriction as well? Last edited by BlogButler; 01-17-2011 at 02:36 PM.
Posted by BlogButler, 01-17-2011, 02:38 PM
My VPS has been down 94 minutes now. I've sent a support request for a status specifically on my VPS but no response yet. Ticket number is #NNA-455075
Posted by WiredTree James, 01-17-2011, 02:48 PM
Jedito, I responded to your PM. Blogbutler, Thank you for the ticket ID. I am pulling a status for your VPS server now. James
Posted by BlogButler, 01-17-2011, 02:54 PM
Thanks James. Going on 120 minutes now.
Posted by WiredTree James, 01-17-2011, 02:56 PM
I just wanted to update everyone and let you know if your ticket has not been answered yet, it should be shortly. Our Support queue is getting shorter by the second and in addition to the staff we would normally have on hand, I have called in many additional hands to help out. Thank you everyone for your patience and I will continue to refresh this thread so that questions can be answered. James
Posted by diesel12, 01-17-2011, 02:59 PM
@james: What is preventing this issue from happening a third time in as many months? It has been over a month since maintenance to fix the root cause was planned. What date is this maintenance planned for?
Posted by Harzem, 01-17-2011, 03:03 PM
WiredTree had posted a maintenance announcement 3 days ago for Wednesday, January 19th, 2011 between 11:30PM and 5:00AM CDT(05:30 - 11:00 GMT) I don't know if it's related to this.
Posted by CreedFeed, 01-17-2011, 03:04 PM
I just received a response to my ticket, but it's a canned, generic response and gives me no status on my exact server. Ticket GSB-575312
Posted by VidiJoe, 01-17-2011, 03:04 PM
Others reporting they are up but we are still down - 3 dedicated and 2 vps Ticket: ABN-533062 Last edited by VidiJoe; 01-17-2011 at 03:07 PM.
Posted by BlogButler, 01-17-2011, 03:05 PM
my VPS is up but sites are still down on it. WHM and CPanel are all down still. Response time on my ssh connection is very very sluggish.
Posted by WiredTree James, 01-17-2011, 03:14 PM
CreedFeed, I just responded to your ticket and made sure it was escalated into our outage queue. We are waiting to hear back from our techs/admins on that particular node and should have an update for you very shortly. The last I heard was that it was in the process of booting but I will have more solid information for you shortly.
Posted by Jedito, 01-17-2011, 03:17 PM
James, can you give me the status of the fsck on the server? %?
Posted by WiredTree James, 01-17-2011, 03:17 PM
Thank you for your patience with that ticket. I am seeing that all of your servers are reporting as online. I did go ahead and respond to your ticket. If you are still experience problems, just update the ticket and we will respond to you quickly Joe.
Posted by WiredTree James, 01-17-2011, 03:19 PM
This maintenance was due to upgrades of new equipment on our side. What happened today was due to CoreSite. We will release more information shortly as we receive it from CoreSite.
Posted by WiredTree James, 01-17-2011, 03:22 PM
This may be the case as things settle on the VPS node after reboot. You can expect this to get better as the minutes move on. Please be aware though as an automatic safety feature, your container may undergo a quotacheck during which it would be offline for just a few minutes.
Posted by LiquidR1, 01-17-2011, 03:24 PM
I got the same canned generic response. Going on close to 3 hours now. Ticket: UPF-962189
Posted by diesel12, 01-17-2011, 03:24 PM
@james: What is preventing this issue from happening a third time? It has been over a month since maintenance to fix the root cause was planned. What date is this maintenance planned for?
Posted by WiredTree James, 01-17-2011, 03:25 PM
Jedito, Sure, your server is still fscking /home right now. I am checking for a percentage with our staff and if one is available I will update your PM. Thanks, James
Posted by WiredTree James, 01-17-2011, 03:30 PM
Because of the last issue they had, CoreSite had previously planned a major maintenance to take place on the 31st of January. This would be replacing breakers, moving our circuits and more. Our maintenance email concerning this was planned to go out 7 days prior to the maintenance. This maintenance will still take place and will prevent this issue from repeating itself.
Posted by VidiJoe, 01-17-2011, 03:30 PM
Thanks James, looks like everything is up and running again
Posted by WiredTree James, 01-17-2011, 03:35 PM
The first response was indeed canned to let you know what we know and that we are looking into the issue. I see that your VPS node is booting and fscking at the moment. I have an admin reporting back to me regularly on this node and we will keep you updated. Thank you for submitting a ticket so that we can more easily provide you with an update. I have gone ahead and updated that ticket for you. James
Posted by diesel12, 01-17-2011, 03:36 PM
Thank you for clarification! Last edited by diesel12; 01-17-2011 at 03:49 PM.
Posted by WiredTree James, 01-17-2011, 03:38 PM
Fantastic Joe. I assure you and everyone else that we will continue to work our butts off and will be handling things as necessary with Coresite. James
Posted by CreedFeed, 01-17-2011, 03:41 PM
My VPS is up as well. Thanks for the updates James.
Posted by BlogButler, 01-17-2011, 03:41 PM
James, my server is back offline - cannot connect via ssh. Can I get % of fsck or quota check?
Posted by WiredTree James, 01-17-2011, 03:46 PM
There is a maintenance window on January 31st (1/31/11) as mentioned that will be covering significant replacement of new power hardware. This was already planned by Coresite prior to the issue today and will be emailed the week of the 24th as soon as we have the specific times involved for the maintenance. If you have any questions about this, just let me know. I am going to check into maintenance schedules but I am unaware of any maintenance that was missed on 12/31/10. Thanks, James
Posted by WiredTree James, 01-17-2011, 03:47 PM
Steve, You are welcome. Just update your ticket if you need anything else. We are decimating this support queue with all the extra help. James
Posted by diesel12, 01-17-2011, 03:48 PM
@James: thank you, misread and thought you meant 12/31 not 1/31 since you mentioned they "had planed a major maintenance" as past tense. Since the maintenance is yet to occur, I was incorrect about any maintenance not occurring. You gave me the info I was looking for. Revising original post to reflect. Last edited by diesel12; 01-17-2011 at 03:53 PM.
Posted by Kevin K, 01-17-2011, 03:52 PM
Both servers were up for over an hour running just fine. Now all of a sudden one of them just went offline again. I have had it with Wiredtree. I am already on deploying new server in other locations to move these servers to. One too many down times in few months for me. I can only be so patient.
Posted by BlogButler, 01-17-2011, 03:56 PM
Everything is back for me now. Total downtime for my sites was 169 minutes, beginning at 11:03am Central. Thanks James for providing status.
Posted by andrewtees, 01-17-2011, 03:56 PM
James, We are nearing 3 hours of downtime for one of our dedi's. Can you please provide status of ticket # YMM-886936
Posted by WiredTree James, 01-17-2011, 03:57 PM
Sure. I see that you are back online now with this VPS and that WHM is responding. It may be sluggish for a bit but you should be good to go. Feel free to update your ticket if you need anything else. Thanks again to you and everyone else while I try to courier information from our staff.
Posted by WiredTree James, 01-17-2011, 04:03 PM
I just checked your server and see that your VPS is online but that we are waiting for your dedi to come back up. I am waiting for a status from our staff and will update you in just the next few minutes Andrew. You can expect a reply asap in your ticket.
Posted by WiredTree James, 01-17-2011, 04:07 PM
Kevin, Do you have a specific ticket I can use to help track down your issue. While I am not familiar with your account yet, it sounds like you might be experiencing a quotacheck on a node. This is expected for VZ nodes when they reboot and usually only lasts a few minutes. I can provide you with more details though if you share a ticket ID or hostname.
Posted by Kevin K, 01-17-2011, 04:11 PM
The server came back up a few minutes ago. I have not opened any tickets since it would not accomplish anything. Had to wait for the service to come back up and then wait for the server to cycle through everything. So there really was no point and would not have gotten the service back any faster. I guess it does not really matter at this point since since I have made my decision and I am moving forward from this. I do thank you for addressing my response though..
Posted by WiredTree James, 01-17-2011, 04:14 PM
I just wanted to pop you an update and let you know that I am now seeing your dedi back online as well. If you need anything, please update your ticket and we will get right to it.
Posted by WiredTree James, 01-17-2011, 04:18 PM
You are welcome Kevin. If there is anything else I can do or answer for you, please feel free to contact me in a Sales/Support ticket at WiredTree. As always, we want to make sure everyone is well taken care of.
Posted by Bob30, 01-17-2011, 04:34 PM
Well done James i really appreciate your courage and matter of solving this problem, but you need to talk to the guys at CoreSite to stop spoiling business for you.
Posted by WiredTree James, 01-17-2011, 04:42 PM
Thanks Bob. Fully agreed. I wouldn't want to be Coresite today. If anyone else is having residual issues of any kind, just let me know. I am going to be sticking around in this thread until we have every last issue resolved. Ticket ID's help me track down your identity very quickly.
Posted by andrewtees, 01-17-2011, 04:46 PM
James, Thanks for looking into my one dedicated that was slow to return. All appears to be running correctly now. I want to also echo that it is important that this same issue not happen again. While I can appreciate the effort required to make power changes in a DC config, there needs to be a fix immediately. I am on the verge of bringing the rest of my servers over but I may need to wait and see.
Posted by WiredTree James, 01-17-2011, 04:48 PM
Thank you for the kind words. We have planned maintenance with CoreSite on the 31st that we will be announcing the week of the 24th. They are moving our circuits and upgrading / replacing a significant portion of our power equipment. Once we near the 24th next week I will have a more complete picture for everyone and will post the information both as an email for our clients and within our Grove.
Posted by Anthony, 01-17-2011, 05:08 PM
I'm surprised with the constant outages.. Hopefully they get your server back online soon. Good luck. Last edited by Anthony; 01-17-2011 at 05:22 PM.
Posted by UnderEstimated, 01-17-2011, 05:11 PM
My server went down again rite now, replied on the ticket 20 min ago on hold for 15 min on the phone now. What is going on?
Posted by Jedito, 01-17-2011, 05:14 PM
One of my servers is still offline, more than 4 hours of downtime....
Posted by UnderEstimated, 01-17-2011, 05:20 PM
I am still waiting for the phone to be picked up or answer on my tickets.
Posted by WiredTree James, 01-17-2011, 05:20 PM
It could be down for quotacheck on the Node. What is your Ticket ID? I will look into it for you.
Posted by WiredTree James, 01-17-2011, 05:21 PM
You are at 87% of an fsck on your dedicated server. It should be back soon. Just waiting to hear back.
Posted by UnderEstimated, 01-17-2011, 05:23 PM
ZBK-916579 IXC-850104
Posted by WiredTree James, 01-17-2011, 05:25 PM
Just a quick update Jedito, you should be online now or very soon. The check is finished and the server is booting your services. Thank you for your patience.
Posted by Jedito, 01-17-2011, 05:31 PM
Server 50 UP at Jan 17 2011 06:23 PM (GMT -3:00) Was down for: 04hr 24min Can you feel the love around?
Posted by WiredTree James, 01-17-2011, 05:40 PM
Glad to see you back online Jedito. If you could open a billing ticket, I will have them take a look at your ticket asap.
Posted by UnderEstimated, 01-17-2011, 05:44 PM
Any updates on mine?
Posted by WiredTree James, 01-17-2011, 05:54 PM
Under, I am seeing you as being back online and running. Your WHM is showing up for me and I think your system load is under control now after the reboot. If you need anything else, just update your ticket. Thank you! James
Posted by INS Digital Michael, 01-17-2011, 05:56 PM
James - can you check my ticket? Still down. Ticket XNB-721911 Thanks Michael
Posted by WiredTree James, 01-17-2011, 06:00 PM
Michael, Sure. Your Hybrid node is still in an fsck. I don't currently have an ETA for you but I can tell you that while this fsck is running, we have a simultaneous full node restore running in parallel as a precaution. James
Posted by INS Digital Michael, 01-17-2011, 06:17 PM
Thanks James. Got a question regarding power problems. How can this happen with power sub-stations, battery backups and diesel generators? What type of power equipment failed?
Posted by WiredTree James, 01-17-2011, 06:37 PM
I believe it is related to a main breaker failure or overload. I am waiting to hear more and don't want to say for sure until they provide us with more information. We expect the best from Coresite and will act accordingly after we receive their full report. James
Posted by VidiJoe, 01-17-2011, 07:19 PM
James - having some issues since the server went down. Please review ticket: CQI-453815 - My customers have not had access for hours now.
Posted by WiredTree James, 01-17-2011, 07:27 PM
Joe, Sure. I just replied to your ticket. If you could reply back with answers to my questions, we will look right into it. James
Posted by lostpeaks, 01-17-2011, 10:44 PM
WT, If I were you I would be on the horn with Coresite telling them to fix the problem sooner then later. As you can see you have lost customers over this and have lost a lot of new sales. I was about to sign up for 4-5 servers until I checked out reviews and found this and the others. I know it's outside of WT control however you should be putting some rod on the DC to solve it in a timely manner. What will happen to your client base if this happens again before the 31st???
Posted by WiredTree James, 01-17-2011, 10:52 PM
Lostpeaks, You read my mind. We have had Coresite working all afternoon/evening long prepping to move a row of servers onto additional circuits as a safety precaution. All clients affected have been emailed about emergency maintenance. While things may have been fine until January 31st, we will not willing to take the risk. James
Posted by lostpeaks, 01-17-2011, 11:14 PM
We I pray that nothing happens, Will consider you guys again after 6 months. . Always looking to add servers all over the world to serve my customer base.
Posted by WiredTree James, 01-17-2011, 11:16 PM
Thank you Lostpeaks. Have a great evening and feel free to contact me any time. James
Posted by dvgrhl, 01-17-2011, 11:41 PM
This is exactly the type of situation for why a Wiredtree forums or IRC channel would be helpful. I asked a year ago and was told this was in the works. Guess it's pretty low on the priority.
Posted by racersimage, 01-18-2011, 02:21 AM
Oops, I didn't know that there was an outages area. Sorry!! It came back up after an hour and a half and then went back down for another 30 minutes later this afternoon. They were quick to respond with the explanation and details though.
Posted by INS Digital Michael, 01-18-2011, 09:49 AM
James - server is up, I can connect, even sites are up, but (strangely) accounts aren't showing and email accounts are missing. I've got to have an ETA on when this will be resolved. We're getting calls and calls and I have no answers.
Posted by WiredTree James, 01-18-2011, 10:13 AM
Michael if you could bump your ticket in our Support queue, it is most likely just a matter of restoring your mail directory or taking a look at those specific files and directories. James
Posted by INS Digital Michael, 01-18-2011, 10:16 AM
James - I bumped it at 8:27am EST
Posted by WiredTree James, 01-18-2011, 10:32 AM
Thank you Michael. I see it now and have responded myself. If you could provide answers to my questions in the ticket we will continue forward as quickly as possible.
Posted by VidiJoe, 01-18-2011, 11:30 AM
James... Still having issues with our servers since they went down noon yesterday and I have my developer ready on standby to fix any other issues that were caused because of this. Ticket: GPX-416023 Hope we can get this resolved soon - thanks in advance
Posted by WiredTree James, 01-18-2011, 01:41 PM
Thankyou Joe. I have been watching this ticket and I see that one of our admins is active in assisting you. He is telling me that your issue has been resolved. If it is not, please update your ticket and I will have that particular admin ready to assist. James
Posted by pagecom, 01-18-2011, 02:09 PM
James, I have had nothing but good service with you guys up until now but it has been more than 24 hours and my server still down.The last I heard a restore was in progress and that was yesterday, how long a restore should take? This is not acceptable , I'm loosing sales by the minute , how much longer do I have to wait? Thanks Joe
Posted by WiredTree James, 01-18-2011, 02:37 PM
Joe if you could PM a ticket ID, I would be happy to look into your issue. There is a particular Hybrid node that we are running restores on for a few containers.
Posted by Mike V, 01-19-2011, 01:52 AM
Thread cleaned. Just a reminder the Outages Forum is for customers to discuss their current outage only. If you feel compelled to write a review, please use our Main Forums. Carry on
Posted by outlookemail, 01-23-2011, 12:17 PM
Did you move & where to?
Posted by Kevin K, 01-23-2011, 12:25 PM
I don't think this is the best place to discuss this since this thread has been cleaned several times.
Posted by outlookemail, 01-23-2011, 12:27 PM
Can you PM me please then? Or can you give me your email address. I am considering signing up with you, if you are with WT, will think > twice.
Posted by Kevin K, 01-23-2011, 12:28 PM
I have already answered your pm's.
Posted by BloggerDude, 01-26-2011, 08:31 PM
anyone else noticing very very slow loading times at wiredtree ??
Posted by BloggerDude, 01-26-2011, 09:00 PM
are you able to log-in to GROVE? are your sites loading fast and fine? I posted asking in the wiredtree outage threads, but nobody answers, I guess people check more this forum..
Posted by frki1, 01-26-2011, 09:06 PM
Yup,pages just loading... Again down?
Posted by HUGOBLUE, 01-26-2011, 09:06 PM
Their whole site seems to be down as well, maybe they are experincing problems with network/power ?
Posted by RamboLeer, 01-26-2011, 09:06 PM
nope, all down here
Posted by manukabay, 01-26-2011, 09:08 PM
Looks like they're down again. Can't get to my server or wiredtree.com.
Posted by RamboLeer, 01-26-2011, 09:08 PM
down again, what's going on with WT?!
Posted by VidiJoe, 01-26-2011, 09:08 PM
Down again here, wasn't it last week this happened?
Posted by manukabay, 01-26-2011, 09:10 PM
My servers back but still can't get to wiredtree.com
Posted by Jedito, 01-26-2011, 09:14 PM
wiredtree.com is down to me
Posted by VidiJoe, 01-26-2011, 09:14 PM
Looks like sites are back up for us... but not able to reach wiredtree
Posted by frki1, 01-26-2011, 09:16 PM
Sites back up butt WT down, cannot acces grove also
Posted by BloggerDude, 01-26-2011, 09:17 PM
this happened a lot of times lately.. really don't know what is going on and how they are not able to find a solution for this.. incredible..
Posted by BloggerDude, 01-26-2011, 09:18 PM
im not an expert, but its deinitely not an outage, sites go up and then down again for some times... then down and up again.. Am I wrong if i say that this looks like ANOTHER DDOS mass attack against them? Like the one that happened after the assange raid, at the end of last year.
Posted by Fastian, 01-26-2011, 09:23 PM
WT used to be rock solid ... Just got up after getting multiple sms from my monitoring. VPS came back shortly (I hope it will stay that way) From Grove May be this is also interrupting services for clients.
Posted by RamboLeer, 01-26-2011, 09:23 PM
ok, back up now...
Posted by diesel12, 01-26-2011, 09:32 PM
Can't get to wiredtree.com, but can get to Grove where they have a message about DOS attack..... which isn't much help to those that can't access it. WiredTree: Please communicate with your customers via Twitter when your site is not accessible or set up an alternative means of communicating when things aren't working and WiredTree.com is down. Last edited by diesel12; 01-26-2011 at 09:45 PM.
Posted by WebHosting Zac, 01-26-2011, 09:35 PM
It's a large, complex incoming DDOS targeting our main web site and Grove. Our engineers are working on filtering it now. It is only effecting clients connected to one rack switch, so a relatively small number of clients, and not power related. We apologize for the inconvenience.
Posted by Kevin K, 01-26-2011, 09:44 PM
Well my last server with them just went down within the last minute. Everything was fine until then. So it looks like this is spreading farther than that switch now.
Posted by frki1, 01-26-2011, 09:44 PM
down again...
Posted by sdscripts, 01-26-2011, 09:45 PM
My VPS down now too and can't access their site.
Posted by VidiJoe, 01-26-2011, 09:46 PM
Well my sites were back online... now they are down again across 3 dedicated servers and 2 VPS. Service has always been top notch with wiredtree and I really hope this is solved soon - downtime is becoming a regular occurrence it seems which is unfortunate.
Posted by Kevin K, 01-26-2011, 09:46 PM
Yup down again here within the last 2 minutes. Bye Bye Wiredtree......
Posted by RamboLeer, 01-26-2011, 09:47 PM
Yep, down again. Was down before but it came back up, then it went down again. Unfortunate
Posted by diesel12, 01-26-2011, 09:47 PM
No more Grove and server is now unavailable. No communication from WiredTree in this thread or Twitter over an 1 hour and 15 minutes since initial report of issues in this thread .... Good times.
Posted by teck, 01-26-2011, 09:48 PM
Down for me.
Posted by Energizer Bunny, 01-26-2011, 09:49 PM
Site was up all day apparently but everything just went down. Still problems with coresite ? Any news would be good news at this point.
Posted by Tyler, 01-26-2011, 09:50 PM
Down yet again...All servers, sigh.
Posted by MyForces, 01-26-2011, 09:50 PM
this is ridiculous..
Posted by sfoma, 01-26-2011, 09:52 PM
okay now I am frustrated. Down again . . Really!
Posted by Helstrm, 01-26-2011, 09:52 PM
Down here too... I have been with them for over 3 years so I am not going anywhere yet... Hopefully they can figure this out
Posted by frki1, 01-26-2011, 09:54 PM
You said that last time..just like me also
Posted by sross, 01-26-2011, 09:55 PM
Same here, but it is starting to look like a terrible fall from grace. 100% up time and tickets answered within 5 minutes has slowly become sporatic outages and tickets only answered when phoning and asking if it's being looked at. COuld this be related to some scheduled maintenance? I don't think it is so much wiredtree as it is their datacenter letting them down. Might be time for them to cut their losses and find another DC to host in.
Posted by jordanriane, 01-26-2011, 09:55 PM
*sadface* All of my servers down here (as well as wt/grove) I can honestly say that there has been A LOT of downtime the last year I've been with them but it doesn't bother me as much as it would if I were with say Burst (or any of the others I've been with.) I'm just hoping their network reliability and stability will get better. I used to have servers go for almost a YEAR of uptime (hardware AND network wise) but now I can get a few months, if that. I still love 'em regardless though. No, nothing scheduled for today. Tomorrow there's some maintenance though:
Posted by Kevin K, 01-26-2011, 09:56 PM
Yes and I have started moving servers out of there. I have one left and I guess I did not get that one out of there early enough. Anyways I just ordered a new server elsewhere to move this one out.
Posted by teck, 01-26-2011, 09:57 PM
Back up here.
Posted by sross, 01-26-2011, 09:59 PM
semi-back slooooooooooow.. i think their network is crapping itself
Posted by Tyler, 01-26-2011, 09:59 PM
Dedicateds are back as well, Grove and my VPS are still down.
Posted by frki1, 01-26-2011, 09:59 PM
Good luck with new provider..seams that i will need to do the same..
Posted by sfoma, 01-26-2011, 10:01 PM
"We are currently experiencing a DDOS against www.wiredtree.com and Grove which we are working on mitigating." According to Wiredtree via Twitter
Posted by CreedFeed, 01-26-2011, 10:02 PM
I've been defending WiredTree left and right throughout all of these issues, but this is now just crazy. Time to move on I think...
Posted by Fastian, 01-26-2011, 10:02 PM
Initially when it started, I thought it was a Planned maintenance. (Soon realized of my mistake) I managed to log into Grove and there was a message about WT under DDOS attack. So may be that's what going on. If it's another coresite power issue, I am moving out of WT for sure. Last edited by Fastian; 01-26-2011 at 10:06 PM.
Posted by ronaldv, 01-26-2011, 10:02 PM
I can't access their web, nor their admin panel nor my sites hosted with them!
Posted by JustJerk, 01-26-2011, 10:03 PM
It just went down it seems. While I was checking my email.
Posted by Kevin K, 01-26-2011, 10:04 PM
Check the outages forum: http://www.webhostingtalk.com/showthread.php?t=1014815 http://www.webhostingtalk.com/showthread.php?t=1017653
Posted by aiki, 01-26-2011, 10:04 PM
Yep they are down.
Posted by sross, 01-26-2011, 10:04 PM
My servers are responding as if a DDOS is in progress so doesn't seem like their previous issues. So are people with external DNS running fine?
Posted by Forward Web, 01-26-2011, 10:07 PM
Seems like they have been having a lot of trouble lately?
Posted by UnderEstimated, 01-26-2011, 10:08 PM
Are they down again because I cant even open wiredtree.com
Posted by Awesys, 01-26-2011, 10:08 PM
I cannot access my VPS via IP either. Definitely disappointed with all the recent downtime. Looking for alternatives now.
Posted by jordanriane, 01-26-2011, 10:09 PM
Looks like they're getting services back up. A bunch of my servers and VPS just popped back online .
Posted by sross, 01-26-2011, 10:11 PM
Well you can hardly blame them for a DDOS attack, I am not sure there is much that can be done other than identifying the attack and trying to mitigate it. Look at what ANonymous did to Visa not too long ago, it was down for ages from a DDOS.
Posted by Helstrm, 01-26-2011, 10:12 PM
well if this keeps up where is everyone going?
Posted by frki1, 01-26-2011, 10:12 PM
still down
Posted by Energizer Bunny, 01-26-2011, 10:13 PM
Probably down under
Posted by WebHosting Zac, 01-26-2011, 10:14 PM
We're preparing documents which will be released in 2 parts over the next two weeks which will describe what has already been done to ensure power is stable for the racks which had issues, and what is going to be done going forward to ensure continued stability. The second document is going to describe the network upgrades we're putting in place to help mitigate DDOS (among other things) in the future - we're investing hundreds of thousands of dollars in network upgrades and increasing our public Internet capacity to 40Gbps. Core network is being upgraded to 10GE with expandability all the way to 100GE(!) with the same gear. In any event, our main focus now is getting this DDOS resolved. Please check Grove for more information later.
Posted by FortuneGirl, 01-26-2011, 10:14 PM
Down again is not acceptable when im paying good $$... anyone else have suggestions? I really just cant seem to find a good host...
Posted by JohnDonovan, 01-26-2011, 10:14 PM
I tend to agree. They posted a tweet that they'd suffered a DDoS attack and were working to mitigate it. No sure what they could do to prevent ... other than be responsive when the attack occurs and work to resolve.
Posted by Helstrm, 01-26-2011, 10:18 PM
Bottom line is this could happen to any host. As long as they get me back up and take steps to prevent further incidents I am staying with them.
Posted by Jedito, 01-26-2011, 10:18 PM
That small number includes to ALL my servers.
Posted by Kevin K, 01-26-2011, 10:19 PM
Well when this has been completed and you have proven your services and stability I might consider coming back, but at this point...what's done is done. While all this might not be entirely your fault, we have to think about our customers and how this effects them. With all the downtime with your service last few months I feel our customers deserve better.
Posted by JonFatino, 01-26-2011, 10:22 PM
My 1024mb vps is still down http://96.251.47.70:81/STATUS_EARTH....ORG/index.html
Posted by jrandall, 01-26-2011, 10:23 PM
Quoted For Truth.
Posted by lokomon, 01-26-2011, 10:24 PM
I agree Iam with them almost 1 year and they are very good
Posted by frki1, 01-26-2011, 10:24 PM
Any ETA on resolving issues?
Posted by SEEKER2007, 01-26-2011, 10:24 PM
Mine also down !
Posted by diesel12, 01-26-2011, 10:26 PM
@Zach: WiredTree has also been pretty weak in billing and new orders / signups as well in the last few months, very very very slow turn times and sloppy at best, so supplementing your existing staff with more bodies wouldn't hurt either. We've been with you for 3 years and times have gotten pretty rough as of late on the front and back ends. I've already heard the "we put our resources into our technical support" argument and it's weak.
Posted by SEEKER2007, 01-26-2011, 10:27 PM
My site is coming-up and going down and why would anybody target WT for a complex DDOS ?? strange !
Posted by xrvel, 01-26-2011, 10:27 PM
ETA to up again for now? 1 of my dedi is down..
Posted by WebHosting Zac, 01-26-2011, 10:27 PM
I'm sorry you feel the truth is weak, but it is the truth. We're actually moving into new offices next week as we've run out of room in our current offices and we need more room to bring staff onto the WT team. Before anyone says "just hire more people, it's easy", I suggest they have a read of "The Mythical Man-Month": http://en.wikipedia.org/wiki/The_Mythical_Man-Month as it isn't as "easy" as it seems. Having said that, we are actively hiring and bringing more qualified staff on board as fast as as we can, and that is the truth. Last edited by WebHosting Zac; 01-26-2011 at 10:31 PM.
Posted by FortuneGirl, 01-26-2011, 10:28 PM
I understand that a DDOS can happen to anyone.. and i wouldnt be annoyed if i didnt already have 3 days of the sites being offline last week due to a power outage issue... and then 3 days to get the backup properly installed and restored? Not to mention like posted everywhere here.... several HOURS between updates.... im not paying $30 a month anymore... i expect this type of response and issues if i was... perhaps i am being unreasonable but if my sites are not reachable i dont give a whoot about network uptime... you've been offline for me for over 72 hours this month... Last edited by FortuneGirl; 01-26-2011 at 10:32 PM.
Posted by Fastian, 01-26-2011, 10:30 PM
As long as it's a DOS attack, I can understand I was hoping WT will take a step forward in right direction. Good to see something is being worked on. Especially DOS mitigation, this will surely be a great addition.
Posted by TC-Chris, 01-26-2011, 10:33 PM
Best of luck to Zac and the WT crew. I know that these issues are absolutely not expected, nor desired, and know that you folks are working on getting this matter resolved successfully as usual. Cheers for your hard work and efforts.
Posted by Jedito, 01-26-2011, 10:34 PM
Zac, can you please PM me, or tell me any way of contact to Joe D. I need to resolve a situation that you probably are aware already IMMEDIATELY.
Posted by WebHosting Zac, 01-26-2011, 10:35 PM
Your guess is as good as ours. The last time we had a major DDOS it was a client who was targeted and had nothing to do with Assange (I'm not sure why people are spreading that rumor). It was actually a provider on WHT who was getting DDOSd and they moved to a server to us, while it was being DDOSd, and didn't tell us they were getting DDOSd when they moved, and in fact tried to hide that from us. They were terminated. This time it is against us, and I can only guess it is some sad person with an axe to grind.
Posted by pilsentico, 01-26-2011, 10:37 PM
wow this really sucks....
Posted by Energizer Bunny, 01-26-2011, 10:38 PM
Will do but how long is the "later" going to be ? I am sure you guys are there battling it out, but do they have any workarounds that could be put in rather quickly or maybe rough estimated time with regards to coming online ? I had 2 new customers signup only last night and I do not know how to explain to them why we are down .
Posted by strikernr, 01-26-2011, 10:39 PM
Any this time I only to care to have my sites up. I'm not too pleased with moving my dns setting back to my old host. All the files are old there and back up is not up to date either. When will you have the servers up?
Posted by mrcbrown, 01-26-2011, 10:39 PM
I know you probably can't give a 100% accurate estimate - but what kind of *ROUGH* timeframe we talking here for getting things back to the land of the living? Got things seem to come up for a bit then go down? I can't even reach grove to even ponder opening a ticket - and I have customers getting anxious.
Posted by pagecom, 01-26-2011, 10:39 PM
Same here down again.
Posted by LiquidR1, 01-26-2011, 10:40 PM
Wiredtree down again, website down. This is not good.
Posted by jrandall, 01-26-2011, 10:41 PM
The power outages, as bad as they were, were isolated and by my logic, very bad luck for so much to fail. I only experienced a few hours of downtime, and my server was fine. Restoring backups can take a long time, especially when they're not yours but, stored and encrypted at the DC. You are never going to find a perfect host. Nothing is 100% secure, and nothing is 100% reliable. Ever work with computers?
Posted by UnderEstimated, 01-26-2011, 10:42 PM
ETA! something?
Posted by Daniel B, 01-26-2011, 10:42 PM
mine are back up, wiredtree.com still seems to be down though.
Posted by JonFatino, 01-26-2011, 10:43 PM
Monitoring is nice http://96.251.47.70:81/STATUS_EARTH....ORG/index.html earth.wazzhost.org is backup! Woot!
Posted by WebHosting Zac, 01-26-2011, 10:44 PM
Our website will likely be down for a bit longer. Client services are all in the process of being restored if they aren't restored though. BGP had to reroute a couple times which takes a few minutes each time if your routes are affected.
Posted by diesel12, 01-26-2011, 10:44 PM
@Zach: Providing solid customer service on the front end (hosting) and the backend (billing / bookkeeping) is not an either or proposition and never should be, imho. The premise that somehow resources are in such short supply that both cannot be adequately handled would suggest an issue with the model or management of existing resources, which is why I don't accept the premise that it is an either or situation and consider the entire premise and argument to be WEAK. In any case, glad you're getting into an office that can accommodate your growth. Last edited by diesel12; 01-26-2011 at 10:49 PM.
Posted by WebHosting Zac, 01-26-2011, 10:46 PM
All client services are in the process of being restored if they are not already restored. A network reconfiguration required a couple of BGP reroutes which can take a few minutes each if you are coming in or leaving our network over a provider that was changed. Our web site will likely have spotty availability while we work on mitigating the attack.
Posted by WebHosting Zac, 01-26-2011, 10:49 PM
Do you want to pay more so your billing tickets get answered faster?
Posted by UnderEstimated, 01-26-2011, 10:50 PM
I moved to wiredtree not even a year ago. This is too much of a problem for me after paying this much and getting so many problems is not good. No matter whose fault it is. I need an ETA atleast. Because I have too much competition and that is not good for my site's reputation either.
Posted by xrvel, 01-26-2011, 10:52 PM
My server is up again.. Thanks.
Posted by jrandall, 01-26-2011, 10:52 PM
ETA and DoS Attack are not in the same vocabulary.
Posted by azn_romeo_4u, 01-26-2011, 10:53 PM
According to my 3rd party stats, my server has been down for 2 hours off and on. Still not up. Just came back home too =/
Posted by UnderEstimated, 01-26-2011, 10:53 PM
Did I ask you? Or you some kind of a spokesman?
Posted by Energizer Bunny, 01-26-2011, 10:53 PM
Haa good one imagine a ticketing system having a bidding system on them and whoever bids the highest gets their ticket answered first . Anyways my server is back up lets see if it stays that way.
Posted by frki1, 01-26-2011, 10:58 PM
Up again
Posted by DGRing, 01-26-2011, 10:59 PM
Down for me as well. Zac is obviously back from out of town, so he needs to start addressing the frequent outages, and soon. I've been with them over two years and the poor service of the last month is about to erase all those years of goodwill. I just migrated a major client over to my VPS and I'm start to get very, very concerned.
Posted by annatech, 01-26-2011, 11:01 PM
Good evening, everyone. I also have a dedicated server with Wiredtree which was affected by this DDoS issue. But before we all jump on Wiredtree for this "failure", let me tell you a story... Up until recently I developed/webmastered the web infrastructure for a multi-million dollar company which hosted with Rackspace. Ah, Rackspace, the be-all, end-all of premium, boutique dedicated server hosting, complete with smooth Jazz hold-music. Well, about a year ago they had a few issues with the cooling and UPS systems at their DFW (Dallas Fort Worth) hosting center. One incident brought the *entire* hosting center down for up to 6 hours for some customers. That's from a company commonly considered the "best" hosting company in the world, charging, on average, $500 - $1500 (or more) per server, plus firewall, IDS, etc. It took them a little while to work out the kinks, there was some more downtime here and there (we eventually setup Alert-site to monitor externally) and the problems were resolved. A DDoS attack is a serious issue which requires lots of peripheral infrastructure to circumvent it. Do I want to complain? Sure... but given what I pay for my dedicated server hardware and the support I've gotten far, my faith is still with Wiredtree. My recommendation: If you want redundancy on the cheap, back-up your websites on Wiredtree to an external VPS host with rsync and a few MySQL triggers (you can even automate database mirroring via phpmyadmin now!). This way, you can "flip-the-switch" at your domain manager (like Godaddy). Or... something like that... My point is, Wiredtree's service record is no-worse than some of the the best dedicated server hosts out there. BTW... I'm in Annapolis, MD and we are getting some *SNOW* right now - just witnessed my first "thunder snow" a couple hours ago.
Posted by WebHosting Zac, 01-26-2011, 11:02 PM
Sorry, I didn't mean to come off as snarky. My apologies if I did. In any event, if your server is having issues, please open a ticket and we will get to it as fast as we can.
Posted by SEEKER2007, 01-26-2011, 11:03 PM
Zac, I can understand the current situation and this can happen to any web-host but ticket response time is what WT needs to work upon in future as it has been getting slower by the day.
Posted by LiquidR1, 01-26-2011, 11:03 PM
Man you're lucky. I don't get how some servers can be up when wiredtree's own site is down? very strange.
Posted by JD45, 01-26-2011, 11:03 PM
That is THE most ridiculous statement I've ever heard. With that type of philosophy or even thinking.....you can kiss my account and patronage goodbye.
Posted by azn_romeo_4u, 01-26-2011, 11:05 PM
server is still down this sucks
Posted by diesel12, 01-26-2011, 11:05 PM
"Do you want to pay more so your billing tickets get answered faster?" @Zac: Do you mind that because your billing department is so slow and understaffed they sometimes get sloppy and fail to bill for items and give unnecessary credits in the hundreds of dollars? I know several people with this experience while using WiredTree, ourselves included. Just trying to give you a little different perspective here. You are simultaneously stating that you're moving into new offices that will enable you to hire more people and defending providing poor customer service which is a result of understaffing. The "will you pay more argument" is a copout because proper customer service under your existing pricing is assumed. If you're not charging enough, that's your fault dude. We're not talking about expecting one minute billing response times, we're talking about taking simple billing related issues and taking them down from WEEKS to resolve to days or under a week. When ticket responses come once a day or every few days or tickets get closed out altogether with no answer, there's an issue and it isn't dependent on me paying more for better service, it's dependent on how you choose to run your business. Last edited by diesel12; 01-26-2011 at 11:10 PM.
Posted by RamboLeer, 01-26-2011, 11:08 PM
How is it that my server is inaccessible but one of my "tell-a-friend" scripts is being abused to hell during this downtime?? WT plz bring it back, good luck!
Posted by HUGOBLUE, 01-26-2011, 11:10 PM
Seems like you managed to ride off the DDOS attack, Wiredtree.com is loading for me now
Posted by Kevin K, 01-26-2011, 11:12 PM
Well said!!! That comment was extremely uncalled for under any circumstance. Nice way to calm and reassure customers in a situation like this.
Posted by LiquidR1, 01-26-2011, 11:13 PM
We're down, and can't access wiredtree's site.
Posted by sross, 01-26-2011, 11:15 PM
currently I can get to my site via a USA proxy server but not my country.
Posted by WebHosting Zac, 01-26-2011, 11:15 PM
We've already been addressing them, that is the hard part to relay to clients because there is no way for them to see that. We've made large investments (~$250k) in upgrading to brand new state of the art network gear for our core routing and switching. The new gear is already at our datacenter and actually already partially installed. We're expecting to have the new gear installed and the changeover completed by the end of February. We were planning to make an announcement when it was completed, but I suppose now is as good as any. CoreSite has flown out their SVP of Datacenters to go over everything top to bottom to ensure our power setup is stable and there are no issues going forward. They are taking this just as seriously as you and I do, and we will have more information for you next week in Grove. They agree the issues are completely unacceptable. Our current offices only support around 15 people on shift at any time, and we've recently filled that up. For the last two months we have been busy building out 5500 square feet of new office space which will give us room for about 45 staff to be on-site at any time. We are immediately increasing our support and sales staff as well. I think one area where we could do better is relaying this information to you all, and this is something we will be addressing.
Posted by SEEKER2007, 01-26-2011, 11:15 PM
HACKED !!!!!!!! I prey not
Posted by Zealus.com, 01-26-2011, 11:17 PM
Same here, down, can't access support page. Guess that's it for my patience
Posted by sross, 01-26-2011, 11:18 PM
You have to admit there is a bit of "drama queen" action going on in here. Here is a vid for those upset by a forum comment: http://www.youtube.com/watch?v=unkIVvjZc9Y
Posted by WebHosting Zac, 01-26-2011, 11:18 PM
I'm sorry, but I do not agree. Obviously we want our Billing to have fast responses, and in order to do that, we have to provide Billing at level X with cost Y. If you want to increase level X, cost Y will increase. It is basic business. Also, you need to consider that I am speaking at a theoretical level where we have already done all that we can to increase our level of service at this time without doing anything that would increase cost. Resources aren't free. Last edited by WebHosting Zac; 01-26-2011 at 11:21 PM.
Posted by WebHosting Zac, 01-26-2011, 11:21 PM
Please send any tickets which have taken weeks to resolve, I would like to look into it. I admit that we have had issues with billing, as have many other companies. We have also done our best to resolve them in a flexible and efficient manner. You presume to know what caused these issues, but you are incorrect. Our average billing turnaround time is 1 business day. Most of our clients find this to be quite acceptable. I'm sorry if you would prefer faster billing response times. Last edited by WebHosting Zac; 01-26-2011 at 11:25 PM.
Posted by mrcbrown, 01-26-2011, 11:23 PM
I'd honestly stick to the status and situation at hand, save this type of back and forth maybe for some type of town-hall chat after things have calmed down, last thing anyone needs is replies posted during high stress and not posted in maybe the clearest frame of mind. Just my two bits.
Posted by Kevin K, 01-26-2011, 11:24 PM
Ya well I got one for you too. Use your imagination.... Last edited by Kevin K; 01-26-2011 at 11:29 PM.
Posted by WebHosting Zac, 01-26-2011, 11:26 PM
Do you have tickets open? Have you tried http://www.hyperspin.com/en/quicktest.php to ensure you aren't blocked by the firewall for some reason?
Posted by WebHosting Zac, 01-26-2011, 11:27 PM
You are completely right.
Posted by Zealus.com, 01-26-2011, 11:38 PM
Your site loads only partially - I can login to the Grove, but Support just times out.
Posted by WebHosting Zac, 01-26-2011, 11:40 PM
Please open a ticket if you haven't already and include a traceroute to your server, that will help us the most.
Posted by diesel12, 01-26-2011, 11:41 PM
We're talking about customer service level degradation here. It used to be good, now not so much. If you are saying you've changed your mind and now we need to pay more to get old level of service, maybe you should raise your prices? 1 business day would be freaking awesome but that's not even close to reality in our experience. I could post our last ticket that got closed out twice with non-responses and the apologies that came with it each time we re-opened the ticket (issue took over a week to resolve), but 1) Grove is down 2) you seem pretty convinced that our feedback is either irrelevant or our experience is the exception to the rule (maybe it is?), so we won't keep beating a dead horse and we'll leave it at that. Last edited by diesel12; 01-26-2011 at 11:48 PM.
Posted by Zealus.com, 01-26-2011, 11:42 PM
Let me repeat that - I wasn't able to get any further than clicking on Support tab in Grove. I am unable to open the ticket (unless I'm forgetting that there is an e-mail option).
Posted by BirdBrained, 01-26-2011, 11:42 PM
I can't get wiredtree or grove to come up along with my dedicated server. Hyperspin shows wiredtree up, and shows my hostname in all but Ashburn VA, which reports unknown host.
Posted by WebHosting Zac, 01-26-2011, 11:44 PM
Yes, sorry, please try emailing support@wiredtree.com - that should go through and you should get an autoresponse. Let me know if that isn't the case.
Posted by BirdBrained, 01-26-2011, 11:44 PM
tracert quits after: 9 35 ms 33 ms 34 ms ae-14-53.car4.Chicago1.Level3.net [4.68.101.72]
Posted by LiquidR1, 01-26-2011, 11:47 PM
Yeah for some reason my IP was blocked, but I managed to login from another IP. The site is loading extremely slow but I was able to open a ticket.
Posted by WebHosting Zac, 01-26-2011, 11:47 PM
Grove is getting DDOS'd as is ns1.wiredtree.com - we are working on that now so it will be slow for a bit, sorry.
Posted by Zealus.com, 01-26-2011, 11:55 PM
E-mail sent, got an auto response (Ticket #FVT-526326)
Posted by Sam.Serverfreak, 01-27-2011, 12:26 AM
submitted a ticket with no response.
Posted by azn_romeo_4u, 01-27-2011, 01:00 AM
dang this is gay, just finished "The A-Team" and my server is still down.
Posted by Sam.Serverfreak, 01-27-2011, 01:02 AM
still down from certain location. traceroute sent to them with no response again....is 1pm here and is peak hour. look like we have to off for holiday today.
Posted by azn_romeo_4u, 01-27-2011, 01:03 AM
Server is still down. How does a Dos attacking take down every single server in your network? Down still atleast 5-6pm PST. Now 9PM pst. This is getting beyond ridiculous.
Posted by defen, 01-27-2011, 01:06 AM
I submitted a ticket. I cannot access my site and a traceroute via several locations shows most make it but one or two do not.
Posted by JonFatino, 01-27-2011, 01:12 AM
I dont believe Wiredtree has any ddos mitigation. *example cisco guard* (Please correct me if i am wrong wiredtree) Matter fact you would be surprised as a lot of hosts dont have ddos mitigation. DDoS mitigation is very expensive. Not only is the hardware very expensive, but it costs a lot yearly to have the devices updated and managed because DDoS is always changing and becoming smarter. Most data centers just contact carrier(s) and they null route the ips that are sending the attacks. I believe total downtime was near 1 hour. OFFLINE - Jan 26 2011 - 05:41 PM ONLINE - Jan 26 2011 - 06:39 PM I am on the vps 1024 mb plan. http://just-ping.com/index.php?vh=earth.wazzhost.org Last edited by JonFatino; 01-27-2011 at 01:18 AM.
Posted by Jedito, 01-27-2011, 01:14 AM
It seems like its down from some locations, some customers report their sites as down while they are up to us.
Posted by Sam.Serverfreak, 01-27-2011, 01:14 AM
we got 4 hours of downtime and counting....
Posted by Bob30, 01-27-2011, 01:17 AM
Maybe i'm just lucky or it's because i'm in a dedicated server, my two servers we're up for the past 24 hrs.
Posted by Sam.Serverfreak, 01-27-2011, 01:18 AM
you probably need to double check. some location are up and are down.
Posted by Bob30, 01-27-2011, 01:47 AM
but i can't access grove.
Posted by WebHosting Zac, 01-27-2011, 01:52 AM
Grove and the main www.wiredtree.com site are still under major DDOS. Everything else should be up. We are working on restoring service to those. In the meantime, please email us: support@wiredtree.com or +1.312.447.0510 toll-free in the US: +1.866.523.8733 Last edited by Harzem; 01-27-2011 at 01:23 PM. Reason: fixed the email
Posted by diesel12, 01-27-2011, 01:53 AM
@Zac: thanks for the update, that makes a little more sense. Was hoping that Grove being down again wasn't a sign of a second wave that might be coming ....
Posted by target, 01-27-2011, 03:40 AM
Why is there no information about it on the Grove ?
Posted by ddosguru, 01-27-2011, 06:32 AM
Complex attacks are often more complicated than just buying hardware, especially Cisco Guard. We've been moving to exclusively Juniper on everything above our access layer due to performance considerations.
Posted by vB-WP-Admin, 01-27-2011, 11:23 AM
I have made this suggestion earlier and would like to do so again, please have a support forum, which is hosted elsewhere to ensure in case of network or DC issues, customers can get access to information faster and at a place where they are sure, they will get answers, if they go to it!
Posted by diesel12, 01-27-2011, 01:12 PM
When woothemes.com (not even a host, mind you) was up and down over a 3 or 4 days after a continued DDOS attack, they set up a simple P2 WordPress site to give updates on a different unaffected server: http://demo.woothemes.com/dos/. It isn't working properly now and is filled with spam, but you get the idea.
Posted by UnderEstimated, 01-27-2011, 01:40 PM
I cant contact them through the phone it comes up as busy
Posted by DGRing, 01-27-2011, 01:59 PM
My concern is downtime, not ticket response or billing issues. Outages on 12/11, 1/17 and last night are making me quite nervous. And the fact there is a major h/w upgrade on the horizon, quite frankly scares me now. The outage on 1/17, since it was during business hours here on the East coast, had a major impact on my business as I had just migrated a big client over to hosting with me. This is reassuring, as that seems to be the cause of two of the outages. I think your support staff are _outstanding_ and I've never rated any ticket less than a 10. Again, my major concern is downtime. If there's a h/w failure, it doesn't matter how great your support staff is if there aren't redundant systems in place to get my system back online immediately. I'm a mid 50's old fart and don't do Twitter or Facebook but that would seem to be the best place to start. I believe James jumped in during the 1/17 outage and posted here and provided a link to your accounts. When my sites are down, I need something truthful to tell my clients. One other request from an old mainframe MVS systems programmer, don't schedule maintenance during normal M-F, 7a-7p business hours. I was petrified at the 1/25, 1800-2200 CST Power Maintenance schedule (given the 1/17 outage) as I have many customers on the West coast who are still open for business at that time. You're much less likely to get irate customers if the outage is on a Saturday night at 2:00AM than if it's on a Monday morning at 10:00AM. How you and your company respond to these recent challenges will determine just how effective a manager you are... and whether I'll stay with you or not. I'm _so_ much in your corner and I wish you the very best of luck during this transition period.
Posted by Bob30, 01-27-2011, 02:09 PM
Also think about WiredTree clients not in america.
Posted by Samuraid, 01-27-2011, 05:56 PM
We just (within the past 15 minutes) started seeing packet loss again on a few VPS servers at wiredtree. Also, grove.wiredtree.com isn't loading. Is this perhaps another attack?
Posted by pilsentico, 01-27-2011, 06:06 PM
groove is not loading for me neither
Posted by diesel12, 01-27-2011, 06:11 PM
No Grove here either. Still have a ticket open from this morning that still has not had a single response .... Joe posted the following tweet but why isn't Wired Tree communicating about this???? From twitter: joeonhosting‎ And the DDoS attack continues on grove.wiredtree.com. We are working on it folks. http://twitter.com/#!/joeonhosting/s...26908991774721
Posted by Samuraid, 01-27-2011, 06:15 PM
less than a minute ago via web
Posted by WebHosting Zac, 01-27-2011, 06:16 PM
I'm looking at our Support queue and see no unanswered tickets. What's the ticket ID so I can look into it? We post Twitter updates as soon as we can. We're not going to be able to post up to the second updates all around the clock. I don't think a few minute lag time is unacceptable - you need to consider what people are doing. That status hasn't changed either. We are still under DDOS, just like we were last night, except it has been nearly all filtered (and just directed at WT sites) so you just haven't noticed. Did you notice how we brought client sites back online first as well?
Posted by diesel12, 01-27-2011, 06:25 PM
@Zac: Not saying anything was unacceptable except that I haven't received a response to a ticket from 7 hours ago and the first mention or update I've seen that the DDOS attack has not yet stopped since last night was from a WT employee.... we have no way of knowing that minutes later WT may or may not communicate an update, so I don't think it's unreasonable to wonder why WT hadn't communicated up to that point. "Did you notice how we brought client sites back online first as well?" Good job man. Last edited by diesel12; 01-27-2011 at 06:28 PM.
Posted by WebHosting Zac, 01-27-2011, 06:27 PM
What's the ticket ID?
Posted by pagecom, 01-27-2011, 06:27 PM
My sites just went down, this is way too much I feel for WiredTree but can't make money if my sites are off line.
Posted by diesel12, 01-27-2011, 06:30 PM
Zac: I can't get ticket ID until Grove is back up, don't save those emails.
Posted by frki1, 01-27-2011, 06:34 PM
Problems again..this is now realy too much
Posted by WebHosting Zac, 01-27-2011, 06:38 PM
We are seeing some packet loss (1-2%) and increased latency on one segment of the network which can result in slower than normal page loads. Our engineers are working to resolve this now. If you are experiencing issues greater than this, please email us a traceroute and ping summary to support@wiredtree.com or PM it to me.
Posted by whitevitz, 01-27-2011, 07:18 PM
My sites are down as well. Sent you a pm as I cant get on groves. Thx
Posted by WebHosting Zac, 01-27-2011, 07:32 PM
Packet loss should be resolved now. If you are having issues and cannot reach Grove, please email support@wiredtree.com
Posted by whitevitz, 01-27-2011, 07:33 PM
I still cannot reach grove.
Posted by Qubicle, 01-27-2011, 11:04 PM
I can't reach grove, as well as I have sent support tickets to support@wiredtree.com and am not even receiving e-mail confirmation.
Posted by eva2000, 01-28-2011, 02:47 AM
exactly same situation for me
Posted by Sam.Serverfreak, 01-28-2011, 05:21 AM
anyone else having problem to connect to wiredtree now?
Posted by lokomon, 01-28-2011, 05:23 AM
Yes, offline 30 min now
Posted by Sam.Serverfreak, 01-28-2011, 05:24 AM
certain location not able to connect to them for more than 30mins and counting. grove totally down.
Posted by vB-WP-Admin, 01-28-2011, 05:38 AM
While I can access my websites and most of the other users as well (as far as I can tell), some aren't able to. Looks like DDoS attack is continuing.
Posted by flashlight, 01-28-2011, 05:44 AM
Yep, I'm down as well for about an hour now. Can't even see wiredtree, and also emails to support aren't even making it into the ticket system. Was able to get in on the phone and they said that traffic is being rerouted to new equipment.. so hopefully that will be fixed soon as we're stuck as far as getting work done. May be middle of the night there, but 9:44am here.
Posted by lokomon, 01-28-2011, 06:03 AM
I think somebody hate them very much, 2 days now wiredtree is under attack ( ! ) or what ?
Posted by Sam.Serverfreak, 01-28-2011, 06:04 AM
is coming back slowly now.
Posted by Sam.Serverfreak, 01-28-2011, 06:23 AM
down again.....
Posted by TimothyH, 01-28-2011, 06:31 AM
I am currently having trouble connecting to Grove. However our servers are responding, very slowly. Edit: Server response times have returned to normal. However still unable to access Grove. Last edited by TimothyH; 01-28-2011 at 06:45 AM. Reason: Update
Posted by Sam.Serverfreak, 01-28-2011, 06:33 AM
up and down and is up again now. apparently they have problem with savvis.
Posted by humawebdesign, 01-28-2011, 07:01 AM
Its happening due to sever is down. You should check it and if it is happening on regular basis then you may have to take decision to change it.
Posted by Harzem, 01-28-2011, 07:28 AM
My VPS has been up for the last few days, no problems for me.
Posted by target, 01-28-2011, 09:31 AM
My VPS loads ok, but i can not get into the grove ... another DDOSS attack ??
Posted by Fastian, 01-28-2011, 10:53 AM
Grove is not accessible for about 11-12 hours now. My VPS was also having trouble earlier (About the same time when everyone else was having issue) Zac, what's up with Grove??
Posted by srider, 01-28-2011, 11:21 AM
I was going to login to Grove to tell them I'm done with my dedi there. After all of the power problems in December, plus two incidents of them powering up a second server with my main IP address, I started moving out on Christmas day. All of my clients are on my new server now, and it is not at Wiredtree! With hindsight I think I made the right decision.
Posted by Kevin K, 01-28-2011, 11:34 AM
I don't blame you one bit. We have been moving out slowly but surely. Our last move will be this Saturday if their systems stay available. I can say I had a good run with them for a long time, but over the last few months the outages and downtime has caused too many issues for us and our customers. If I was just running my own sites I think I could bare with it, but when it comes to effecting customers, well then we have to do what is best for them.
Posted by thezombiehunter, 01-28-2011, 11:47 AM
Gotta love the lack of communication going on here. Where is Zac?
Posted by diesel12, 01-28-2011, 12:04 PM
Was just thinking about that too zombiehunter. His last response was 16 or so hours ago on this board and this is day 3 of a DDOS attack with no Grove access for a day and busy phones that don't pick up (when I last checked yesterday). They did a great job yesterday of keeping sites up, but that doesn't mean I'm not on here every hour or two or scanning twitter constantly for updates or reports of more outages since these can be warnings that we too may be affected shortly. The fact that there is an ongoing issue necessitates communication. Not trying to give anyone a hard time just for the sake of it, but I cannot imagine communicating this little with my clients and I'm sure our clients are fractional in number compared to WiredTree. An ongoing attack means we are potentially also at risk and necessitates regular communication. What is the current status? Is progress being made? Have attacks intensified or are they waning? Regular communication every 3 to 6 hours just to let us know that the wheels haven't fallen off would go extremely far in setting people at ease and letting us know that the situation is under control. There has been no standard set for when and how updates will be given, something of critical importance given that the support portal, Grove, has up and down for days with about 24 hours of straight downtime. A simple blog with updates or regular updates here or via twitter would be so simple and creates a win / win. I mean if you can't communicate with your users in a time of crisis consistently 3 or 4 times a day .... there's really no excuse since writing a sentence or two takes a couple of minutes, worst case scenario, it requires so little effort! It's 11 AM in Chicago right now and it didn't occur to anyone that an update to start the day might be a good idea since last update was last night? I just don't get it. Last edited by diesel12; 01-28-2011 at 12:14 PM.
Posted by SwiftModders, 01-28-2011, 01:01 PM
I switched VPS servers last night. I have been with Wired Tree for so many years now it's a real shame. I loved their support and had wonderful uptime with them. Recently though this has gotten horrendous. Even though I am paying a bit more now, I'm much happier I moved. Sorry Zac, but enough is enough.
Posted by thezombiehunter, 01-28-2011, 01:16 PM
Still down and no updates at all. Can someone shoot Zac a PM or something?
Posted by WebHosting Zac, 01-28-2011, 01:41 PM
If you are having connectivity issues, call or open a ticket. A DDOS is still ongoing, but only against our sites and it hasn't had customer impact besides Grove being unreachable for some. No customer service is impacted. Ie. Same status as when we last posted.
Posted by WebHosting Zac, 01-28-2011, 05:03 PM
Grove has been reachable for a couple hours now.
Posted by lokomon, 01-28-2011, 07:04 PM
I think after all these problems wiredtree is faster than ever I will stay with these guys because they have good support and 100% uptime until now I hope after this problems they will be more pro in the future
Posted by jo2jo, 01-28-2011, 08:10 PM
ive been up and down all day..including right now. so how is it that this does NOT affect customers?
Posted by BloggerDude, 01-28-2011, 11:04 PM
Are you noticing slow loading times, high latency and timed outs on your wired tree sites?
Posted by VidiJoe, 01-28-2011, 11:18 PM
Here we go again... can't get my websites/servers. Hope its temporary
Posted by Kevin K, 01-28-2011, 11:20 PM
Here we go AGAIN. Server is down.
Posted by diesel12, 01-28-2011, 11:23 PM
Yep, sites down, alerts coming in from siteuptime and zero communication all day about the present status and outlook going forward as we approach day 4 of DDOS with WiredTree.
Posted by VidiJoe, 01-28-2011, 11:24 PM
Yep same here...again
Posted by techguy85, 01-28-2011, 11:24 PM
Plus one on that one. Perhaps it is time for Wiredtree to bring in some heavy weapons against these DDOS attacks. There have got to be some specialists in Chicago that can help them out. Other than these issues I've had great service from them.
Posted by Energizer Bunny, 01-28-2011, 11:24 PM
thats why the reporting service says site down !! And i thought i lost internet connection . Grove is working i wondering if they are moving servers to a safe room ?
Posted by Kevin K, 01-28-2011, 11:25 PM
26 hours more of this and then I am free of these Wiredtree issues. That moment cannot come fast enough right now.
Posted by techguy85, 01-28-2011, 11:28 PM
Wow, this must be a crazy run of DDOS attacks. Hopefully they bring in some specialists soon. I really like WT so far, seems like these DDOSs aren't their thing though.
Posted by Kevin K, 01-28-2011, 11:31 PM
From WiredTree's Twitter:
Posted by Fastian, 01-28-2011, 11:31 PM
Down again as well. I am really at the end of my patience level here. And the worst part, WT support is saying that yesterday's outage was "planned maintenance" even though it was clearly out of "Maintenance window". Instead of saying like "We are sorry for the trouble" (Or something along the lines), They are treating like "It was coming, you should be prepared" Now that is something I wasn't expecting from WiredTree.
Posted by TimothyH, 01-28-2011, 11:32 PM
Our server is also unreachable at this time. After almost 3 years of service from WiredTree, we have made the decision to move to another provider.
Posted by MyForces, 01-28-2011, 11:33 PM
Down again? .....
Posted by HostingDeals, 01-28-2011, 11:34 PM
What the heck happened at WiredTree. All these rave reviews about their service, I switch a few months ago and wham, always offline. I pay good money to them for this? Did they not ever plan on a DDoS issue or what? Who did they piss off to cause all of this? I'm SICK of this. Second month I've asked for credits and I've yet to see them give any sort of refund even tho Im advertising and paying for traffic to go to a black hole.
Posted by Awesys, 01-28-2011, 11:34 PM
Yep, my VPS is down again. Watching all the Twitter rants too...
Posted by strikernr, 01-28-2011, 11:35 PM
down again. just got a notice.
Posted by VidiJoe, 01-28-2011, 11:37 PM
Sites up...hopefully they stay up
Posted by JustJerk, 01-28-2011, 11:37 PM
I'm back up.......................
Posted by strikernr, 01-28-2011, 11:38 PM
you must be a CPA affiliate. You must make a lot already. What are you complaining about? Get your own server.
Posted by techguy85, 01-28-2011, 11:38 PM
My VPS is slowly beginning to respond again.
Posted by TimothyH, 01-28-2011, 11:39 PM
Connectivity appears to have been restored, for the time being.
Posted by Fastian, 01-28-2011, 11:41 PM
Are you sure you will even get anything? The longest outage (Within last few days) I suffered was during yesterday's issue. And according to them it was "Planned Maintenance" (Which it clearly wasn't)
Posted by Kevin K, 01-28-2011, 11:41 PM
Back online for now......
Posted by gauson, 01-28-2011, 11:45 PM
Been with WT for 1+ years now but recently it has been taking along time to connect to the sites. If it connects at all.
Posted by Fastian, 01-28-2011, 11:45 PM
Back after 20 mins (as per Pingdom) This month is gonna be the worst month I ever had with WT.
Posted by USEYOURHEED, 01-28-2011, 11:46 PM
This is way too much down time. I am going to lose my clients. And twice I've had a lead on new clients - but when they tried to visit my web dev sites - they were down. WiredTree is killing me. I'm going to post a link to this forum in twitter so it will show in real time results. It's awful that WiredTree isn't telling us WHY they are getting targeted. I've never heard of such a thing for a mid-sized web host. Wiredtree - not only are you going to have some jerks attacking you ... your good clients are going to turn on you. SO FRUSTRATING.
Posted by techguy85, 01-28-2011, 11:49 PM
I haven't seen anyone at wiredtree claim that last night's event was planned maintenance. When did they tell you this?
Posted by BloggerDude, 01-28-2011, 11:49 PM
I have read something related to the "assange" issue. Not sure about the details, if they were hosting someone related to assange and took them down or what.. If they don't give details, it's normal that we start speculating and creating hypothesis. But anyway, besides knowing the truth, WE want our sites to work, and that wiredtree stops this mass DDOS attack as soon as possible, if ever.
Posted by Awesys, 01-28-2011, 11:51 PM
I don't know if they claimed anything, but this notice was sent out:
Posted by NeoComp, 01-28-2011, 11:53 PM
High delays in a VPS according to our monitoring system ... from typical 2.4 [ms] to 105+ [ms] Not available from some locations according to SiteUptime : January 28, 2011 19:24:09 San Francisco United States Failed January 28, 2011 19:24:33 Chicago United States 0.002 Ok And a downtime a few minutes ago according to 3 external monitoring systems !!!
Posted by techguy85, 01-28-2011, 11:54 PM
I see. That was sent out a while ago and I would be willing to bet it got put on hold when all this happened.
Posted by USEYOURHEED, 01-28-2011, 11:57 PM
wiredtree down again tonight. Too much. Down again at 10:48 pm Eastern on Jan 28 2011. I'm posting a link to this forum on twitter - so real time tweets for wiredtree will show this site and maybe help someone else. This is my second web host that has cost my career. I've lost two potential clients and my other clients are ready to leave. About 18 months ago I lost a $100,000 a year client because of crap like this. Yeah, if I ever get my web dev business built back up -- I'll have to have TWO web hosts. Why can't wiredtree spend some bucks to get some help from whoever helps the big guys? NEVER heard of days after days of DDOS attacks that could NOT be fixed. I'm no SysAdm - but what gives??? So frustrating. I work hard for my money. I'm not young. I had a great "lifetime" deal with wiredtree of a coupon. If I change hosts - not only will I suffer at least a week of grueling moving of sites/code with arthritic hands ... I'll have to pay at least twice the fee. Meanwhile - I can't do work for my clients. Terrible. Wiretree PROMISED me this would never happen. That's why I picked them after the fiacso I had with previous host - which was hostforweb.
Posted by mstickey, 01-28-2011, 11:59 PM
I've got about a dozen sites on a VPS at wiredtree and all appear to be loading ok. Grove is coming up fine for me as well. On a Charter internet connection. However, I would agree that something needs to be done to solve the problem. I do appreciate the fact that they are at least communicating to some extent. I have a ticket requesting some information on the magnitude of the attack and have yet to hear back. Last twitter post I saw indicated that WT doesn't know the who or why for the attack. I've got little to no experience myself with handling DDOS attacks so I can't speak to what needs to be done to solve the problem. Only that other hosts have to have encountered this issue before and I would assume they have a handle on it.
Posted by WiredTree James, 01-29-2011, 12:03 AM
We did experience another DDOS attack tonight. This seems the be the same attacker that has been hitting other similar hosting companies lately. We have been keeping extra network staff on hand and this latest attack was mitigated within 12-15 minutes. Thanks to everyone's support while we fight back. If anyone believes they are having any additional issues, PM your support ticket and I will investigate the issue for you.
Posted by USEYOURHEED, 01-29-2011, 12:08 AM
Has anyone ever heard of a DDOS attack that goes on for days? I'm getting suspicious. I'm looking over reports that I saved from InternetSeerAlert - and I've had plenty of trouble with slowed/stopped connection for months. I'm SO SAD. I loved wiredtree and I didn't not want to move. But last time I was patient with a host ... it cost me big time. What are you guys doing? Thanks!
Posted by Kevin K, 01-29-2011, 12:10 AM
Running for the hills, lol.
Posted by WiredTree James, 01-29-2011, 12:11 AM
We completely understand that any DDOS can be frustrating. Please know that because of the recent DDOS attacks on hosting companies, we have been keeping extra network administrators on staff (even over the weekend). This attempt to attack WiredTree tonight was mitigated within just 13-15 minutes. We will continue to fight back against the scourge. Please PM me with a ticket ID if you think you are experiencing any problems. I will make sure your requests are looked into.
Posted by Energizer Bunny, 01-29-2011, 12:13 AM
Lets see what happens after the storm has passed... afterall they do have this : http://www.wiredtree.com/legal/sla.php "100% Network Uptime WiredTree guarantees network availability for our public Internet network to be 100%. In the event that our network does not experience 100% network uptime in a given month, WiredTree will refund 5% of the customer's monthly service fees for those servers affected, for each 30 minutes of network downtime experienced up to 100% of the monthly service fees for those servers affected." Bottom line they will need to provide free hosting services for a month atleast to almost (well i had downtime of about 1-3 hrs yesterday + 20 mins today) all the clients. It did be nice to actually not have any network downtime, server has been running fine, just network cut off, so was not power failure or natural disaster at play here. Maybe they can find the ddosers and sue them for billions. Last edited by Energizer Bunny; 01-29-2011 at 12:16 AM.
Posted by Fastian, 01-29-2011, 12:14 AM
Yea I have this. But if you look at the maintenance timings and when the issue happened, it was clearly out of that maintenance window. But they seems to think it's ok to hide behind "Maintenance notice" Yea, above 1 hour downtime wan't that "noticeable"
Posted by gauson, 01-29-2011, 12:15 AM
Hope WT resolve the issue I don't want to change server yet.
Posted by USEYOURHEED, 01-29-2011, 12:16 AM
My sites are back up. But I'm still nervous. Thanks wiredtree -- at least you guys communicate with us. Most appreciated WiredTree James. It's that kind of extra effort that makes me believe in you. I've been web dev since 1992 - I've been through heck and back again with hosting companies. Please - I do not ever want to move hosts again. So I'm praying to the hosting gods to bless wiredtree. And sending voodoo to the terrible people with nothing better to do than cause trouble via DDOS attacks. OK - webhostingtalk forum - have I posted enough messages to leave an icon yet.
Posted by mstickey, 01-29-2011, 12:17 AM
WT Twitter post indicates that DDOS should be filtered now.
Posted by USEYOURHEED, 01-29-2011, 12:26 AM
Ah ... Energizer Bunny - thanks for the excellent post. I knew I remembered feeling quite safe moving from hostforweb to wiredtree - I just didn't have time to find the link. So thank you so much for the helpful post. (sorry - Guess I'm still not allowed to use quote) I think we ALL want wiredtree to succeed. And yes, I do hope they take legal action against someone. I'm just a one-person web shop and count on the host to keep sites up. They have been very very good in the past. They just need to take care of this issue - or naturally we'll begin to worry and doubt. Bad for wiredtree business - and bad for my business. It's so embarassing as I've recommended wiretree to everyone - including friends and family. Doh.
Posted by diesel12, 01-29-2011, 12:30 AM
@Energizer Bunny: good luck with that. Not gonna happen. Last edited by diesel12; 01-29-2011 at 12:42 AM.
Posted by NeoComp, 01-29-2011, 01:14 AM
Because wiredtree.com and grove have been unavailables in recent outages ... WHT is one of the best ways to find info about massive downtimes actually
Posted by Energizer Bunny, 01-29-2011, 02:32 AM
I suppose ... i did not get no maintenance notice, so if they did send that notice was probably to cover their losses.
Posted by Valderrama, 01-29-2011, 06:56 AM
I'm with wiredtree.com and have been for over a year. Until about October they were pretty good. Sure, we had issues with our site going down and there was the occasional outage, but support was always excellent. Now though, it's a very different story. In the last few weeks, with support tickets set to "Emergency" they've taken over 1 hour to respond. By that time often the site has come back and you get the "Everything looks good from our end" stuff. There's also a couple of bad stories I could list about support knowledge seeming poor, but at this point we're hoping it's a blip. So, server response getting progressively slower, the time to answer support tickets getting worse, we're seriously considering alternatives. Now if Knownhost just had phone support...........
Posted by XTremo, 01-29-2011, 07:40 AM
In reality you never need it.....as sheba59 has said before, you tend to actually forget about KnownHost cos it just works! I probably use Support more than most because I'm virtually incapable of doing anything server-side myself.....so really they just take up my slack. And when I do want them, they're there 24/7/365 within 10 minutes (with very knowledgeable staff).....you can't ask for more than that.
Posted by WebHosting Zac, 01-29-2011, 07:48 AM
The maintenance was separate and planned in advance as part of our network upgrades we spoke about earlier in the thread.
Posted by Valderrama, 01-29-2011, 07:51 AM
I know what you mean. I suppose I must be a bit old fashioned in that I'd just like the safety net of actually being able to speak to someone and know they're there. Can't deny they seem pretty impressive though.
Posted by Sohan, 01-29-2011, 08:06 AM
If they give nothing from their SLA (making it useless and worthless), then as soon as everything is stable again I'm moving. Everyone is losing serious money through no fault of our own, and then for Wiredtree to not honor their SLA - well f'that.
Posted by Harzem, 01-29-2011, 08:35 AM
Everyone losing serious money is implementing failover/redundancy solutions like me. Others are only trying to avoid an extra $50 per month, and any money lost more than this amount is their fault. You can't find a provider who will never go down. If you are losing serious money over some downtime, spend a fraction of it for redundancy solutions and don't lose that money.
Posted by itcracker, 01-31-2011, 07:51 AM
Yep on 28-01-2011 their grove was down.. it was not opening. Also past few months time for replying tickets has increased. Dont know what is happening, still reliable..
Posted by mannix, 02-01-2011, 09:38 AM
WiredTree promises are as good as their uptime. Joe Doss promised me they would do much better after a provisioning and uptime fiasco on one VPS with them. I was dumb enough to order a second and the experience was even worse. WiredTree: Never again!
Posted by Amitz, 02-02-2011, 12:42 PM
My dedicated with WiredTree is down since over 35 minutes - Am I the only one having problems right now (server issue) or are others facing the same problem (service issue)? My ticket (marked as "critical") has not been answered during the last 15 minutes so I thought I ask here to see if someone has the same problem while waiting for their response. //Edit: Okay, I received a response and it seems as if it is an issue with my server itself. Last edited by Amitz; 02-02-2011 at 12:45 PM.
Posted by Fastian, 02-03-2011, 09:32 AM
Well, I think WT is back on track now. (No issue in last few days) Not sure if anyone received any credit for SLA.
Posted by humawebdesign, 02-03-2011, 12:25 PM
am also facing same problem this is really annoying me why all these things happen
Posted by racersimage, 02-03-2011, 02:01 PM
I don't believe that they believe in their SLA or truly care about their customers. I've received no credit for any of the 3 known outages since December nor have they increased allotted resources. Seems that Knownhost is the only ones that care about keeping their customers with the annual increases that they give out and that's without having to compensate for downtime.I still have my WiredTree VPS but am looking to move on to someone who cares.
Posted by WebHosting Zac, 02-03-2011, 02:40 PM
We always follow our SLA, and always have, make sure you are: http://www.wiredtree.com/legal/sla.php Like Fastian said, the issues were resolved so please open a ticket if you are experiencing issues, huma.
Posted by CD Burnt, 02-04-2011, 01:46 AM
looks like the group problems are in the past. /thread.