HostDime Orlando DC2 Outage

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > HostDime Orlando DC2 Outage

Posted by HostDime, 01-10-2011, 12:34 PM
Hello all, We are currently investigating the current outage our Orlando DC2 location is experiencing and we will update this thread as we have more information. Thank you for your patience at this time.
Posted by ccalby, 01-10-2011, 12:41 PM
Ahhhh Thats why my server from VPSLatch is down ^_^ Thanks HostDime. Look forward to your response.
Posted by jj@24khost, 01-10-2011, 12:47 PM
so far 27 minutes of downtime
Posted by Aeroz, 01-10-2011, 12:52 PM
Im hosting on HostDime, my domains are not available, including hostdime main web. Any one have some news? I hope that everything is ok with them
Posted by M_R_W, 01-10-2011, 12:54 PM
Theres a post in the Outages forum http://www.webhostingtalk.com/showthread.php?t=1012819
Posted by skullbox, 01-10-2011, 12:56 PM
This is unreal.
Posted by quad3datwork, 01-10-2011, 12:56 PM
Thanks for checking up on this. Got two (diff vendor) VPSs at this location.
Posted by Aeroz, 01-10-2011, 12:56 PM
Thanks a lot dude!
Posted by Agurus, 01-10-2011, 12:57 PM
Anyone know whats up with hostdime.com? My server is offline, their site is offline and their phones are all cut? They were adding additonal UPS but its way past 9 AM EST:
Posted by Dustin B Cisneros, 01-10-2011, 12:58 PM
Via there Twitter looks like some connectivity issue that they are working on. All our servers with them are down -.-
Posted by Aeroz, 01-10-2011, 01:00 PM
I guess hostdime.com.mx is alocated on Orlando, right?
Posted by Mr Terrence, 01-10-2011, 01:01 PM
Why is is unreal? outages happen..
Posted by Dustin B Cisneros, 01-10-2011, 01:03 PM
Ours are down i will keep an eye on this thread. This is why we have our corp site offsite so we can still communicate with our clients.
Posted by ErnieQ, 01-10-2011, 01:03 PM
Any updates?
Posted by skullbox, 01-10-2011, 01:05 PM
Because it's their main location and with all the customers/equipment in there I would assume they could manage to keep a major outage under an hour.
Posted by ccalby, 01-10-2011, 01:06 PM
I agree, I think though we should all be rest asured knowing that a level 3 tech has been paged to come in-from home (If there wasn't one on site). A massive outage like this doesn't tend to go un-noticed.
Posted by Dustin B Cisneros, 01-10-2011, 01:07 PM
Im noticing ours coming back online :d
Posted by ErnieQ, 01-10-2011, 01:10 PM
I'm sure they are on top of it..
Posted by racassel, 01-10-2011, 01:19 PM
I am in the middle of migrating 30 Exchange mailboxes and will have to start over. Is it a repeat of the 2008 outage.... I'm not feeling to0 good that they can't even answer their phones or put up a recording, or fail over to a static page that says we're working on it. Im in Orlando and want to drive over there to make sure they aren't loading up my servers and moving to Costa Rica .... the lack of any phone or their own webpage down is totally unacceptable.... maybe they should host their site at Rackspace.....Just in case... asdf-datacenterknowledge.com/archives/2008/05/23/power-outage-affects-hostdime-data-center/"
Posted by Konrad4, 01-10-2011, 01:20 PM
Seems like someone else posted around the same time as you in this forum. Hope it gets resolved soon.
Posted by jj@24khost, 01-10-2011, 01:20 PM
They have a voice over ip phone setup so they can't put up a recording if there are network issues!
Posted by Dustin B Cisneros, 01-10-2011, 01:22 PM
They just expanded their DC and everything for the most part runs smooth there. I dont think they would be leaving to Cost Rica with your server save your self the drive
Posted by racassel, 01-10-2011, 01:23 PM
remove the asdf- above. It would not let me post a url. History repeates it's self???
Posted by HostDime, 01-10-2011, 01:28 PM
Hello everyone, Our Datacenter technicians are working on this issue and restoring service as quickly as humanly possible. We will be updating this thread will a full incident report as soon as we have information to provide further. We thank you all for your understanding at this time and we assure you we are taking all steps necessary to restore service to all of our clients.
Posted by racassel, 01-10-2011, 01:29 PM
Corporate offices are in downtown Orlando, the NOC is 12 miles away......Phones should be ringing...Guess all their eggs should not be in one basket, neither should mine.... taking a real lesson the hard way with this.... my phones are ringing off the hook, and I don't know what to say to my customers other than... ummmmm....they're working on it. All their ads say, "redundant power" "redundant internet" .....
Posted by jj@24khost, 01-10-2011, 01:29 PM
I have no doubt they are. How long before they get to the sla mark of 99.5% uptime?
Posted by racassel, 01-10-2011, 01:32 PM
Thanks for the update...,how about just a rumor and a w-a guess about what went wrong, and when it will be fixed?
Posted by skullbox, 01-10-2011, 01:34 PM
That's what I was saying. A setup that size should have N+1 for everything. It would be nice to know what the problem is, even if they haven't fixed it yet. I only have 1 exchange server thank god.
Posted by racassel, 01-10-2011, 01:35 PM
99.5 means they can be down six months out of 10 years?
Posted by HostDime, 01-10-2011, 01:36 PM
Hello, As far as what went wrong, we had an unforseen issue with our scheduled power upgrade. The issue has already been resolved and service is being restored to servers as quickly as humanly possible. We already had a larger than usual staff at DC2 today for the upgrade and we have brought on every technician possible to work on this issue. We will continue updates as we receive further information.
Posted by ccalby, 01-10-2011, 01:38 PM
Ahhh well, at least your staff was prepared. May I suggest though, making your website fail over if that is possible for you, and even making you phones available during times like this.
Posted by ALS, 01-10-2011, 01:39 PM
VPSLatch just inform it was server maintenance. So that doesn't count against uptime or how they justify it. Last month it was someone on the node that took the downtime to: 91.935% with 5 outages. This month at: 99.128% with the 2 hour outage. Both outages they explained away. So neither counts against the uptime guarantee.
Posted by vasho, 01-10-2011, 01:40 PM
Hello ! It looks strange .Hostdime.in was down for almost 7-10 hours yesterday .But all the servers including core.hostdime.in were ok.Yesterday Hostdime.com don't know about that issue until I reported them after waiting for 9 hours But ,Today all the websites -hostdime.in,hostdime.com and my server from hostdime are down .Are every one having this issue? Are any one having trouble with hostdime at this moment? Attached Images hdin down-9 january.jpg (165.4 KB, 85 views) hostdime servers are down-10 january.jpg (164.5 KB, 66 views) Last edited by vasho; 01-10-2011 at 01:46 PM.
Posted by racassel, 01-10-2011, 01:43 PM
If it was a power outage, could some one go to 34/6 and push the button to turn my server on? When I see hostdime.com come back up , i'll know they are making progress.
Posted by skullbox, 01-10-2011, 01:45 PM
Power upgrade at ~11:00 AM EST... Who scheduled that?
Posted by ALS, 01-10-2011, 01:48 PM
Yep we were informed of that but it was to be at 9:00am est. The downtime me started at: 10:20am est. As far as we were told we weren't suppose to get any downtime or no more then 20mins if we did. So it doesn't count against the guarantee or uptime.
Posted by racassel, 01-10-2011, 01:50 PM
Happy New Year and Happy Monday! I know I am going to get fired a dozen times as soon as it comes back online. My customers don't have any patience and pay me good money to keep this from happening... I have to say they have been a very good company with excellent support....but I've only been with them for six months... They are 1/2 the price of the Colo next door, so we'll wait and see.... gotta go, six lines are blinking ,all wanting to know that is going on.....Glad it's not my problem....
Posted by racassel, 01-10-2011, 01:54 PM
All I was aware of was what is coming up this Saturday: Maybe the IOS gave up the ghost too.... Hope it lasts till Saturday.... Good Evening, As part of our commitment to provide quality web hosting services, network maintenance has been scheduled at our datacenter (DC2) in Orlando, Fl. Please read the details carefully. Scheduled Date and Time: ================ Saturday, January 15, 2011, 2 AM â€“ 4AM EST Maintenance Type: ================ Our admins will be updating the IOS on the distribution switch. This will fix an issue that has been causing intermittent issues with some clients noticing latency or sporadic packet loss. Estimated Downtime: ================ We will do everything in our power to expedite the maintenance. While this will most likely go un-noticed on your end, there may be up to 5 minutes of latency/downtime during the maintenance window while the IOS is updated. If you have any questions or concerns, please feel free to submit a support ticket or contact us through our Live Concierge Chat system. Thank you very much for your attention to this matter.
Posted by skullbox, 01-10-2011, 01:57 PM
I remember getting that one, but don't remember seeing anything about a power upgrade? Although, the server that is down is my exchange server so I can't even get into it and check...
Posted by XeHSean, 01-10-2011, 01:58 PM
They don't seem to work on my end either. It looks like they are already working on the issue - http://twitter.com/hostdime
Posted by skullbox, 01-10-2011, 01:59 PM
The hostdime.com site is showing now. That's progress!
Posted by teck, 01-10-2011, 01:59 PM
It's being discussed here: http://www.webhostingtalk.com/showth...post&t=1012819
Posted by jj@24khost, 01-10-2011, 01:59 PM
well hostdime.com is now coming back online
Posted by ALS, 01-10-2011, 01:59 PM
Yep I got that one too for Jan 15, 2011 from HostDime and one from VPSLatch for today Jan 10, 2011.
Posted by Kevin K, 01-10-2011, 01:59 PM
Have a look at this thread. Always good to check the Outages forum first just to double check. http://www.webhostingtalk.com/showthread.php?t=1012819
Posted by racassel, 01-10-2011, 02:00 PM
Skullbox is your home page supposta look like that?
Posted by skullbox, 01-10-2011, 02:02 PM
Temporarly yes... Long story. That site isn't hosted at hostdime though.
Posted by VL-Adam, 01-10-2011, 02:04 PM
We apologize about the outage, since all of our production servers are based out of HostDime/DimeNOC our clients VPS were unfortunately affected about this small outage. Again, we apologize and I'm sure HostDime has taken steps to ensure such from happening again in the future. Thanks for your understanding That's a monthly uptime guarantee as far as I understand. The servers at the datacenter were online before their website went back online. Let's all be a little bit more patient here and let HostDime completely resolve the issue, they are a solid and reliable company so we can all be rest assured they are doing what it takes to bring our servers back online. Things appear to be coming back online at this time.
Posted by vasho, 01-10-2011, 02:05 PM
Hostdime.com is up now.I think they are working on hostdime.in.HDIN and My server are down. I haven't see any downage of vpslatch at that time.That was ok.I think that was just for a few minutes as VPSLatch's servers are under HDUS .
Posted by gymgangsta, 01-10-2011, 02:07 PM
At 8:25 AM PST, I saw all my websites go down. I fielded several phone calls from potential customers affirming them that we are still in business. Who knows how much money is being lost. I am kicking myself for still hosting with hostdime after 5 years with them. I've had nothing but grief from them. Their own site hostdime.com is down so you cannot get an update. Why don't they at least host that somewhere..??? Instead I had to track down this forum through Google...to find out what's happening. And I don't like that their phone is Voice IP, cause that goes down too.... So i freaked..and I'm moving my stuff to hostgator. enough said. I can't take this anymore...I got a business to run!! So long hostdime.... Attached Images hostdime-no-resolve.jpg (48.1 KB, 64 views)
Posted by HostDime, 01-10-2011, 02:08 PM
Hello everyone, Our internal systems are restored and we are now working to respond to all customer queries regarding the downtime. We will be continuing to update this thread as we have further information. Thank you again for your patience.
Posted by vasho, 01-10-2011, 02:09 PM
HDUS servers are up now HDIN is still down Update : @hostdime.com Thanks for your very quick response!
Posted by vasho, 01-10-2011, 02:13 PM
Server outages happen some times I am with hostdime for a few days and I love it. Their support as well as service seems perfect to me.
Posted by vasho, 01-10-2011, 02:16 PM
Thanks for every one ! HDUS is up . They should have mail us about it. Yesterday their HDIN(hostdime.in's main site) is down for 9 hours.They don't have that information until I report about server downage after 9 hours to hostdime.com But I still love their support.I vote them next to hostgator's support.I never faced server down issue from hostdime Last edited by vasho; 01-10-2011 at 02:31 PM.
Posted by HostDime, 01-10-2011, 02:19 PM
Thank you for your kind words. We assure all of our clients that we are working as quickly as possible to resolve this issue and we will be doing a full incident report as soon as we have further information regarding this downtime.
Posted by gymgangsta, 01-10-2011, 02:20 PM
Just you wait, your site will have troubles and they will keep insisting it's your site and that you need to upgrade to a dedicated server, they constantly pitch you for an upsell. I'm on VPS and they do it to me all the time saying one of my scripts is overloading their server etc... Hostgator said that is nonsense. Their servers should support those scripts no problem... I don't want to get off topic here and bash hostdime anymore. Good luck to those who stay.. I've had enough after 5 years, this was the straw that broke the camels back... Last edited by gymgangsta; 01-10-2011 at 02:24 PM.
Posted by gymgangsta, 01-10-2011, 02:23 PM
10:21 AM PST and all our sites are still down, VPS still down. Yet hostdime.com is up? That's not right. 10 packets transmitted, 0 packets received, 100% packet loss
Posted by VL-Adam, 01-10-2011, 02:24 PM
Things appear to be back online at this time. HostDime was on top of the issue and resolved things promptly. http://www.webhostingtalk.com/showthread.php?t=1012819
Posted by jj@24khost, 01-10-2011, 02:25 PM
core.hostdime.com is now down again
Posted by racassel, 01-10-2011, 02:26 PM
I guess Colo clients come last. Could someone go to 36/4 and see if the power in on?
Posted by VL-Adam, 01-10-2011, 02:29 PM
It's possible that your VPS is still booting up. It's online here.
Posted by Rick-WHSuite, 01-10-2011, 02:31 PM
We've got an offsite reseller account for internal stuff. Its still down.
Posted by skullbox, 01-10-2011, 02:32 PM
I second that. I still can't reach my firewall, so I know power hasn't been restored to my stuff yet.
Posted by HostDime, 01-10-2011, 02:32 PM
We are working to return service to every single server as quickly as possible. Some VPS servers may be rebooting quicker than others as the Hypervisor prioritizes booting of VEs. We will ensure this process goes as quickly as possible. Thank you
Posted by jj@24khost, 01-10-2011, 02:33 PM
It went down for a few minutes and then came back up.
Posted by jj@24khost, 01-10-2011, 02:36 PM
2 hours with out my server being up.
Posted by jpmaster, 01-10-2011, 02:40 PM
We need urgent help with HD7542 We see many dedis online now. Our server is still down!
Posted by domaincart, 01-10-2011, 02:50 PM
I cannot access to all of our reseller accounts that hosted Hostdime and Surpasshosting servers for few hours. Also I cannot access to Surpasshosting web site. Do you know the problem?
Posted by VL-Adam, 01-10-2011, 02:52 PM
Hey buddy, there is a thread about this and things appear to be starting back up now. Take a look at this thread, hope this helped
Posted by jj@24khost, 01-10-2011, 02:57 PM
still not able to get answers
Posted by HostDime, 01-10-2011, 02:59 PM
Hello, I apologize that your server has not been brought online as of yet but I assure you we are working as quickly as possible to return full service to all clients. Thank you
Posted by jj@24khost, 01-10-2011, 03:06 PM
Are server has come back online finally. Hope everybody is getting theres back now
Posted by racassel, 01-10-2011, 03:08 PM
HDCOLO548 - 36/4 ------ Help!
Posted by HostDime, 01-10-2011, 03:10 PM
All remaining servers should be restored soon. The remaining service in our datacenter is being restored at this time and we will continue to update this thread with our most up-to-date status.
Posted by domaincart, 01-10-2011, 03:12 PM
Thank you, Adam.
Posted by jpmaster, 01-10-2011, 03:21 PM
And we're up 'n running
Posted by skullbox, 01-10-2011, 03:23 PM
Me too! I put in a ticket and they responded quickly and I'm back online.
Posted by racassel, 01-10-2011, 03:24 PM
Thank you . It is up now. Accesing damage on 16 Virtual machines from an unexpected POWER OUTAGE
Posted by martincad, 01-10-2011, 03:39 PM
I cannot believe our server hosted in HostDime cannot be reached for more than 3.5 hours !!! Even their website could not be reached for more than 1 hour. This is the second time they have issues in their network in less than 15 days. How can be this possible ? I can't believe they sent me an email explaining the amazing work they have been doing to increase the network availability. This is having an incredible bad impact in our Corporate image. I can't even imagine what are we going to do to get back our customers again !!! I think we should change our Server provider. What do you think ?
Posted by Rick-WHSuite, 01-10-2011, 03:45 PM
Pass 11 is still down here. Also no official word on this 'power implementation issue'. If they were performing maintenance thats idiotic time planning. Monday Afternoon... Last edited by Rick-WHSuite; 01-10-2011 at 03:46 PM. Reason: damn auto-correct...
Posted by HostDime, 01-10-2011, 03:52 PM
We sent out notifications regarding our power upgrade last week to every HostDime client. We apologize if you did not receive this notification but we also posted about it on our Twitter account (http://twitter.com/hostdime) as well as our Forums (http://forums.hostdime.com). As for an update, the vast majority of our datacenter service has been restored. We have all hands on deck working on this issue and we are hoping to see a complete resolution very soon. Thank you all for your patience during time.
Posted by gone-afk, 01-10-2011, 03:53 PM
I think you haven't provided enough information for anyone to provide a reasonable opinion on the matter. Was this scheduled maintenance? If not, what is the reasoning provided for the outage? All providers have outages, what is important is how well you feel they have handled the issue and how likely it is that things will improve. Have you called them to discuss your concerns?
Posted by domaincart, 01-10-2011, 03:54 PM
The issue seems fixed. Thanks Hostdime.
Posted by 24x7group, 01-10-2011, 03:55 PM
What do they say about the downtime caused? Are they aware of it or not?
Posted by Rick-WHSuite, 01-10-2011, 03:55 PM
With respect, why was this being done in the middle of a Monday? What was wrong with say, 3 AM this morning? Ludicrous... Also, our account is at surpass and we received no email informing us that we should expect a major outage in the middle of the day.
Posted by Aeroz, 01-10-2011, 04:17 PM
All working for me, thank you.
Posted by skullbox, 01-10-2011, 04:19 PM
My concern as well...
Posted by martincad, 01-10-2011, 04:19 PM
This is the auto-response the give you when you try to reach them: "Thank you for your inquiry. At this time we are currently investigating the matters which took place that caused this unforeseen power issue. We sincerely apologize for this unfortunate circumstance. At this time, all servers are coming back online as quickly as humanly possible. We will keep you updated with all available information, and for the time being, please go to our forums and twitter accounts for any further updates:" We tried to contact them by phone for more than two ours but their lines were busy all the time.
Posted by martincad, 01-10-2011, 04:21 PM
Do you think it is Ok to have your site down for 4 hs 20 minutes in office time ? Of course was not scheduled. They recognize they are having big issues but that is not enough for me.
Posted by plumsauce, 01-10-2011, 04:21 PM
Think about it. It is always 3:00AM somewhere in the world. There are two things that can happen. 1. things go right, and no one notices a thing. Then the scheduled time is irrelevant. 2. things go wrong. and it's all hands on deck. If it is noon EST, the east coast is awake, the west coast is just getting to work, and it is just after 5pm in Europe. That's eally not too bad. At least it isn't causing everyone to roll out of bed and deal with it while 3/4's asleep. The 3:00AM thing only really applies to corporate 9-5 servers where the window can let you get things right before the CEO wakes up to check his email. As for notifications from surpass, isn't that a whole other company? Last edited by plumsauce; 01-10-2011 at 04:26 PM.
Posted by racassel, 01-10-2011, 04:25 PM
Good thing the server is up. Now I can fix the EXCHANGE STORE ! Luckily it's only one. Exchange servers and HDD Controllers don't like it when you UNPLUG THE POWER WITHOUT A BATTERY BACKUP. Guess I need to carry a battery backup over there, as their redundant power is NOT! Some data is missing for sClientPath=D:\ Cache\Cache\1259942945625\COMxxxxx\Microsoft Information Store\First Storage Group\Mailbox Store (COMPxxxxx)\priv1.edb, sServerPath=G:\Dait Users3\user\carolbu@tcom\files\1257880994875\Current\0x38\0xb7\0xf7\0x32\19 55764888, lSEQ=20349, lExpectedFileSize=2666939392, lSizeInDisk=2495104000
Posted by skullbox, 01-10-2011, 04:26 PM
Seriously, anything would have been better than 10 AM EST on Monday. Hell, I'd take anytime on Sunday. At least those of us who live and die by normal business hours would be least affected. I am interested to see this scheduled outage notice about upgrading the power. The only notification I received was to update the IOS, which was scheduled for sometime in the early AM.
Posted by HostDime, 01-10-2011, 04:30 PM
I understand your concern and we will be addressing all concerns in our incident report. As far as your account being at Surpass, I can assure you that the notifications were sent out to Surpass clients as well. If you did not receive this notification, please check your Spam folder and if it is not there, please open a ticket in the help desk so we can investigate this to prevent it in the future. For another update: We are still assessing the situation but we are closing in on a complete resolution of this issue. We apologize again for the inconvenience but we are working as quickly as possible to resolve these issues for all of our clients affected. Thank you
Posted by ItsChrisG, 01-10-2011, 04:31 PM
All datacenters have downtime, stuff happens, no matter what.
Posted by plumsauce, 01-10-2011, 04:32 PM
You were posting while I was adding. See the bit about 9-5 servers. The thing is, in a mass all hands on deck situation, if it is 10AM local time, all of them are there, had their morning wobblies, are good to go. The other alternative is to send in 1 guy to do the job at 3:00am, and then have to page everyone at home. That is a minimum 1 hour delay. Everyone is half asleep and grumpy. Including the customers who got paged, and are being paged.
Posted by skullbox, 01-10-2011, 04:38 PM
Found this on thier forums, don't think I ever got an e-mail about it: We are now ready to complete the final phase of our facility's power expansion. This involves the integration of an additional 320KVA UPS system to our facility's infrastructure. We have scheduled the integration of this additional UPS on Monday January 10th, 2011 at 9 AM EST (UTC-5). The integration will be completed within two hours. No down time is expected during this integration because this new system has been already successfully inspected and stress tested by a third party. However, in a worst-case scenario, up to 25% of the servers in our data center may experience a brief loss of power and reboot themselves automatically. We do not expect this to happen, but we are well-prepared in case it does. We have scheduled a special standby response team that will take action as needed to minimize the impact of any problems that arise during this work.
Posted by M_R_W, 01-10-2011, 04:55 PM
What everway you look at it, its always going to be a bad time for someone. 3am EST is 8am GMT - Bad time for us Brits, everything down when we arrive at work, not a great way to start Monday morning.
Posted by Ed-Freethought, 01-10-2011, 05:37 PM
That is a pretty bold statement to make given that you don't yet know what part of the infrastructure failed to cause the outage. For example, battery backups (which HostDime have) don't help if there is a problem on the internal power distribution infrastructure.
Posted by HostDime, 01-10-2011, 05:50 PM
Thank you all for your patience. As of now, we are almost at complete service restoration and seeing a small amount of servers still experiencing issues. If you are among those who are still experiencing issues at this time, please update or open a ticket with our Help Desk and we will assess the issue. We are still keeping all hands on at our Datacenter as well as our Callcenter to ensure our clients have the most up to date information possible regarding this downtime. Thank you again for all of your understanding during this difficult time. We assure all of our clients that we will have a full incident report regarding this issue as promptly as possible.
Posted by gone-afk, 01-10-2011, 06:14 PM
It sucks no doubt, but it happens at any provider: ThePlanet explosion (people offline for days) Rackspace outages RealityCheckNetwork (entire server farm hacked, thousands affected, data lost, offline for weeks) Only you can decide if you need to move on, all I'm saying is that despite millions of dollars that the big boys spend, outages still happen. If your sites are that important to you then you probably already have off-site backups or servers provisioned elsewhere to take over/move to. Good luck, hopefully you'll be back online soon.
Posted by larryinfl, 01-10-2011, 06:14 PM
I was with HostDime for 3 or 4 years starting in 2002. Around 2006/2007 there was a two day outage with the server. I lost a lot of clients and got a lot of people mad. In the years with them there were a couple three of four hour outages. I switched nearly 4 years ago to LiquidWeb and have not had a minute of downtime. It appears nothing has changed over at hostdime.
Posted by JohnnyUtah, 01-10-2011, 06:32 PM
After 5 hours of downtime. Im back up at VPSLatch. Cant stand your generic emails. Stop emailing me and get it done.
Posted by Rick-WHSuite, 01-10-2011, 07:04 PM
HostDime: Since this very extended downtime was not part of the original scheduled maintenance, and was clearly something majorly wrong on your end, I assume you'll be offering SLA credit to all affected - correct?
Posted by joako, 01-10-2011, 10:06 PM
What is going on here? When I had this page up earlier it was 6 pages... now it's 2!!??
Posted by Smitty927, 01-11-2011, 01:13 AM
joako, it is 6 pages. All those whining that there was downtime: It is going to happen, at some point EVERY host will go down!! To those like gymgangsta: You will not be happy anywhere. The grass isn't always greener on the other side. To those saying Surpass didn't email about this, they did. I have been with Surpass for almost a year and have been receiving all emails about any update they do, including this one. To Hostdime/Surpass: I would think that a weekend day would even be better than M-F. You should seriously reconsider these kinds of moves and remember what can go wrong will, case in point today! Maybe you should track where your largest count of customers are and start major work as their business day ends. My guess is that 6 or 8 PM PST would probably have alleviated most headaches for people. Luckily I only received one call about this outage and quickly sent an email to all of my customer's alternate email addresses, so I am not feeling it like others did. But I gotta say, if you train your customers well enough, they will know that man-made items fail! It is a fact about technology!! Another piece of advice to help you alleviate the loss of communication: Move your email to Google. Google will host your email for free for up to 50 accounts (more than enough for most people). The changes are made at your registrar, so it completely removes a hosting outage problem. Overall, I have had about 4-5 years of Hostdime/Surpass reseller hosting and have to say that they always step up and take care of things!!
Posted by Dustin B Cisneros, 01-11-2011, 01:39 AM
Our servers are back down. Anyone else? Dime is down for me aswell.
Posted by cruisx, 01-11-2011, 01:39 AM
Same here.
Posted by gymgangsta, 01-11-2011, 02:03 AM
I don't jump ship at the first outage. I've been with hostdime for over 5 years. This is just the last straw for me. Like I said, I didn't like their communication problems, their own domain went down and their voice over IP phone also went with it. That's not acceptable. They can host a hostdime.com anywhere and post blog updates on hostdime.com. Not everyone knows about this forum. They can also change to different phone service. That's just a bit of my gripe...like I said I am keeping my frustration to this outage only
Posted by GSalam, 01-11-2011, 02:09 AM
As a customer, I received an email from HostDime regarding this upcoming upgrade. And they did mentally prepare their customers. I'm wondering how come you didn't receive notice about the upgrad Happy it's up now
Posted by VL-Adam, 01-11-2011, 02:10 AM
According to someone from HostDime, it was a large inbound DDoS that affected a section of their datacenter. Things appear back online and back to normal now, it was only a small 5-10 min outage.
Posted by LoyaltyV, 01-11-2011, 02:11 AM
Ouch !
Posted by vasho, 01-11-2011, 02:40 AM
Servers with HDIN (hostdime.in) were up after 4 hours from the downage
Posted by maniondagger, 01-11-2011, 02:55 AM
I guess that happened because of emails with notifications go to the spam folder.
Posted by vasho, 01-11-2011, 03:48 AM
I think HDIN(Hostdime India)forgot to send notification .How ever the server is up after 4 hours
Posted by TheChemist, 01-11-2011, 04:33 AM
My goodness. I understand the frustrations from the web hosts. But you were notified and HostDime has a twitter account. Not everyone know's of WHT but they do Google. A simple "HostDime down" search phrase would bring you to their twitter page. But if you paid attention to your e-mail you would have known about this SCHEDULED outage. They even mentioned that a possible 25% of servers may go offline. I'm sure a few of you have been to a datacenter. I doubt any during an outage or when serious work needs to be done. They bit the bullet and got everything up and running in a decent time frame. I have had shared hosting down for 3 days before. A datacenter upgrading had a slight malfunction of which you were warned/told about and you are (most) complaining because you have to deal with your clients. If you are complaining because you have to deal with your clients, then why are you in business? I would be pissed if my web host was on this thread complaining about having to answer phones. I understand this is your financial means (some of you) but give the company that is a damn good company a break
Posted by prewebhost, 01-11-2011, 05:10 AM
It apppears it would be fixed..
Posted by Sparrow-Sean, 01-11-2011, 05:13 AM
I think this can be rename, Hosdime is UP?!
Posted by plumsauce, 01-11-2011, 06:18 AM
Excellent point. Especially given that the claim to fame of a reseller is "value add". What used to be called a VAR. (value added reseller). Without the value add, there is no point in adding another level to the sales chain from the perspective of the customer.
Posted by Wes0805, 01-11-2011, 07:22 AM
With all respect to everyone, I don't understand why you should over exaggerate yourself and get too mad with Hostdime. I also own a reseller from Surpasshosting and I was affected by the downtime. I completely understand that we all have our own businesses to run and clients to provide quality services but it is not as if Hostdime did not post any updates about this unforseen outage. I will really get mad if I did not receive any news at all but they were able to utilize their twitter and other media (like this forum) to inform their customers. Getting them pissed off will not help at all. We also have our responsibilities and limitations as customers and we should all remember that.
Posted by Rick-WHSuite, 01-11-2011, 07:32 AM
I agree. However must say (after looking at the twitter feed) that its pretty unfair that techs were favouring Twitter requests for servers being brought back online over standard support requests. Either way - lets hope the person responsible for this gets a good talking to for choosing the middle of a weekday to perform maintenance.
Posted by HostDime, 01-11-2011, 08:31 AM
Hello, I can assure you, no favour was being given to any request over another and we were treating every single support ticket that came in with the highest possible urgency. Please do know though that at many point we had over 200 active support tickets and attending to each and every one took quite a bit of time but we tried to assess the situation in a first come-first serve basis. As for another update, our Post-Outage Report has been posted at our internal HostDime forums regarding the downtime experienced yesterday, 01.11.11: http://forums.hostdime.com/showthrea...0-2011&p=40539 Thank you again for all who supported us during this downtime and for the understanding felt by our long term clients. For those who have hosted with us awhile, you know that we take all downtime very seriously and we care deeply about the service we provide. We assure each and every one of our clients that we are taking every single step necessary to ensure downtime such as the one experienced today does not recur. From our extensive datacenter expansions to our daily growth in technical support staff, HostDime is in the Hosting business for the long term and we hope our clients are with us in the journey. Thank you again and have a great day
Posted by cheminroad, 01-11-2011, 12:15 PM
They are up now indeed.
Posted by PrivateLayer, 01-11-2011, 02:22 PM
When picking a provier you should be asking about there data center, also check the web for different reports on data centers.
Posted by KarlZimmer, 01-11-2011, 03:16 PM
From what I can tell, LiquidWeb HAS been affected by downtime: http://www.web-hostingreview.com/liquidweb/ Now I am certainly not saying that makes them a bad provider, just re-iterating that "everyone has downtime" at some point. They certainly do have a very good uptime record overall.
Posted by LiquidWebBenny, 01-11-2011, 03:40 PM
You're right. Eventually, somewhere, humans and human error get involved. http://www.seroundtable.com/google-n...ine-12783.html We actually took that outage very seriously, and made significant changes to our infrastructure and have since more than doubled our available bandwidth.
Posted by TmzHosting, 01-11-2011, 04:02 PM
Outages will always happen. Things Happen! But if those things start happening more often than it's time to move . - Daniel
Posted by LiquidWebBenny, 01-11-2011, 04:08 PM
Agreed. And, in my opinion, it's not the uptime that really is the good way to judge a host, it's how they react when you go down.
Posted by Coolraul, 01-11-2011, 04:13 PM
If uptime is critical for you (really critical not just I want it up!) then you HAVE to have redundancy between providers. Even a smaller system that allows you to stay online in degraded mode. That is why I like the things datacenters are doing in diversifying geographically.
Posted by bhavicp, 01-11-2011, 09:02 PM
+1 for that. Like everyone else said, a provider's going to have downtime for one reason or another. It's best to have your own redundant infstructure, so you're not affected by it.
Posted by JakobG, 01-11-2011, 09:42 PM
Downtime is one of the risks with hosting. Like someone before me posted... Go with a redundant infrastructure. This will avoid having any big downtime.
Posted by skullbox, 01-11-2011, 09:48 PM
Here is the issue... Yes it was scheduled. HOWEVER... Here are the two things that scare me: 1. The scheduled outage said up to 25% of the servers in the data center may loose power for a few minutes. If that doesn't concern everyone reading this, you better get out of the IT world. 2. They choose to do this on a Monday morning... There is absolutely NO EXCUSE... NONE, to do this on a Monday morning. Eastern or Pacific time, everyone in the US was affected by this. Did they forget days like Saturday or Sunday occur every week of the month? This is by far one of the stupidest outages I have ever seen in 10+ years of IT. So, The Planet had their power transformers completely blow up in Fort Worth. But at least they didn't knowningly cause a possible outage (after admitting it could affect up to 25% of the data center), on a f*cking Monday morning. I have searched my inbox high and low. I never got notice of this. However, I did receive all the other ones. Had I read they were going to do something on a Monday morning that could affect up to 25% of the data center I would have flipped out. I'm not into twatting, so I don't check twitter, but I was able to find this on their forums: We are now ready to complete the final phase of our facility's power expansion. This involves the integration of an additional 320KVA UPS system to our facility's infrastructure. We have scheduled the integration of this additional UPS on Monday January 10th, 2011 at 9 AM EST (UTC-5). The integration will be completed within two hours. No down time is expected during this integration because this new system has been already successfully inspected and stress tested by a third party. However, in a worst-case scenario, up to 25% of the servers in our data center may experience a brief loss of power and reboot themselves automatically. We do not expect this to happen, but we are well-prepared in case it does. We have scheduled a special standby response team that will take action as needed to minimize the impact of any problems that arise during this work. I can honestly say I am shocked a company as large as hostdime operates like this.
Posted by Layer3, 01-11-2011, 10:09 PM
I couldn't agree with you more. I'm going on 10 years in IT as well and this has to be the most idiotic thing I've heard of from a company like HostDime. I expect this type of stupidity from a 14 year old running web hosting business. Almost like they didn't want to pay the overtime for the full staff to be on hand during the weekend night hours.
Posted by HostDime, 01-11-2011, 10:27 PM
1. We would like to address some concerns that were raised so that everyone understands why we scheduled the upgrade when we did. The upgrade was done at that particular time and day for reasons that we believed far outweighed the potential issues. *We have been around for over 8 years, and we do understand the importance of choosing the appropriate maintenance window times. A. Based on our knowledge of the upgrade procedures, what was about to take place, and the historical data we continually collect from our electrical service, Monday morning was the best time to rely on commercial power. Based on data collected throughout our entire history, there was never a loss in commercial power on Monday mornings. Because we knew we had to rely on commercial power for 1 hour while the work by our electrical engineers was completed, we wanted to pick the best possible time. Since many of our electrical panels were also being upgraded, we could not count on our generators during this work. B. We felt that it was crucial to have our Eaton/UPS systems manager on site just to be prepared in case anything went wrong. This person knows our systems personally and was the best person to have on site that day. *The earliest available Monday timeslot that the systems manager was available was January 10 (due to the holidays and a busy schedule). This was another reason why we selected this day and time. C. The upgrade needed to be done as soon as possible to create a more secure and stable environment in our facility. Time was of the essence. 2. SLA Credit - We will be issuing credits per our SLA for anyone effected by this outage. While we realize it was maintenance, things didn't go as planned and we take responsibility for this. We will honor our SLA in full. Please file a ticket to get this credit added to your account. 3. Communication - We understand communication is crucial during times like this. Unfortunately, there was a breakdown in our lines of communication, and for this we apologize. We want to assure you that we will be making any and all necessary changes so that this will not be an issue in the future. *This will include having a backup system in place so that we will still be able to be contacted in case of any future outages. As mentioned above we do not take any downtime lightly. We know what a great responsibility and obligation we have to each and every one of our clients. Our team mobilized quickly and we had well over 20 support personnel on site within minutes of the power outage. Regardless, we pledge that an issue like this will never happen again. We also would like to take a moment to thank those who understand our situation and were exceedingly patient during this situation, but in the end, the buck stops with us. We will do whatever we can to make this up to our clients. We have recently invested over $2M of our own funds to expand our facilities with better technologies to benefit all of our clients. We couldn't do it without you. Thanks for your support!
Posted by ALS, 01-11-2011, 10:32 PM
Thanx for clearing that up for sure. My VPSLatch server is still offline so it makes no difference if I get credit or not.
Posted by HostDime, 01-11-2011, 10:41 PM
Please check our most recent reply to our downtime post for further information regarding some of the questions raised in this thread which we agree needed to be addressed fully.
Posted by bhavicp, 01-11-2011, 10:44 PM
Link please? Edit: NVM Found it: http://www.webhostingtalk.com/showth...1012819&page=7
Posted by HostDime, 01-11-2011, 10:47 PM
http://www.webhostingtalk.com/showpo...8&postcount=92
Posted by AdmoNet, 01-11-2011, 11:20 PM
Or you can pick a host that has systems in facilities that house big players like Visa, Burger King etc :-) I may know a host who chooses enterprise class facilities :-) At any rate, all datacenters are subject to downtime. It just depends on how well the power, network and cooling is engineered. Good luck with hostdime.
Posted by PrivateLayer, 01-12-2011, 02:36 PM
Outage's dont happen in every data center, thats what separtes the great providers from the rest!
Posted by gymgangsta, 01-12-2011, 02:43 PM
I did receive the email on jan 6 regarding the down time: "We have scheduled the integration of this additional UPS on Monday January 10th, 2011 at 9 AM EST (UTC-5). The integration will be completed within two hours. No down time is expected during this integration because this new system has been already successfully inspected and stress tested by a third party." It was down quite some time on a busy Monday morning. I lost a bunch of sales. In my business people buy when they arrive at work on Monday's at 9am. Who knows how much revenue went down the toilet. Regards to communication, I was not aware of a Twitter account. I simply went to hostdime.com and then tried calling. Both failed. Not everybody is savvy enough to find them on Twitter and this message forum. Communication sucked on their part. I'm still mad. And switching providers...
Posted by KarlZimmer, 01-12-2011, 02:51 PM
Sorry, but yes, they do. Even those who spend the money, such as Equinix, Google, and Rackspace have outages. That a facility has not yet had downtime does not mean they will not. Never underestimate the potential problems due to "human error."
Posted by HostDime, 01-13-2011, 09:11 AM
Hello gymgangsta, Can you please post your domain you have with us here or PM it to us? We would like to contact you regarding your concerns regarding your service with us and ensure your transition to another host is as smooth as possible. Thank you
Posted by KarlZimmer, 01-13-2011, 12:25 PM
Show me someone that has a Tier IV rated (not just a hollow claim) data center that is selling space out of it... The only Tier 4's I know are for high end financial facilities, security operations, or telco cores. None of those would sell space to other tenants. Last edited by KarlZimmer; 01-13-2011 at 12:33 PM.
Posted by M_R_W, 01-14-2011, 06:50 PM
Is there another problem with HostDime? Cant access their site (Or my VPS with a provider based in their Orlando DC2?)
Posted by GORF, 01-14-2011, 06:52 PM
Yup, here we go again !! Back up. Looks like it was a network issue this time as the servers did not reboot.
Posted by Rick-WHSuite, 01-14-2011, 06:52 PM
Yeah somethings up - our VPS is stupidly slow (took 2 mins to load a static page. Bad month for HostDime!
Posted by M_R_W, 01-14-2011, 06:54 PM
A short one this time.. Everythings back up my end
Posted by HostDime, 01-14-2011, 07:08 PM
We're investigating client reported issues with latency now. I will update this thread when we have further information. Thank you
Posted by GORF, 01-14-2011, 07:14 PM
We are not talking latency, there was NO connection to anything.
Posted by HostDime, 01-15-2011, 01:22 AM
The issue we experienced earlier was a small network outage only affecting a portion of our datacenter. We have located the issue and put into place measures to prevent a recurrence of this issue. Thank you
Posted by TheChemist, 01-15-2011, 04:24 PM
Don't forget, HostDime was also responding to requests from disgruntled members here. And how do you know that they were favoring support twitter requests over tickets. I am sure they had more then one person dealing with disgruntled clients.
Posted by Rick-WHSuite, 01-15-2011, 04:42 PM
Funnily enough, by reading their twitter feed... http://twitter.com/#!/hostdime/status/24542950130192384 http://twitter.com/#!/hostdime/status/24541232214573056 Expedited on request of twitter users...either way - I'm not bothered..just thought it was a little unfair on everyone else that twitter users nagging took priority over nagging elsewhere.
Posted by HostDime, 01-15-2011, 05:08 PM
I don't quite understand why you may think that a person over twitter was given any priority. We were trying to assist every client as quickly as possible over all of our support mediums, twitter included. We treat it just as many of our other support systems (Desk, Live Chat, Calls, Forums) and try to assist our clients there as well.
Posted by th1chsn, 01-19-2011, 02:11 PM
I've had a reseller plan with HostDime for a few years and overall they provide great value for the price and also great support . Compared to other hosts HostDime gives flexibility over the server settings where other hosts require me an upgrade. For example, I have a client hosted with another popular host and they have a WordPress site. They wanted to upload a bunch of pictures taken from their camera into their photo gallery. But because those pictures were large they were getting upload errors. This other host said it was a limitation on the server setting to prevent hogging resources and the only way to change it was to upgrade to a VPS or dedicated box. I can understand imposing limitations but it was an inconvenience since they now have to manually shrink the file sizes before uploading them. And WP sites are getting more and more popular. In contrast, I had the same problem with another site but I was hosting it on my reseller account. HostDime just tweaked the setting for that specific account which made things so much easier for me. It's the little things that make HostDime good. I've been monitoring their uptime since January 2009 and it's been acceptable. But when they have a problem it's usually a big problem. The outage on 1/13 caused all their sites to go down for over 3 hours straight. My clients don't care that Monday was the best time to do the maintenance. The bottom line was that their sites were down right smack in the middle of the day and it caused a lot of frustration and tap dancing with my clients. Is it enough to make me move hosts? Probably not. There aren't a lot of reseller hosts out there and I'm not quite sure if it's worth the hassle of moving. This wasn't just a problem with their service. They're trying to make their service better which should be best for everyone. If anyone is interested here is the link from Pingdom showing HostDime's stats over the past couple of years. I'm monitoring HostGator, GoDaddy, Lunar and 1&1 should anyone be interested in those as well. http://www.pingdom.com/reports/4diqy...?name=HostDime
Posted by Cyandin, 01-19-2011, 02:21 PM
Downtime is statistically inevitable, but honestly, customers should not see it more than once every several YEARS, if the right redundancy measures are put in place.
Posted by linuxissues, 01-19-2011, 02:24 PM
That just tells me that the other host cares more about all their customers, not to please one customer with a stupid setting that can cause a whole lot of issues on the server for everyone. If you wanted custom settings to upload large files like that, yes you need a vps or a dedicated, atleast that's what I think. And this is a whole diversion to the topic. Maybe you should make a separate new thread about your review of hostdime..
Posted by th1chsn, 01-19-2011, 03:31 PM
linuxbaby, this whole thread is a diversion of topic. But funny how you complain about it yet still manage to put in your 2 cents.