ultrahosting.com [scheduled] 16 hour downtime!!! unbelievable!!

Portal Home > Knowledgebase > Articles Database > ultrahosting.com [scheduled] 16 hour downtime!!! unbelievable!!

Posted by professionalhosting, 10-16-2010, 08:03 AM
we have a backup office mail server hosted at ultrahosting.com / momentum.com and they are stating that they are having a 16 hour downtime,!! --- we received this notice from ultrahosting.com support techs --- Hello, Your server is currently unreachable do to the fact that we're in the middle of our data center move. The migration was announced to all of our registered clients' e-mail for the first time two months ago, and has since then been sent twice (once as a reminder, and a 3rd e-mail went out stating the migration was pushed back to the next week). The outage started at 2am EST and is expected to last until 6PM EST for a total of 16hrs. The migration is a physical move to a new, bigger data center so we can keep growing. This is a regrettable outage and I sincerely apologize in advance for any inconveniences this may cause. Please note, we will be updating our recorded greetings to provide updates throughout the day as to the status of our migration. Please feel free to call us at our toll free to hear the updated status. Thank you. Best regards, Technical Support - Ultra Hosting, Inc. http://support.ultrahosting.com/ | support@ultrahosting.com | (866) 988-5872 --- im sure glad we dont have any other servers at ultrahosting!! 16 hours of downtime is unacceptable for business, amazing!! 16 hours down,
Posted by Amitz, 10-16-2010, 08:27 AM
Where is your problem? A physical data center move is not done in 1-2 hours and you were notified quite some time in advance. You could have taken measures if 16 hours is too long for you. But you have chosen to do nothing. And now you come here whining...
Posted by cycomhk, 10-16-2010, 08:31 AM
The title is misleading. This was scheduled due to a physical server move. It is not like they had a downtime and were unable to get the server up after 16h.
Posted by professionalhosting, 10-16-2010, 08:34 AM
actually we were never notified, checked all emails, and support tickets, --- to be honest, I would expect a company to expand to a new datacenter, not switch datacenters entirely, --- anyways, maybe theres more to the story,,
Posted by whocky, 10-16-2010, 09:25 AM
Hi Professionalhosting Did you not just contridict yourself in your post? expand to a new datacenter? isnt that the same as switching? i would of thought so
Posted by RandomThoughts, 10-16-2010, 09:32 AM
In my view, expanding to a new datacenter is migrating services to new hardware already up & running in the new datacenter. From the sounds of it, what the OP's provider are doing is shutting down everything at the old datacenter & physically moving it to the new one.
Posted by NelsonT, 10-16-2010, 10:24 AM
The vps hosted data center and network is a key factor I consider when selecting the hosting source. If my vps moved to another data center, I will seriously consider if stay with the company, especially no notification in advance. Don't move server to elsewhere just for the cost/profit issues.
Posted by NelsonT, 10-16-2010, 10:28 AM
Okay, ultra hosting actually announce this move 15 days in advance and give you a chance to discuss with them, see http://www.ultrahosting.com/blog/ind...uled-downtime/
Posted by wlsiew, 10-16-2010, 10:41 AM
Can't read anything - show IIS error below: - HTTP Error 403.6 - Forbidden: IP address of the client has been rejected. Internet Information Services (IIS)
Posted by BCVH, 10-16-2010, 11:40 AM
They sent out an email, I got it and planned accordingly. I thought they did a great job in implementing the move.
Posted by NelsonT, 10-16-2010, 11:51 AM
I just tried again, loaded fine for me. Don't know why you could not reach that url. May you try any US web proxy again?
Posted by Ultra-Hosting, 10-16-2010, 11:51 AM
We had sent out multiple emails to all of our customers well in advanced We are moving today in an effort to accommodate new customers as our current datacenter would be full in a couple months This is a small inconvenience to better our service in the future for all current and future customers.
Posted by Ultra-Hosting, 10-16-2010, 11:54 AM
You can keep track on updates by following us on our blog http://www.ultrahosting.com/blog/
Posted by rj2media, 10-16-2010, 11:55 AM
The best thing is that I never received email about this transfer, I have website with couple thousand accounts.... what should I do, It is just destroying my business. Last edited by rj2media; 10-16-2010 at 11:55 AM. Reason: Spelling
Posted by Ultra-Hosting, 10-16-2010, 11:56 AM
There hasn't been 16 hours of downtime we had initiated the move at 2am today The timeline is a worst case scenario Last edited by Ultra-Hosting; 10-16-2010 at 12:01 PM.
Posted by Ultra-Hosting, 10-16-2010, 12:06 PM
Well on a smaller scale this would be acceptable but when you have a 40-50 racks to move it is more efficient to do it this way when we all know a DNS change can sometimes take up to 48 hours
Posted by BCVH, 10-16-2010, 12:09 PM
Thanks for posting Julian.
Posted by professionalhosting, 10-16-2010, 01:52 PM
in my opinion, the recommended method of datacenter expansion would be to "legacy off" the existing datacenter e.g. stop putting new customers in the "full" datacenter and simply put all new customers in the new datacenter e.g. expanding within the new datacenter, the above method would eliminate the need for 16 hours of downtime potentially ruining peoples business and uptime guarantees. I would be extremely cautious doing business with any company that would physically move racks and servers from one facility to another because it could be a sign of a larger issue such as lease termination, financial issues, etc. most professional operations would never physically move servers and racks simply to "move to a new, bigger data center so we can keep growing." maybe there is more to this story,, Last edited by professionalhosting; 10-16-2010 at 02:06 PM.
Posted by Ultra-Hosting, 10-16-2010, 02:06 PM
There is no need to worry about lease terminations etc because we own both data centers, we do not rent anything unlike most companies on this forum. Fact of the matter is we are doing this because of growth not because of financial issues etc.
Posted by Psyber Chris, 10-16-2010, 02:10 PM
This is a lot of fuss over a backup mail server....
Posted by Ultra-Hosting, 10-16-2010, 02:37 PM
We apologize to everyone for the inconvenience we are working diligently to have everyone's service restored
Posted by BCVH, 10-16-2010, 02:49 PM
Your insinuations really make you come off as a professional..
Posted by Amitz, 10-16-2010, 02:51 PM
Yep. One has to be brain dead and blind not to feel what the OP is up to.
Posted by kinitex, 10-16-2010, 03:44 PM
I got two e-mails about the move awhile ago, dont see what the big deal here is. Growing companies are forced into situations like this all the time.
Posted by professionalhosting, 10-16-2010, 06:16 PM
currently the outage is still persisting, (~6:15pm est localtime) server is unreachable, last update on your blog states "All equipment has been moved and connected, however our electrical work is taking significantly longer than projected. Our electricians are completing the final connections and we will start the power up immediately after. We anticipate electrical work to be completed ASAP. We apologize for this unexpected delay." --- is there an updated eta available for restoration of services? (my support ticket seems to be locked and cannot be updated ) please advise updated eta Last edited by professionalhosting; 10-16-2010 at 06:22 PM.
Posted by Amitz, 10-16-2010, 06:27 PM
What makes this "backup office mail server" soooo important? You really seem to exaggerate the whole thing a lot...
Posted by borsec, 10-16-2010, 06:53 PM
Our production server hosted by ultrahosting is down for over 16 hours now and still is not up and running - server that hosts the corporate websites and the applications of our clients. That makes us look awfully unprofessional. And we are one of those ultrahosting clients that didn't receive the notifications that seemed to be sent to others.
Posted by rj2media, 10-16-2010, 07:14 PM
Well done Ultrahosting.. 17 hours. You should move probably to smaller data center after this. Unbelievable, every couple months something happens. Year ago HDD failed on my server, all data was lost and took 24 hours to restore, about 2-3 months ago, electricity shortage, took couple hours to get back. Now without notification over 17 hours shortage. Good job
Posted by professionalhosting, 10-16-2010, 07:18 PM
currently the outage is still persisting, (~7:15pm est localtime) server is unreachable, no more updates on your blog since the last update as quoted within previous post, my trouble ticket is in locked state (cannot post updates) and no one is answering your 24 hour technical support line, this is unacceptable,, can someone from ultrahosting.com please update this thread with the latest updates and another potential eta for resolution,
Posted by Ultra-Hosting, 10-16-2010, 07:19 PM
Issues have since been resolved we are bring service back up to clients this process should be completed within 2-3 hours as we work out the final details Your service might even be available now
Posted by BlackThunder, 10-16-2010, 07:24 PM
Guys, Seriously, it's not like there weren't letting us know that this would happen. It's not like they move their DC's every day or every year. The whole reason for this move is to improve their record which I know has not been the best this year, but in the years prior has been absolutely flawless. As long as they go back to their previous record of constant up time I really don't care if it takes 2 or 5 more hours than predicted. No one can predict a move of who knows how many servers. Though, I have to be honest, I completely forgot about this move (even though I was discussing it earlier yesterday) and left my servers running which unfortunately probably led UH to kill the power to them. Tom
Posted by CanSpace, 10-16-2010, 08:31 PM
We have a server with Ultrahosting and were never informed of the move. I've gone through all our emails for the past few months and there was nothing communicated to us until today. Needless to say, we'll definitely be looking for a new provider once this is resolved (and sadly I'm sure some of our own clients will be too ). An un-communicated 18 hour downtime is beyond completely unacceptable. Our uptime over the past year or so hasn't been particularly spectacular either. Last edited by CanSpace; 10-16-2010 at 08:36 PM.
Posted by borsec, 10-16-2010, 08:43 PM
Don't know about the others, but ours are still down... almost 19 hours at this time
Posted by professionalhosting, 10-16-2010, 08:46 PM
our server is still inaccessible (~8:43pm est localtime), our ticket has not been updated , still no answer via 24 hour phone support, last status update on ultrahosting.com's blog so far is ~7pm, --- yet again, I think this is completely unacceptable level of service, is anyones server back online yet?
Posted by CanSpace, 10-16-2010, 08:50 PM
Completely unacceptable is an understatement.
Posted by BlackThunder, 10-16-2010, 08:58 PM
My servers came back about an hour ago. Up and running 100% Tom
Posted by Chasem, 10-16-2010, 09:00 PM
Absolutely unbelievable... Terrible service. Server went down at 1:05AM EST (I did not receive any emails of the service going down) I talked to Jullian (The only person that works at Ultra Hosting...?) and apparently the server IP address is changing as well. How the hell is this possible? When you sell a static IP address that usually means it doesn't change. How can you seriously run a business like this? I purchased a windows VPS for a client of mine, as I only provide Linux servers UGH What a gigantic mistake that was. - Signed an extremely disappointed UltraHosting customer. Note: As of 9PM the server is still completely down.
Posted by borsec, 10-16-2010, 09:04 PM
Yep, same here - server still down. Hopefully we will retain the IPs that we have because that will be a nightmare to change them at this time.
Posted by professionalhosting, 10-16-2010, 09:17 PM
current localtime is ~9:10pm est , our server is still inaccessible, no update to our trouble ticket, still no answer via ultrahosting.com 24 hour tech support phone, --- [Querying whois.arin.net] [whois.arin.net] # # Query terms are ambiguous. The query is assumed to be: # "n 74.213.164.18" # # Use "?" to get help. # # # The following results may also be obtained via: # http://whois.arin.net/rest/nets;q=74...showARIN=false # Momentum Solutions Inc THTI-MOMENTUM (NET-74-213-164-0-1) 74.213.164.0 - 74.213.165.255 Cogeco Data Services Inc. CDSI (NET-74-213-160-0-1) 74.213.160.0 - 74.213.191.255 # # ARIN WHOIS data and services are subject to the Terms of Use # available at: https://www.arin.net/whois_tou.html --- ping and tcptraceroute tests still fail during testing --- ping 74.213.164.18 PING 74.213.164.18 (74.213.164.18) 56(84) bytes of data. --- 74.213.164.18 ping statistics --- 22 packets transmitted, 0 received, 100% packet loss, time 21167ms tcptraceroute 74.213.164.18 Selected device eth0, address 192.168.3.220, port 39322 for outgoing packets Tracing the path to 74.213.164.18 on TCP port 80 (www), 30 hops max 1 192.168.2.20 0.224 ms 0.107 ms 0.164 ms 2 10.240.32.1 7.011 ms 8.256 ms 5.872 ms 3 69.63.254.93 6.194 ms 11.465 ms 7.499 ms 4 69.63.249.177 13.517 ms 9.139 ms 9.174 ms 5 69.63.250.77 17.360 ms 8.575 ms 8.706 ms 6 64.71.240.230 8.328 ms 8.621 ms 6.873 ms 7 ge-4-3-0.gw02.mtnk.phub.net.cable.rogers.com (66.185.83.1) 8.525 ms 8.207 ms 7.568 ms 8 64.71.240.174 8.565 ms 9.562 ms 9.770 ms 9 * * * 10 69-10-224-74.onx.com (69.10.224.74) 9.775 ms 14.246 ms 10.414 ms 11 69-10-224-220.onx.com (69.10.224.220) 12.550 ms 12.290 ms 11.381 ms 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Destination not reached --- also onx.com website is still inaccessible during testing, ping onx.com PING onx.com (66.240.161.125) 56(84) bytes of data. --- onx.com ping statistics --- 11 packets transmitted, 0 received, 100% packet loss, time 10081ms tcptraceroute 66.240.161.125 Selected device eth0, address 192.168.3.220, port 39200 for outgoing packets Tracing the path to 66.240.161.125 on TCP port 80 (www), 30 hops max 1 192.168.2.20 0.187 ms 0.124 ms 0.192 ms 2 10.240.32.1 6.890 ms 6.312 ms 6.198 ms 3 69.63.254.93 7.420 ms 8.004 ms 6.320 ms 4 69.63.249.177 8.192 ms 8.804 ms 8.804 ms 5 69.63.250.77 9.134 ms 9.654 ms 8.887 ms 6 64.71.240.230 8.073 ms 8.311 ms 7.905 ms 7 ge-5-3-2.gw02.mtnk.phub.net.cable.rogers.com (66.185.83.21) 8.598 ms 30.530 ms 8.921 ms 8 64.71.240.174 8.218 ms 9.805 ms 9.189 ms 9 * * * 10 69-10-224-65.onx.com (69.10.224.65) 9.574 ms 10.144 ms 10.818 ms 11 69-10-224-120.onx.com (69.10.224.120) 10.116 ms 10.133 ms 11.065 ms 12 * * 69-10-224-36.onx.com (69.10.224.36) 10.802 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Destination not reached --- inaccessible via siteuptime.com remote tests aswell, Quick check for: onx.com (http) Date City Country Time (s) Result October 16, 2010 18:04:58 San Francisco United States Failed October 16, 2010 18:05:22 New York United States Failed October 16, 2010 18:05:46 London United Kingdom Failed ---- I would agree that changing ip addresses would be a complete nightmare and is unprofessional in my opinion, I really hope we do not have to change ip's ,
Posted by thomas_m, 10-16-2010, 09:28 PM
I'm at 19:10 of downtime on both of my servers with UltraHosting. My clients and boss are livid. I never received any notification about this (I've searched through my email). The first I heard of it was in an apparent stock reply to a trouble ticket I put in early this morning. The lack of understanding that UltraHosting expressed in that response and in this thread and the lack of ongoing communications only adds insult to injury. Other than invoices, the last communication I got from UltraHosting was what I thought was a sincere apology from their president for downtime, and a promise to do better. Wonder what he'll write this time?
Posted by CanSpace, 10-16-2010, 09:28 PM
heading towards 20 hours now...
Posted by professionalhosting, 10-16-2010, 09:56 PM
our server is now accessible, both ping and ssh responds normally, this has been a real unpleasant experience , --- ping 74.213.164.18 PING 74.213.164.18 (74.213.164.18) 56(84) bytes of data. 64 bytes from 74.213.164.18: icmp_seq=1 ttl=55 time=10.6 ms 64 bytes from 74.213.164.18: icmp_seq=2 ttl=55 time=11.6 ms 64 bytes from 74.213.164.18: icmp_seq=3 ttl=55 time=9.47 ms 64 bytes from 74.213.164.18: icmp_seq=4 ttl=55 time=12.6 ms 64 bytes from 74.213.164.18: icmp_seq=5 ttl=55 time=11.0 ms 64 bytes from 74.213.164.18: icmp_seq=6 ttl=55 time=10.8 ms 64 bytes from 74.213.164.18: icmp_seq=7 ttl=55 time=13.5 ms
Posted by borsec, 10-16-2010, 10:03 PM
wow, this is very sad, we are still down after all this time and now it seems nobody in support is answering...
Posted by thomas_m, 10-16-2010, 10:12 PM
My servers are back up. Total downtime was about 19 hours 50 minutes. Hope you go up soon too.
Posted by CanSpace, 10-16-2010, 10:25 PM
ours is back up now too
Posted by borsec, 10-16-2010, 10:28 PM
We are still down. It's been 20 hours and 30 minutes
Posted by Chasem, 10-16-2010, 10:31 PM
Still down. 21 hours of downtime... Holy hell
Posted by borsec, 10-16-2010, 10:50 PM
still down, at 21 hours and still cannot reach anybody in support
Posted by Chasem, 10-16-2010, 10:56 PM
Jeee this is so professional... I'm definately asking for a complete refund. I paid 6 months in advance too.
Posted by BCVH, 10-17-2010, 12:36 AM
Our servers are back up
Posted by ebertech, 10-18-2010, 09:30 AM
21 hours? One of my servers was down for 36 hours. Had to call them the next day to get someone to follow up as to why it still wasn't online. Frustrating is an understatement.
Posted by Amitz, 10-18-2010, 10:50 AM
Okay. Now I understand your anger. I take everything back what I wrote before.