DEHE and Solidhost Down [merged]

Portal Home > Knowledgebase > Articles Database > DEHE and Solidhost Down [merged]

Posted by guillon, 03-18-2006, 07:33 PM
For 4th time in 4 months my VPS is offline because of provider failures. I have suffered all the problems told by SDH1 in "DEHE Disaster" thread (thread 494727). Those problems were aparently solved last sunday, but yesterday (Friday 17 March) my server started to fail because the filesystem was mounted "read-only". After 3 hours i get an incredible response: "The problem is that Virtuozzo mounted your VPS with read-only mode for some reason and we are not able to change it back to read/write mode. We and SWsoft techs are not sure why it did that" (It seems that Virtuozzo its alive and doesn't likes me) They offered moving the VPS to another node, which taked another 6 hours. After that, many files have aparently dissapeared from disk, but still referenced from their directories (filesystem corruption). After spending hours with excuses like 'It seems that perl module is misconfigured' or 'It looks like your server under attack' (which wasn't true) they accepted that there is a problem, but still haven't resolved it. In the meantime, my server has been offline almost 2 day and counting. This isn't my first experience with VPSs, previously I went on bad experiencies with Tektonic and RapidVPS. I'm starting to think that VPS tecnology isn't stable enough for production sites. Also, I want to attach the log of the ticket I started on this issue, to show how bad was the support DEHE offered to this issue. Guillermo Narvaja My sites: fierro-soft . com . ar and libreriahernandez . com . ar Attached Files DEHETicket.txt (9.2 KB, 65 views)
Posted by Energizer Bunny, 03-18-2006, 08:03 PM
Ah i like it when people provide full proof. And definately seems like dehe is not able to correct the problem in timely manner it could be another file corruption like they had few days back. Hoping Dehe improves and tries to prevent such corruptions.
Posted by 01globalnet, 03-18-2006, 08:35 PM
Really bad luck ... some problems do happen and require more time to resolve .. at least DEHE stuff is there and working on your issue. I also start to think VPS's are not appropriate for reselling hosting, but only to host 1-2 big sites....
Posted by BeDifferentSolutions, 03-18-2006, 09:29 PM
This is false. The technology is perfect for resellers. But the issues are three fold: 1) Many buyers look for the cheapest plans. Cheapest provider have to proved cheap equipment and support and in turn cram VPSs on low quality boxes. This leads to more chances of failure and more importantly, repeat failure. 2) Many buyers try to do with a VPS that they did on a full dedicated box or what they should do on a full dedicated box. A VPS is not a dedicated server. 3) VPSs require more technical management but require only a few dollars to resell. Host could get away with this on shared servers and start web company with no expertise. Not on VPSs. So what we have are new VPS host with no idea how to properly run a VPS company. This can be proven here by the retarded comments some VPS Host owners say here. It is obvious they were fooled with fancy SW Soft sales talk rather then proven with experience.
Posted by EuroVPS, 03-18-2006, 09:48 PM
EXACTLY what he said..
Posted by PersianImmortal, 03-19-2006, 01:05 AM
Unfortunately I am on one of the nodes on which Dehe has moved across such sites, and now my node has become overloaded and is causing a lack of responsiveness on my site. While I am quite positive about the way Dehe manages their business - they provide great support and usually are very responsive to issues - unfortunately if they continue to prove unreliable then I cannot host my site TweakGuides.com with them. It does indeed seem like VPS technology is still not firmed up. I had assumed a VPS would shield me from CPU load problems, since my site by itself is not a CPU hog (mainly static HTML pages with a small optimized VB forum). Does anyone know of a truly reliable VPS?
Posted by PersianImmortal, 03-19-2006, 01:33 AM
Yes I'm considering PowerVPS at the moment, thanks (hmmm...the person posting the recommendation has removed their post now). It's a real shame, because Dehe is not a bad host as such, they do offer great support and the guys who run it (Andre and Milovan) take the time to give a lot of feedback and detailed information to people like myself about what's going on. However at the moment there are just too many technical problems, and presently my site load fluctuates so much (nothing to do with my own traffic), that it is becoming embarrassing for me to explain to my readers why exactly it is that I am constantly having outages, lack of responsiveness and whatnot, even when I assure them that "it will be over soon". Some indication from Dehe of exactly when these issues will come to an end would be a good start. Their support staff just give the usual "we are monitoring the situation" responses, which are effectively the same as "we hear you, but we can't say anything to you". Last edited by PersianImmortal; 03-19-2006 at 01:37 AM.
Posted by lottos, 03-19-2006, 05:22 AM
personally I would never host on any host that agrees with using the word retarded as a great descriptor.
Posted by ForumBV, 03-19-2006, 07:40 AM
I'm running hosting of web/mail and dns on a XEN virtualised server, it running about 500 mailboxes, 100 websites and 150 domains without any problems. The fact that it is virtualised means i have a lot of extra freedom that a normal system doen not give me (like remote XS to a console normaly only available with phisical keyboard/monitor) I think that most problems with 'VPS' relates to the way the host uses its machines (overbooking memory and hardisk, cpu etc) and in the software used (xen obivously has much better performance) I whould not rule out VPS's that fast. Virtualisation is the future, even if you only run one server on 1 machine the OS should be virtualised imho.
Posted by Apoc, 03-19-2006, 08:45 AM
Hi Everyone, I'm not denying any of the above complaints, I'd just like to shed some light on what happened. The first failure in the "DEHE disaster" thread indeed was a filesystem corruption and we had some serious problem recovering the filesystem initially - a full explanation about this can be found in that other thread. At the point when we recovered the filesystem we had to migrate all VPS's to new nodes. We spread out the VPS's on this node over 3 new nodes that were standing by. Now on Friday one of those nodes had a new problem which guillon is referring to. Let me just explain what happened: On friday late in the afternoon the /vz partition of the node was suddenly mounted read-only. We immediately consulted with SWsoft and it turned out that Virtuozzo had mounted the filesystem as read-only to prevent file corruptions, because it detected problems with the 3ware driver. This 3ware driver is copyrighted by swsoft, and it turned out there was a bug in this driver due to which it tried to write to a location in the raid array which doesn't exist. SWsoft explained to us that the only thing we could do was to move everyone over to a new node and in the meanwhile most VPS's would stay online although nobody would be able to write any new data to his vps, which is a problem for databases and such. We did what they suggested and moved everyone over to a new node as quickly as possible. No data was lost in the process and everyone was moved as quick as we possibly could have. It's very unfortunate that these events occured, and especially that they occured so shortly after eachother. I am not denying that we are responsible for this, however both of the issues do not relate to the stability of our services, both of these problems could have occured to any other host as well. The first problem was that the raid controller had failed. We use 3ware 9500 series which are known to be one of the most reliable controllers in the world. The second problem was a bug in the 3ware driver. This is an error on sw-soft's end. Even though we did have these problems I have to say that we did handle these issues as good as we possibly could have. If anyone has any questions about this feel free to ask.
Posted by PersianImmortal, 03-19-2006, 08:59 AM
What I am really after is a timeframe as to when this will all be resolved. As I say above, I am appreciative of the detailed feedback, but I am rapidly looking like an idiot to my readers because I cannot give any indication of when this will all be resolved. Right now I've suddenly found my server is no longer sending emails for example. I have lodged a support ticket, but as you can imagine that now means people registering for my forums will not receive their confirmation emails. So on top of site load fluctuations, I suddenly find another technical issue has cropped up. It's 12:00am here in Australia, and I wonder when I wake up tomorrow what new issue may arise. I think you see what I mean when I suggest that it would just be nice to know when you believe we will return to normal. This node has taken a hammering.
Posted by Apoc, 03-19-2006, 09:07 AM
The load on the node you are referring to is very low. If you can tell me the ticket ID in question I'll take a look at the status on the ticket for you.
Posted by PersianImmortal, 03-19-2006, 09:44 AM
The ticket was SOE-87073, and it is now resolved. Still no idea why it occurred, something on the server end which the technician simply said was fixed, and now it is. Nothing on my end has changed. Not exactly confidence inspiring, because if I had happened not to try sending an email out from my email client, I may not have noticed for a while that my server was not sending out emails. As for the load issues, the load fluctuates significantly, whereas for the last month or two it has been consistently less than 1.00 on average and very responsive. Over the last couple of days because of the new sites on the node, it was up to 12.00, then down to below 1.00, and anywhere in between throughout this afternoon. At any point in time looking at the load snapshots from 1, 5 and 15 minutes I'd see figures consistently above 3.00, with the latest figure often at 6.00 or higher. Right now it is 0.23 which is fine. But as I say it fluctuates a lot more than before and results in visible reduction in responsiveness on my site. I am not expected champagne for the price I pay, I am simply after the same sort of consistency I briefly experienced before we had these problems. Clearly the issue is due to the transfer of sites due to this current issue, I am only after an indication of when it will all be completed.
Posted by Apoc, 03-19-2006, 09:54 AM
The migration of sites/vps's has already been completed. I am monitoring the load on the node along with one of our techs and we will continue to do so until we are sure the load is stable. If nothing on your end has changed it's indeed likely this is something on our end and we'll make sure this gets stabilized as soon as possible. Again our apologies for this inconvenience.
Posted by guillon, 03-19-2006, 10:27 AM
And for mail servers, dynamic web pages, and every application that needs to write one byte to disk. Maybe you have to review your definition of "online". This wasn't what hapened with my VPS (vps255). As you can see on ticket CRU-45785, many plesk files are "missing" and you and your people haven't recovered them until now. There is a great communication failure on your team if you didn't noticed this, as the ticket has 40 hours open and 28 replies. We are paying to DEHE, not to SWSoft, so you must resolve your problems with your provider (oh, its the same my clients said to me), I doesn't care if the problem its on sw-soft's end. Also, you have to test the products before delivering to us, specially with something so new as Virtuozzo 3. I don't think you handled my issue good. As you can see on the ticket, you take up to 12 hours to give a response, and in the meantime nothing was do with my server.
Posted by Apoc, 03-19-2006, 10:53 AM
That is what I said. I have already contacted you about this. [QUOTE=guillon] We are paying to DEHE, not to SWSoft, so you must resolve your problems with your provider (oh, its the same my clients said to me), I doesn't care if the problem its on sw-soft's end. Also, you have to test the products before delivering to us, specially with something so new as Virtuozzo 3. We have tested Virtuozzo 3 very thoroughly and there never have been any problems with it until now. As I already said I know towards you as a customer this is our responsibility, and I wasn't using the fact that it's a bug in SWsoft's software as an excuse, I was just explaining the situation. Within that timeframe we have been working constantly on the problem. As I said I have already contacted you regarding this and it wouldn't really make a difference to discuss that here as well.
Posted by PersianImmortal, 03-19-2006, 08:22 PM
In my case it appears that the site load has normalized to stay consistently below 1.00 again for a while now and the site is responsive and trouble free for the moment. Thank you Andre, hopefully things should go smoothly for the most part from now.
Posted by KnownHost, 03-19-2006, 11:36 PM
[QUOTE=Apoc This 3ware driver is copyrighted by swsoft, and it turned out there was a bug in this driver due to which it tried to write to a location in the raid array which doesn't exist] SWsoft does not copyright 3ware drivers. This is not true. They are the vendor of Virtuozzo but have nothing to do with such an issue. 3ware is 3ware... -Ja Last edited by KnownHost; 03-19-2006 at 11:40 PM.
Posted by PersianImmortal, 03-20-2006, 08:09 AM
And now the entire site is down - not just my site, but all of Dehe.com is unreachable! This is honestly becoming ridiculous.
Posted by SDH1, 03-20-2006, 08:13 AM
Is it me or is DEHE down again? Our site is down, other sites from DEHE appear to be down and it even looks like DEHE and Solidhost are down. I've tried tracert and it's timing out The rest of our connection to the Internet seems fine. Anyone else having the same problems?
Posted by PersianImmortal, 03-20-2006, 08:14 AM
I just posted about this on the other Dehe thread, but yes they are definitely down. I really am amazed at the hosting industry, it's almost impossible to pick a bloody stable host regardless of the price.
Posted by SDH1, 03-20-2006, 08:15 AM
Actually, your site was one of the first that I checked to see if other DEHE sites were down. Looks like it's another power outage at the datacentre then
Posted by Apoc, 03-20-2006, 08:15 AM
GNAX (our network provide) is having a network outage. Over 50% of their network is affected by this. The error is on a part of their network which we don't control, most likely one of their routers died. They indicated everything should be back within approximately 15 minutes.
Posted by PersianImmortal, 03-20-2006, 08:18 AM
Ok, thanks for the info.
Posted by SDH1, 03-20-2006, 08:22 AM
Great, so everything else will still be up and running, no VPSs down, no corrupt tables etc?
Posted by Apoc, 03-20-2006, 08:22 AM
Please refer to the other thread for more information about this.
Posted by PersianImmortal, 03-20-2006, 08:25 AM
Not much we can do but wait and see. I am now taking pretty much hourly backups of my site and forums.
Posted by PersianImmortal, 03-20-2006, 08:33 AM
For the reference of others, it is this thread (since it has been moved).
Posted by Milovan, 03-20-2006, 08:34 AM
Yes, everything will be up and running - servers are up and running - this is a network problem.
Posted by Apoc, 03-20-2006, 08:52 AM
Looks like we're back up. We will follow up with more information as soon as we receive more information from GNAX.
Posted by Apoc, 03-20-2006, 08:56 AM
For further information about the outage please refer to http://www.dehe.com/showthread.php?t=583. As soon as we have more information available about what happened the information will be posted there.