Portal Home > Knowledgebase > Articles Database > SSHD Rootkit Rolling around

SSHD Rootkit Rolling around

Posted by Steven, 02-08-2013, 12:59 AM
Quick survey, anyone seen a rootkit being used to send spam through sshd involving a library called 'libkeyutils.so.1.9'? If so what OS did you see it on?

Posted by brianoz, 02-08-2013, 08:51 AM
The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change. CentOS release 6.3 (Final) md5sum /lib64/libkeyutils.so.1.9 c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9 -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9* rpm -qf /lib64/libkeyutils.so.1.9 file /lib64/libkeyutils.so.1.9 is not owned by any package uname -r: 2.6.32-279.14.1.el6.x86_64.debug

Posted by TravisT-[SSS], 02-08-2013, 12:35 PM
I too can confirm this. Currently working with clients with spam issues and it is present. I checked other boxes we run and own and the library is no where to be found. It is only found on spam infested machines. uname -a 2.6.32-042stab059.7 md5sum /lib64/libkeyutils.so.1.9 d81217186da61125f4dad7a87857b697 /lib64/libkeyutils.so.1.9 rpm -qf /lib64/libkeyutils.so.1.9 file /lib64/libkeyutils.so.1.9 is not owned by any package

Posted by Steven, 02-08-2013, 01:19 PM
They are not logging in with root, nor are they even spawning a bash process. If the lib is moved out, and sshd is restarted they cannot login anymore fwiw. The key is finding out how they are getting in. Fully upgraded, ssh key restricted sshd, on non-standard ports are being compromised. None of my customers are, but I have been getting alot of sales inquiries with this issue so I don't know the full history of the machines. Seeing it on centos 5, centos 6, cloudlinux 5, cloudlinux 6. Last edited by Steven; 02-08-2013 at 01:27 PM.

Posted by JonnyQuags, 02-08-2013, 03:43 PM
I haven't seen this yet, but will keep my eye out. 64 bit only systems are what you are seeing? Are tcpwrappers or firewall offing ssh making a difference?

Posted by brianoz, 02-08-2013, 04:41 PM
Firewalling off ssh stopped them on the machine I was looking at. And the machine was 64 bit. FWIW I suspect they are getting in initially some other way than ssh, but have no evidence.

Posted by Steven, 02-08-2013, 05:28 PM
John, Iptables will stop them. Tcpwrappers does not. Brianoz, Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.

Posted by brianoz, 02-09-2013, 08:43 AM
Steven - sorry for being unclear. I didn't mean to imply the initial attack was from ssh; what I meant was that an iptables block on ssh stopped them reconnecting, exactly as you've seen. Is the following consistent with what you've seen? 1. User account compromised at PHP level 2. Compromised account used to hack root and backdoor sshd via libkeyutils 3. Spam sent The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.

Posted by unSpawn, 02-09-2013, 09:13 AM
A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/". Would anyone of you be able to dump a copy of the file(s) at sourceforge +.net/tracker/ +?group_id=155034&atid=794187 or send it to me for analysis? Much appreciated, TIA. Last edited by unSpawn; 02-09-2013 at 09:28 AM. Reason: //More *is* more

Posted by unSpawn, 02-09-2013, 09:24 AM
If you don't mind me asking: - What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess? - Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account? Did the user dump files in the system? Does a quick LMD scan reveal any PHP shells or other unwanted items? - If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?

Posted by brianoz, 02-09-2013, 05:48 PM
Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ... In other words, a standard shared server compromise where old WordPress/Joomla/etc installs or plugins are used to break into an account and run as that user. Haven't found an account, but a guess is no, as most user accounts had no shell. Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.

Posted by unSpawn, 02-09-2013, 07:06 PM
Sorry, I misinterpreted what I saw. It should indeed be just /lib64/ or /lib/ like others said. Sure, but still I'd rather have "evidence" or even a partial audit trail for analysis.

Posted by iseletsk, 02-11-2013, 08:25 AM
Anyone has details on the software / versions being installed on the server? Something like rpm -qa from the servers would be very nice start.

Posted by brianoz, 02-12-2013, 09:36 PM
Output of rpm-qa: http://pastebin.com/nTc8wj3U Output of rpm -Va (verify): http://pastebin.com/Fz0AxR3W The "5" means modification, which is often benign, but may help. The -Va was run on a fully infected system, some changes may have been made by the time the -qa output was obtained. See my post #2 above for matching O/S version etc: Last edited by brianoz; 02-12-2013 at 09:40 PM. Reason: add OS version

Posted by iseletsk, 02-12-2013, 11:15 PM
Which control panel do affect servers run? Anyone knows how they are getting infected yet?

Posted by TravisT-[SSS], 02-13-2013, 12:05 AM
All the ones we have seen so far are CPanel and they have been poorly secured to begin with.

Posted by brianoz, 02-13-2013, 02:15 AM
cPanel, and also poorly secured. We don't know how they are getting to root to install the backdoor yet.

Posted by Steven, 02-13-2013, 03:30 AM
Trying to tackle all angles, what imap/pop3 server are you seeing on the servers (dovecot vs courier)?

Posted by brianoz, 02-13-2013, 03:43 AM
FYI, courier on the only server I know that was exploited.

Posted by Steven, 02-13-2013, 11:55 AM
What about you SolidShellSecurity ? Could be off base here, but I have not seen a server with dovecot exploited.

Posted by Patrick, 02-13-2013, 12:05 PM
Been so long since a Courier IMAP exploit existed... What FTP daemon were all of those boxes running?

Posted by TravisT-[SSS], 02-13-2013, 02:41 PM
I don't exactly remember. I would check but the box would be wiped by now as we moved them off to a new server. But I vaguely don't remember them running dovecot.

Posted by JonnyQuags, 02-13-2013, 02:58 PM
Just with the fact only 64 bit servers (so far) are known to be exploited, it could be related to past exploits on 64 bit systems. Its been a while there, but I wouldn't discount previously hacked machines from that kind of exploit.

Posted by Majester, 02-13-2013, 04:02 PM
How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?

Posted by Patrick, 02-13-2013, 09:48 PM
When a server is deemed compromised, it's always a good idea to do a check of all non-user directories to look for recently changed binaries or anything new. My guess, that file showed up in the search.

Posted by Steven, 02-15-2013, 02:54 AM
Alot of debugging. Started with lsof of running ssh binaries, and verified the libraries involved were actually attached to a RPM package.

Posted by Steven, 02-15-2013, 02:57 AM
This is how I found it:

Posted by Zimple, 02-15-2013, 03:10 AM
Many thanks Steven. Do we have hunter for this root kit ?

Posted by unSpawn, 02-15-2013, 07:32 AM
Maybe LMD but not CRT and definitely not RKH until somebody drops the payload like I asked in post #9, TIA.

Posted by Zimple, 02-15-2013, 02:38 PM
It is complete crap. If you remove libkeyutils.so.1.9, it won't start neither nsd or ssh

Posted by iseletsk, 02-15-2013, 02:54 PM
mobilk: remove that file do: yum reinstall keyutils-libs of course the question on how hackers got in is still open. and btw: if you use passwords to login, change them.

Posted by Zimple, 02-15-2013, 03:09 PM
Thanks!. BTW compromised system was CloudLinux one. Just installed 1 month ago.

Posted by iseletsk, 02-15-2013, 03:16 PM
Yes, there are few CL ones, Steven told me. So far, from what we know it affects CL & CentOS systems running cPanel. No idea yet on how hackers get in.

Posted by Dathorn-Andrew, 02-15-2013, 04:21 PM
Were any of the compromised CL machines using CageFS as well? Obviously it may not matter depending on the point of entry but curious nonetheless.

Posted by TravisT-[SSS], 02-15-2013, 04:53 PM
We haven't seen any of our GRSecurity servers hit yet. Anyone else seen a GRSecurity server hit? It might help to pinpoint a kernel vuln.

Posted by Zimple, 02-15-2013, 05:31 PM
Yes, it was.

Posted by FastServ, 02-15-2013, 05:39 PM
Have you guys checked the WHM logs, perhaps correlating with the timestamp on the file? Were the kernels reasonably up to date on the affected systems?

Posted by TravisT-[SSS], 02-15-2013, 05:41 PM
On that subject, what WHM plugins are running? If every server has in someway been cpanel related maybe they found a hole in using one of those? CPanel does not exactly do a good job with securing this area as they all run as root. The servers we have seen have been running a lot of plugins.

Posted by FastServ, 02-15-2013, 05:54 PM
Which is along the line of my thinking. Two most common way in my experience to drop root files would be either thru WHM (don't we all love how WHM and all its 3rd party plugins runs as root), or privilege escalation on an ancient kernel. It's also a good idea to make sure WHM has been updating itself properly.

Posted by Steven, 02-15-2013, 06:26 PM
From a number of servers I investigated since I first saw this.. no whm login nor old kernel on most of them.

Posted by FastServ, 02-15-2013, 06:38 PM
Any root WHM plugins running on their own ports with separate logs, like WHMsonic?

Posted by Steven, 02-15-2013, 06:59 PM
Well technically all whm plugins are root plugins.. just the way whm works. Servers with minimal plugins have been compromised, the common denominator is CSF/configserver plugins but I don't think that is the issue. Servers with the latest centos/cloudlinux have been compromised. Both versions 5 and 6. I have scoured over CVE's for the linux kernel up to the latest 3.x version and I didn't see anything relevant that would cause it in the centos kernels. The earliest server I have seen exploited was Late december. Servers with ksplice have been exploited. I have not see any other control panel with this problem, nor have people mentioned it to me (talking to a few dozen people about this actually). I have removed the library from some servers, and it has not been back. I have yet to see it on our customers, so I don't have any pre hack info (emails about root logins,etc) , all I have had my hands on are post hack servers. Last edited by Steven; 02-15-2013 at 07:05 PM.

Posted by Steven, 02-15-2013, 07:01 PM
Hivelocity if you are reading this, you shouldn't claim you have a fix without having proof to back it up. Removing some files, tossing up some symlinks and upgrading the kernel is not a real fix. Servers with kernels fully updated either by package or ksplice have been compromised. Last edited by Steven; 02-15-2013 at 07:04 PM.

Posted by Steven, 02-15-2013, 07:07 PM
Another place to touch base, servers with both apache and litespeedtech were hit.

Posted by hivelocitygm, 02-15-2013, 09:10 PM
I dont know if we said we had a sure fix. What we have is something that at the moment seems to have worked. However, in step 5 above the command is backwards. It should be... ln -s libkeyutils.so.1.3 libkeyutils.so.1

Posted by TravisT-[SSS], 02-15-2013, 09:19 PM
Could we be looking at a direct CPanel exploit? The only problem like you have said is servers being managed are not being compromised. What about GRSecurity kernels. Have you seen any of those hit yet? I know it is not the most commonly used but so far here it has been in the clear.

Posted by Steven, 02-15-2013, 09:28 PM
Your kidding me right? All you have done is removed the exploit from an already exploited server. You have not done anything to prevent it, it could be back any moment, and no where in your statement do you even mention that.. so I stand by my statement. You said you had a fix, released it, and its not even a real fix. Good job on the false sense of security for your customers, the ones who are compromised are going to go on their day thinking nothing is wrong. The end of the day, root was gained, and we don't know how yet.

Posted by Steven, 02-15-2013, 09:30 PM
Could be a cpanel exploit, but I am not in the position to say it is. I have only seen stock cpanel servers and cloudlinux servers compromised, but thats not to say it has not happened on a grsec server.

Posted by serve-you, 02-15-2013, 09:43 PM
Any details on what versions of cpanel you are seeing this on? Could it be only a specific branch?

Posted by TravisT-[SSS], 02-15-2013, 09:44 PM
It has happened on the latest branch.

Posted by serve-you, 02-15-2013, 09:46 PM
Need more specifics than that. There are 4 different release trees.

Posted by TravisT-[SSS], 02-15-2013, 09:53 PM
Currently, if I remember right, the last 3 all run the same and the latest one run the latest patches so if it is vulnerable I am pretty sure it is safe to say the older ones are too. But all our clients run the latest branches.

Posted by serve-you, 02-15-2013, 09:59 PM
There are 3 different "current" major versions. 11.32, 11.34 & 11.36. 11.30 was just EOL'd last month.

Posted by egillette, 02-15-2013, 10:06 PM
Ok. . .so I'm a Hivelocity customer (probably one of their best customers if you ask Steve)! I probably control an entire class C by myself due to the number of clients I refer to Hivelocity and the sheer number of machines under my control (I have 12 of my own, and manage around 40 or so for clients). Since receiving the e-mail mentioned above by Steve, and then also following up (reading this thread, and actually checking around 26 of the servers I manage). . .I've come up with nothing conclusive that would link or correlate the mentioned file in any way with an exploit. I mean, as server admins, are we not scientists by nature?? One simple correlation is all it takes to freak us out. . .? My clients all forwarded me the e-mail from Hivelocity, so I ended up receiving the e-mail around 26 times, which was a bit annoying, since I'm not sure why clients think I'm not receiving the same e-mail from Hivelocity that they are receiving. But here's what I've discovered by checking multiple systems, mind you some of these systems are running CentOS 5.x not 6.x, some of them are also running Plesk, or DirectAdmin. That eliminates the possibility of this issue affecting cPanel specific systems, or CloudLinux machines only for example. I'm not saying anyone said that it is ONLY affecting these types of machines, I'm just saying specifically that the ice cream sales and violent crime rising at the same time do not share any common denominator. . .all they share is a single factor: season. In the summer, crime rises, as do ice cream sales, yet aside from that common denominator, they have nothing else in common. I suspect this is similar in that respect. I did have a client who was hacked recently, but it was the result of him having directories and files set to 777 which I had forewarned him about previously. We got rid of the hack (Fileman backdoor), and I ran maldet, rkhunter, and chkrootkit just to make sure there were no other files compromised on his system. But anyway. . .onto the points I've made, and my findings about this file (libkeyutils.so.1.9): Plesk 9.5.x system (CentOS 5.9 32-bit Hosted with Hivelocity) As you can see the date on that file is 1/6/2007 -- so if we're going based on the existence of this file somehow pointing to an exploit, this machine would have been hacked in 2007/2008. The problem is, this machine has only been online since 2011. Plesk 10.x.x system (CentOS 5.5 64-bit Hosted with MediaTemple): File Does Not Exist cPanel 11.34.x system (CentOS 6.3 64-bit Hosted with Hivelocity) The above system is my own server -- there are no connections outside of those I don't know about and haven't explicitly allowed in the firewall: Again, the above is my own server, and I'm running courier, not dovecot, and someone even alluded to the possibility of it affecting systems only running courier, so I think that eliminates that possibility as well reasonably. Now before I paste anymore results, I want to say for the record, that I've virtualized a server for one of my clients recently, and have setup a few containers, and installed cPanel from scratch with a CentOS 6.3-minimal ISO. The file mentioned above (libkeyutils.so.1.9) is not present on ANY of those systems -- yet this doesn't lead me to the conclusion that the systems below are *infected* simply because the file exists (whether RPM is reporting it as part of a package or not). Here are some more results from different servers: cPanel 11.34.x system (CentOS 6.3 64-bit Hosted with LeaseWeb) And the connections/ports: Again same file date as the system above. Now, of the 26 systems I've surveyed, this file is either non-existent, or if it does exist it is dated 6/22/2012 (64-bit systems), or 1/6/2007 (on 32-bit systems). Further, none of these systems that according to what I've read so far, may be "compromised" have any connections outside of those I don't know about, and some FTP connections. So I think at best the existence of this file on affected systems, is just a coincidence. This isn't to say that you guys aren't on to something here, but I think as server admins we should attempt to find more substantive evidence that points to an infection or how the infection is happening, rather than simply identifying arbitrary variables/files that may or may not have anything to do with anything. Let me know if I can be of assistance, as I have access to more than 70 machines (around 40 in Hivelocity's datacenter and around 30 in datacenters with SoftLayer, LiquidWeb, Linode, MediaTemple, and LeaseWeb in Europe). And Steve (Hivelocity Steve). . . I think you should hold off on sending out any e-mails like that going forward until we've got some substantive evidence of something, since my clients FREAK OUT when they get e-mails like that from you guys and start getting e-mails and phone calls about something that at best amounts to a coincidence. I mean you guys sent out a fix for something that likely wasn't broken/compromised in the first place -- just a mere coincidence. I know you're probably looking to be better safe than sorry, and I appreciate that, but when my clients freak out, and start e-mailing me and calling me about a "security issue on their server that they want me to look at because they got an e-mail from hivelocity", it's a bit of a drag, especially when the issue from what I've pointed out so far, amounts to nothing based on what I've seen so far personally. So just be conscientious of that before you guys decide to send out a mass hysteria e-mail like the one you sent out today -- or at least call me and talk to me, since you know I have access to wide range of servers even outside your data center. That's a really helpful way to diagnose or at least try to conclude if something is an issue or not before sending out an e-mail like that. Thanks buddy!

Posted by egillette, 02-15-2013, 10:13 PM
I have to agree with Steven on this Steve (Hivelocity Steve). . .let's hold off on sending "fixes" or creating mass hysteria among customers until we have evidence of a true exploit (meaning the file is actually part of an exploit, as I suspect it's simply an orphaned file), or until we have a way to fix whatever exploit/trojan/rookit has been identified. You're my main man. . .but we have to make sure we have substantive evidence of something happening or a way to fix a problem we discovered before sending out mass e-mails.

Posted by hivelocitygm, 02-15-2013, 10:38 PM
We have not given anyone a false sense of security. Just as we sent one email to our customers we can follow up tomorrow and the day after that with more information and as we learn more. Saying nothing would be giving people a false sense of security and letting them go on with their day. Last edited by hivelocitygm; 02-15-2013 at 10:41 PM.

Posted by Steven, 02-15-2013, 10:39 PM
egillette, It is not a orphaned file. It is the real deal. I have spent over 60 hours on this issue so far. You are checking the date wrong. Look at the 'changed' date. Last edited by Steven; 02-15-2013 at 10:43 PM.

Posted by Steven, 02-15-2013, 10:40 PM
I guess I failed reading comprehension class:

Posted by egillette, 02-15-2013, 10:43 PM
Steve. . .I see where you're coming from. Like I said, I see both sides of the equation, since I'm in sales like you, but I'm also where the boots hit the ground, since I'm a server admin too. My only request is that I think we should wait on sending mass e-mails until we can confirm for sure the existence of a specific type of exploit based on the existence of a certain file. What I've discovered in so far is that this file may or may not have anything to do with the exploit being discussed in this thread, so taking any specific action on the aforementioned file, may or may not help a client.

Posted by hivelocitygm, 02-15-2013, 10:45 PM
As I said, as we learn more we will keep our customers apprised. I hope you can see this is a fluid situation and keeping our customers educated on the situation is what we will continue to do.

Posted by Steven, 02-15-2013, 10:45 PM
Disagree all you want, it has been confirmed by myself and others. With the file present, there is unrestricted root login to the server.

Posted by Steven, 02-15-2013, 10:48 PM
Well you need to learn to word emails better, because what you think is not whats happening. Your customers are contacting me to implement this fix because, "it will cure the problem". That is a direct quote from one of YOUR customers.

Posted by egillette, 02-15-2013, 10:49 PM
Steven, maybe I'm not understanding what the 'changed' date is supposed to indicate: Two servers from my original post: Mind you these are two different systems. . .two different control panels, 32-bit versus 64-bit, etc. So what is the 'change' date supposed to identify or indicate exactly??

Posted by Steven, 02-15-2013, 10:54 PM
The modify time which is what ls see is when the contents are modified. The change time is when the metadata changes... aka when the file is placed on the filesystem, the change time will change but not the modify time. Here is an example: create a file: As a result, the 'changed' time is when the server was compromised.

Posted by egillette, 02-15-2013, 11:00 PM
Steven, I'm not disagreeing, I'm saying that the price of rice in China has nothing to do with how much cattle prices in Texas. You can continue to be vague with your response, and hostile, but if you want to be helpful, you can post some substantive evidence to indicate conclusively that with the existence of this file, unrestricted root access to ANY SERVER that has this file is possible. Until then, we can discuss all day how many pock marks radiate the surface of the moon, but until someone produces pictures, it's all just speculation. Ironically, this file exists on many of the servers I manage, and not ONE of them is sending spam as indicated by this post, shows any connections at all to SSH, or demonstrates any other sign of infection. So I'm asking you to show me some substantive evidence, or tell me how you came to the conclusion that has apparently been "confirmed by yourself and others." Identifying a random file and claiming it allows unrestricted root access to said machine is a misnomer if you can't demonstrate or explain how said file does what you're purporting it does. That is my point. So instead of being hostile, and acting like I'm some kind of moron, when I've got nearly 12 years of server management experience under my belt, why don't you explain your case clearly, or be specific. Again, this is science, not magic. In science, we can prove things -- in magic we just claim that they exist irrespective of logic. So be clear -- there is no need to be hostile and act like I'm just disagreeing. I'm simply saying that based on my research and checking at least 26 servers that contain this file, nothing seems to be out of the ordinary. Perhaps it could be because those servers have been secured properly and have appropriate firewalling in place using iptables, and not just tcp_wrappers. . .who knows?? I'm just saying, state your case man. Give me the science. The proof that this file is doing what you say it is doing.

Posted by egillette, 02-15-2013, 11:04 PM
So going by what you're saying here. . . You're saying that this machine: Has been compromised since last year?? That makes no sense. Either that, or you're suggesting that I've protected this server so well, that despite this 'compromise' that occurred on 1/6/2012 (according to what you're saying), this server hasn't sent spam, or been able to provide root access to the attacker? Maybe my security is that tight, fine, I can see that. But otherwise, you're suggesting that the attacker compromised this machine, and then just left it alone for a whole year?? Do you see why what I'm saying what you're stating doesn't add up? So are you saying my security is setup very well, or the attacker decided to just leave this machine alone??

Posted by Steven, 02-15-2013, 11:04 PM
The fact that your trying to disprove me based on 'ls' of a file, I have zero to prove to you. I know my linux, do you?

Posted by serve-you, 02-15-2013, 11:05 PM
So on one of the servers that contains this lib that isn't hacked, why don't you query rpm to figure out what package/version installed it. rpm -q --whatprovides /path/to/lib

Posted by Steven, 02-15-2013, 11:06 PM
If an attacker has root, they can do anything they want including modifying the time it was placed aswell. I am not going to waste my time with you. I know what I see, I know what I found, I know that removing the file stops the spam and the root logins from authenticating without actually having root. Enjoy having your rooted servers, because they ARE rooted. Disagree all you want, I don't care.

Posted by Steven, 02-15-2013, 11:08 PM
You should also google, you will realize that that file is not part of centos in any way or was it ever in a package.

Posted by egillette, 02-15-2013, 11:11 PM
Oh come on now. I specifically stated this was some initial research, so now you want to compare who knows more shell. Jeez, it's not a pissing contest there Mr. Bravo. It's me explaining what I see, and you explaining what you see. The difference is you reached a conclusion based on your findings, which from what I see, don't amount to much, which is why I asked questions, and attempted to understand better what you were suggesting. Instead I was met by criticism, and hostility. I know Linux very well, my friend. . .I've been doing this for 13 years, and have been a programmer since I was 9 years old. I'm 33 now. Do I need to prove anything to you?? Absolutely not, which is why I never mentioned these things. You can run around like chicken little all you want screaming the sky is falling, but before I tell my clients something is actually wrong, I'll look for my evidence myself, since you have refused to provide evidence for anything you've mentioned up to this point.

Posted by Steven, 02-15-2013, 11:13 PM
You didn't know what the 'change' value in stat was? 13 years of linux and you never learned that? If you don't even know that, then how can you even trust 'your' findings?

Posted by serve-you, 02-15-2013, 11:14 PM
Let's stop the pissing match and get back to facts here. egillette can you please see if that lib was in fact installed by an RPM?

Posted by Steven, 02-15-2013, 11:28 PM
Reread the thread, and noticed I didn't provide some info 1.) The connections are not typically logged in /var/log/secure UNLESS you raise the log level to verbose. I originally found the connections using lsof, also how I tracked down the outbound smtp connections. 2.) ps aux |grep ssh look for You will notice it is missing a pts or notty, this is the malicious processes they don't typically run very long. Run lsof -p PID and you will see outbound port 25 connections. 3.) Some compromised servers will have hundreds of sleep XXXX processes. Not all. 4.) If you strace the processes..this is the kind of stuff you will see Last edited by Steven; 02-15-2013 at 11:37 PM.

Posted by Steven, 02-15-2013, 11:42 PM
Here is actual lsof:

Posted by Steven, 02-15-2013, 11:47 PM
As a matter of fact, on this certain server, 'root@notty' is sending spam (first time I saw it). Here is something also that is interesting... -- They will connect to MULTIPLE ips on the same server.

Posted by dimm, 02-15-2013, 11:49 PM
One should change timestamps once injected: Well behaved hackers always do so

Posted by Steven, 02-15-2013, 11:51 PM
While we are at this.. we may as well show you how severe the range of exploited boxes involved this is.

Posted by Steven, 02-15-2013, 11:58 PM
First some quick tests First lets run rpm verify on this. Good here. For consistancy.. not in a rpm package. ##### Here is the strings output of a real lib.. ########## For comparision here is the output strings on a compromised lib Last edited by Steven; 02-16-2013 at 12:03 AM.

Posted by Steven, 02-16-2013, 12:05 AM
Example of those 'sleep' processes I mentioned earlier:

Posted by Steven, 02-16-2013, 12:15 AM
Here 10 packets tcpdump for those interested. I can provide you more if interested

Posted by Steven, 02-16-2013, 12:24 AM
As I stated before.. you typically will not see connections unless you set the loglevel to verbose in /etc/ssh/sshd_config. After you will see these: Unless you set it to verbose, you probably will never even know you had connections based on the log file.

Posted by Steven, 02-16-2013, 12:51 AM
It is worth mentioning, the first server I encountered did NOT have a standard SSH port.

Posted by mattmackman, 02-16-2013, 12:57 AM
I am also faced the same problem, so I have just created a script for capturing the ssh TERM environment, I got the TERM environment they are using is "dump".

Posted by Steven, 02-16-2013, 01:09 AM
On one of the servers I have snoopy logger on it: http://sourceforge.net/projects/snoopylogger/ This is what happens on connection from malicious user:

Posted by brianoz, 02-16-2013, 08:10 AM
egilette: Steven has really covered this fairly well at this point - but a standard technique for expert hackers is to reset the modify time on a file to an expected "old" or safe-looking time so the file doesn't look like it's been hacked. Unfortunately the days when we used to be able to find hacked files like this seem to be gone: find -mtime -7 print | xargs ls -ld The libkeyutils.so.1.9 file looks to me like it is a hacked file as it is not part of an rpm and not part of systems that have not been hacked. This probably isn't great news but if you have that file on all your systems there's a chance many of them may have been hacked. It's clear you know your stuff though, and I have about 3 times your Unix admin experience, and unfortunately I have to back Steven here on this point. On a more general note: It has to be said here that Steven has done some great investigative work. Absolutely amazing work, it just makes clear you know your stuff well, well done and thanks from all of us for your contribution, you're also my main man! Now if we could keep the thread on track for discovering how they got to root that would be really helpful

Posted by bukzrock, 02-16-2013, 08:22 AM
Hello When I run this my VPS gose down. Please help me 1. SSH to server 2. Run 'updatedb' 3. Run 'locate libkeyutils.so.1.9' Please follow the steps below to clear the expliot. 1. SSH to the server 2. cd /lib64/ 3. rm libkeyutils.so.1.9 4. rm libkeyutils.so.1 5. ln -s libkeyutils.so.1.3 libkeyutils.so.1 6. Restart ssh 7. yum update kernel and Reboot to close any active connections

Posted by FastServ, 02-16-2013, 10:28 AM
This machine could be infected and he doesn't know it... http://www.rqna.net/qna/smvrq-keyctl...en-errors.html I wonder if it even has Cpanel. That + based on the infected Plesk machine in this thread, this might be a CentOS specific issue and have nothing to do with Cpanel except for the attention Cpanel brings to a server.

Posted by Steven, 02-16-2013, 11:29 AM
I wouldn't put a firm correlation to it yet. You can basically drop a rootkit on any plesk server if you have the admin panel login if the ability to create root cronjobs has not been disabled. ----- There appears to be a couple variants of this file. Someone I know has a file that has a minor variation.

Posted by Steven, 02-16-2013, 11:30 AM
rm -f /lib64/libkeyutils.so.1.9 ldconfig service sshd restart

Posted by egillette, 02-16-2013, 12:35 PM
brianoz: Thank you for actually making some sense in the way you describe things. My issue with Steven was that aside from "puffing his chest out" and screaming like Chicken Little with his hair on fire, he wasn't offering anything substantive, outside of: "I found this file and I believe it's hacked." In my experience, when a guy runs around like his hair is on fire, and is constantly "puffing his chest out" to attempt to make others look bad, or ridicules the experience of another person, it's usually because they are over-compensating for other areas in which they are lacking. It didn't help that I see his business model is completely dependent on the business that such hysteria drives as well. He stated what he saw, and I stated what I saw -- I simply said I didn't believe the file had anything to do with anything, plain and simple, and he turned it into a pissing match about who knows more shell, which is something that only someone immature would do -- he did the same thing to Steve (Hivelocity Steve), rather than explaining things in clear-cut ways that help the individual who may not be understanding his point, get what he's saying. Perhaps the guy has no friends, or lack social skills, I don't know. I've involved in many debates, even politically during Mitt Romney and Barack Obama's campaigns, and I've always been able to make my points, without making it a pissing contest. In any case, I'm done trying to get him to make any sense, as I'd rather talk with other adults here who can be clear and concise, like you for example, and serve-you, who ironically stated that it wasn't a pissing match (thanks for that serve-you), since the same was also clear to him, proving I wasn't the only one that saw Chicken Little poking his chest out and running around like his hair was on fire, instead of acting like an adult. You can be the greatest investigator in the world, but if you act like everyone else is less intelligent than you are by insulting them or their intelligence, in the end your skills don't make much difference. The old adage says: "A wise man speaks, because he has something to say. A fool speaks because he has to say something." So I'll direct my replies at you guys and just leave Steven out of the equation since he's more concerned about looking like a know-it-all, rather than being helpful and explaining his points clearly (which he attempted to do afterwards, though it was a day late, and a dollar short in my opinion). So, here's what I've gathered in so far. . . This file has been pointed out being no part of any package, which I have also confirmed on every system in which this file appears -- however, I can also confirm even after turning on verbose logging in SSH that ssh connections are *NOT* being made to any of the affected systems. In addition, there is NO SPAM being sent from any of the machines in question, and the version of CentOS and control panel, some of the machines are running Plesk or DirectAdmin for example, vary far and wide. So I guess what I'm saying is. . .if these machines are indeed compromised, outside of the file, there is no other evidence that these machines are sending spam, or are allowing outside SSH connections -- but this could also be the result of how I have these machines firewalled. This is what I tried to explain before. So what I need to know Brian or serve-you if either of you guys wouldn't mind giving me a little more feedback. . .like how can I test and conclude beyond a reasonable doubt that this file indeed represents a compromise? OR. . .how can I test the theory that this file indeed represents a compromise on the systems where the file is present. I want to be sure, before informing my clients, so that I'm not raising any false alarms -- whether this "tumor" we've identified is benign or malevolent which is a point I'm sure you guys who seem to be as logical as me, can agree with. Again, the systems in question vary far wide, we're talking: CentOS 5.x CentOS 6.x Plesk DirectAdmin ISPConfig cPanel 32-bit and 64-bit Thanks for any feedback or insight you or serve-you can provide. Last edited by egillette; 02-16-2013 at 12:39 PM. Reason: Clarification that no SSH connections are being made.

Posted by Steven, 02-16-2013, 12:53 PM
Enjoy your hacked servers, you are so hard headed. There are enough people around that can verify it, that have had spam sent, by direct result of this file, and were able to stop the spam by moving the file, running ldconfig and restarting sshd and are able to bring the spam back by moving the file back in, running ldconfig, and restarting ssh. Why don't you prove that the file is not a rootkit. Go look at the strings, there is a huge difference between a real library and the compromised one. SSH has to be publicly accessible in order for them to be able to do it. If you have ssh firewalled off.. even if you are backdoored they cant login -- therefore you would have no signs of it. Last edited by Steven; 02-16-2013 at 12:59 PM.

Posted by serve-you, 02-16-2013, 01:08 PM
No offense Eric, but your arguments here aren't helping your cause. While Steven can certainly be blunt, he is a well respected member of this community who's been around forever and generally knows his stuff. You on the other hand, came in with the attitude that you knew better, and he couldn't back up his talk. I think Steven has provided more than enough information in this thread to be helpful to others. I'm not exactly sure what anyone else can tell you that hasn't already been stated here. We all think that this lib has been injected on these servers. The question now is just how. The fact that it doesn't belong to any RPM makes it a lot less likely that it was installed by any legitimate means. If you have this on a ton of servers that do not yet appear to be doing anything bad, I'd first get it out of there, then start trying to figure out what the common denominator is on those machines.

Posted by egillette, 02-16-2013, 01:09 PM
By golly. . .I think I just heard something intelligent finally come out of his mouth aside from the acid he's been spewing all this time. That was my original question. . .as I stated previously, and he finally answered it when he took a moment to stop poking his chest out, and acting like his hair was on fire. So now serve-you and brianoz, I guess as you guys originally pointed out, it's time we identify how this occurred in the first place, since the file *might* be compromised, but lays dormant when the affected system is firewalled properly as Chicken Little even stated above. At this time, based on how servers under my management are firewalled, I can confirm that the file, which may or may not be an actual backdoor (or possibly as much as *part* of a backdoor), is present, but none of the other signs of a compromise are present (no SSH connections, no spam being sent, etc). Again, as I stated previously, the affected systems vary far and wide, in terms of OS and control panel software (if any): OS'es: Cent OS 5.x Cent OS 6.x Cloud Linux 5.x Control Panels: Plesk 8.x, 9.x, 10.x, 11.x cPanel DirectAdmin ISPConfig Both 32-bit and 64-bit The only thing they all have in common are the IPTables / TCP_Wrappers rules I created. Willing to listen and deliberate with any adults on the matter, and come to some conclusions about how this happened, and how we can prevent it in the future.

Posted by Steven, 02-16-2013, 01:10 PM
If you are restricting access -- you will not see the compromise.

Posted by egillette, 02-16-2013, 01:20 PM
serve-you, I'm sorry but you're incorrect in your assessment. I didn't say I knew better about anything, I simply stated more than a few times what I had seen, and how I failed to see a correlation between the file, and any compromise, which my later statements have backed up if you take some time to read them up to this point. I'm very humble in every aspect of my life, including posting messages in online forums -- it was Steve who turned it into a pissing contest. That said, I could care less about how "respected" someone is -- the moment they disrespect me, they've lost my respect. You can be blunt without insulting someone, or their intelligence. You don't have to agree with that statement, if you don't feel like it, but again, you can be blunt without being insulting, Steven clearly hasn't mastered that art, despite his level of "respect" in this community. You would think someone who comes along who says: "Hey look I'm investigating and see no correlation between this file you've mentioned, and an actual compromise because I show none of the signs on my system you've pointed out. . ." would be addressed in a manner that establishes a point (which he later did by saying that since I've firewalled these systems in a specific way, that they aren't able to login) instead of trying to insult the person's intelligence and trying to make it seem like the person doesn't have a handle on shell commands. In any case, I backed up everything I said, and I invite you to demonstrate where I didn't since you have this impression that I "came in with attitude that I knew better" -- show me where I said that exactly, or even acted as much. I'd love to see how that turns out. . . Anyway. . .again, adult conversation and deliberation is invited between you, and brianoz in terms of how this occurred is invited. I've asked Steven to no longer address me, since it's clear despite his "respected status" he and I will never have any level of civil discussion.

Posted by dimm, 02-16-2013, 01:40 PM
I have scanned hundred+ cPanel servers serving 700-800 accounts each and found nothing "unfortunately". Do you guys happen to think that hacked servers were just ordinary ssh brute-forced? For the wise attacker this is not a big deal to clean up the mess in wtmp, /var/log etc. Do you have any compromised servers where PermitRootLogin is set No?

Posted by Steven, 02-16-2013, 01:50 PM
When you strace sshd, and login to the server normally there is a outbound port 53 connection to an IP address that is not in /etc/resolv.conf.

Posted by jalapeno55, 02-16-2013, 02:22 PM
Did anyone notice a file in /tmp called "..." while the spam was going out? There was a spam incident I noticed on a server and am wonder if its related to this. The user had an open SSH connection, but I didn't see this file: libkeyutils.so.1.9

Posted by Tyl3r, 02-16-2013, 02:38 PM
That directory is pretty common for any exploit really.

Posted by jalapeno55, 02-16-2013, 02:41 PM
Actually it was a 0 byte file, which I found unusual, like maybe it was suppose to be a socket.

Posted by Steven, 02-16-2013, 02:43 PM
What was the user/group of the file?

Posted by jalapeno55, 02-16-2013, 02:47 PM
User and group were the customer's username (apache running in suphp/CGI mode)

Posted by dimm, 02-16-2013, 02:50 PM
BTW: Linux kernel race condition with PTRACE_SETREGS (CVE-2013-0871)

Posted by hb9aj4fn, 02-16-2013, 03:00 PM
dimm - you did not provide any links, so I used Google, and here is a two: http://seclists.org/oss-sec/2013/q1/326 and a reply http://seclists.org/oss-sec/2013/q1/334 This is scary! I hope we soon will get information about if the latest kernel from CentOS/RedHat is vulnerable or not?

Posted by Steven, 02-16-2013, 03:12 PM
Keep an eye out on this link: https://access.redhat.com/security/cve/CVE-2013-0871 When they release their statement it will be there.

Posted by dimm, 02-16-2013, 03:40 PM
i need one more post to be allowed linking on google

Posted by unSpawn, 02-16-2013, 05:48 PM
FWIW since detection hasn't been added yet to other sources and while it doesn't fix anything I updated RKH in CVS and whipped up a ClamAV signature over at LQ (which you can use with inotify or incorporate in whatever you're running) to at least get an early warning. *Thanks to Steve for submitting samples and thanks to ClamAV for allowing efficient "and-or-or" string matches in sigs.

Posted by hb9aj4fn, 02-16-2013, 05:58 PM
Thanks. They have created the page now, so it's no longer a 404 not found.

Posted by unSpawn, 02-16-2013, 06:11 PM
CVE doesn't point that way yet but https://bugzilla.redhat.com/show_bug.cgi?id=911937 shows Oleg Nesterov of Red Hat committed fixes for CVE-2013-0871 (back in January FCOL).

Posted by iseletsk, 02-16-2013, 06:16 PM
egillette -- plz, really what you have is very valuable to the community. If you have DA & Plesk servers compromised, it significantly minimizes the possible vectors of attack (unless attacker is using multiple vulnerabilities). Could you do: rpm -qf /lib64/libkeyutils.so.1.9 on compromised servers and provide the output? We can disagree if the presence of this file means that the server is compromised (I am with Steve on this one) -- but that output would help us to confirm and move on trying to figure out what is going on.

Posted by iseletsk, 02-16-2013, 06:22 PM
Looks like a real deal: http://seclists.org/oss-sec/2013/q1/326

Posted by iseletsk, 02-16-2013, 06:24 PM
Not sure yet, but this might help as a temp solution: http://people.baicom.com/~agramajo/misc/no-ptrace.c

Posted by Steven, 02-16-2013, 06:51 PM
I'm worried there may be a kernel module being loaded from this backdoor. Waiting a KVM on a box. I didn't see one last week but I'm revisiting it. Igor that looks like a possible fix until a real patch. Ksplice... get on it

Posted by egillette, 02-16-2013, 07:16 PM
Based on the claim that the existence of this file is evidence that a system has been compromised, then a system that has PermitRootLogin set to no would make no difference, since I have systems containing the file that do not allow root login (i.e. PermitRootLogin set to no). Honestly, I'm still not quite sure what to make of it. . .my curiosity is if the existence of this file is evidence that a system has been compromised, than I'd like to discover how it was compromised in the first place. Again, if we're going based on the fact that the existence of this file signifies a compromise, we're talking hundreds of systems -- maybe more.

Posted by egillette, 02-16-2013, 07:19 PM
I believe maldet uses inotify, so maldet should be able to detect this using the ClamAV signature you created -- am I offbase or is that accurate?

Posted by bdraco, 02-16-2013, 07:31 PM
That didn't work for me, however I ported it to linux 2.6. I've attached the updated version. I'm not sure if this will protect against the problem, or if its safe to do. Attached Files disable_ptrace.pl.zip (11.7 KB, 50 views)

Posted by egillette, 02-16-2013, 07:32 PM
Sure thing! I don't disagree or agree that the presence of the file means the server is compromised -- that is the point, and why I'm here. I'm looking to dig deeper, to find out how a system containing this file was compromised in the first place, and if the file really indeed signifies a compromise, or if the file is legit, and perhaps is being used to create a compromise for example. As far as I'm concerned, the jury is neither in/nor out on it. I respect Steve's investigative abilities, what I don't respect is how instead of explaining himself so that I could draw a baseline conclusion, instead he attacked me, which isn't cool, and I could care less about how many people agree with him or not. I have clients to answer to as I'm sure some of the others here do, and I *never* attack anyone, unless they attack me first. I came asking questions, presenting what I had discovered in doing some initial research -- something that in this community should be welcome, instead I was ridiculed, that's no way to get to the bottom of things. In any case, here is the output from a Plesk system and a DirectAdmin system respectively: ^^^ is a 32-bit system. Also, that system does not allow root login (permitrootlogin is set to no in sshd_config). ^^^ is a 64-bit system. This system DOES allow root login (permitrootlogin is set hashed out in sshd_config). So I think that eliminates it being exclusive to a specific control panel (cPanel for example), or OS (CloudLinux). There is one other system I want to check (it's a CloudLinux system). . .I'll report my results here shortly, as I have to wait to get access to the system from the client.

Posted by Steven, 02-16-2013, 07:44 PM
egillette, As a civil question, if you haven't set me to ignore in WHT's settings. Can you try stracing the sshd daemon process and logging into the server using a password and see if you see connect() calls to an IP that you do not have in /etc/resolv.conf on the affected machines. It will look something like this:

Posted by Steven, 02-16-2013, 07:50 PM
For those interested in testing this

Posted by unSpawn, 02-16-2013, 10:18 PM
Haven't tried it but should work, yes. Be aware though it doesn't fix anything, it would only warn you.

Posted by nibb, 02-16-2013, 11:52 PM
Is it possible that this was pushed as package update somehow? What happens is a repository is compromised? Or if the CloudLinux or cPanel update servers where compromised? This would push a compromised file to every single customer on the next update. This is my biggest concern so far regarding updates. You an secure your server all you want, but what about those out of your control?

Posted by TravisT-[SSS], 02-16-2013, 11:53 PM
That would be true but none of the servers we manage seem to have been compromised and we are always running updates. Maybe a 3rd party repo..

Posted by brianoz, 02-17-2013, 03:30 AM
If one of the main servers gets compromised like that, you'll hear about it as headline news in a whole lot of ways. They usually find out very quickly and I'd guess they have safeguards in place to prevent that anyway.

Posted by nibb, 02-17-2013, 04:03 AM
What about CloudLinux? This are third party repos. I don´t think this is the case, as Igor is very smart and im sure he takes security as an important issue. But maybe indeed this is via some update from some repository.

Posted by tomfrog, 02-17-2013, 06:38 AM
It's not a repository problem. A bug was created at red hat for a linux kernel vulnerability affecting fedora-all, by which a local unprivileged user gains root. https://bugzilla.redhat.com/show_bug.cgi?id=911937 If you read this thread carefully, it's not a cpanel or cloudlinux bug or repo issue.

Posted by nibb, 02-17-2013, 07:20 AM
How long would this take to be released, until it arrives to CentOS? So by now everyone is vulnerable?

Posted by iseletsk, 02-17-2013, 10:32 AM
If our repository would be compromised & packages changed - yum would complain about invalid signature of the packages, and wouldn't install them. Also, it wouldn't affect both CentOS & CL systems if that would be the case. ptrace bug sounds as the most probable vector right now.

Posted by Patrick, 02-17-2013, 10:33 AM
Reading comments on another website, I'm not convinced this is the exploit that is being referenced in regards to the SSHD root kit: https://news.ycombinator.com/item?id=5230262 People there are suggesting that the timing has to be perfect for the exploit to work making it extremely difficult to achieve a compromise: "But even with access, this is awfully hard to exploit. You need a precisely timed kill, followed by a precisely timed preemption. That's really (really) hard on a multi-cpu machine, and it's worth pointing out that the exerciser case in the bug has to patch in a sleep for the exploit to work. It's not clear if they have a real world exploit running, though of course the race is real." "The race condition is very possible but fairly unlikely unless you apply the Proof-of-Concept patch to make a huge gaping delay in the kernel where the exploit can run." ... just food for thought but until a simple ./exploit is made public I wouldn't hold my breath that it's related.

Posted by JonnyQuags, 02-17-2013, 10:39 AM
On two test systems (centos 6 + cl 6) I was not able to get the sample exploit to work beyond forking a lot of processes, of course there may be many other variants floating around.

Posted by nibb, 02-17-2013, 10:48 AM
I don´t know if this useful, probably not. But 3 months ago I setup a new cPanel server for a friend and send him the root password, told him to change logins and setup WHM settings on his own. I adviced him to tweak the security settings and configure the server on his own. All I configured was IPtables to only allow cPanel ports. He never did so, never configured anything on WHM, it was all on the default settings that come with cPanel after installation, 2 weeks later, spam was being send from the server. I created a cPanel ticket and they found nothing, not rootkits, nothing. The server was sending allot of spam. And it has severe load spikes. Finally I tracked allot of users from IPs connected to the SSH port, dozes of them. Connections where established but somehow now logins where registered at all in the systems like if nobody was actually logged in the OS, same reason why cPanel said they could not find anything. Me and cPanel checked the server and nobody could see any suspicious activity except from what I pointed out and the Spam being send. I never knew if they got in or how if they executed using some kind of exploit on the 22 port which runs SSH. I killed the server and send him another one, this one I tweaked in settings before sending and restricted port login per IP, he did not had a single problem since then. I never ever knew what the problem was because I did not considered a default WHM so insecure. It was a new CentOS 6 x64 installation, nothing was uploaded, yet, everything was patched, only cPanel was installed and it was hacked in days. Im still curious today. But I find it similar to this thread where complaints come from the server sending spam. What is strange is that this was 3 months ago.

Posted by Steven, 02-17-2013, 11:20 AM
Due to the increasing rate of servers being exploited, and the fact that some are hosting very simple items are being compromised... i personally feel this is a remote exploit in a daemon. Could be off base here.. but I am pretty skeptical its the new kernel vulnerability.. not saying the vulnerability can't be used to hack servers.. I just don't think its the cause here. Its worth mentioning, the first server I encountered had a nonstandard ssh port. Last edited by Steven; 02-17-2013 at 11:24 AM.

Posted by egillette, 02-17-2013, 11:31 AM
Steve I didn't block you. . .I always give everyone the benefit of the doubt, and as I suspected, it appears we just got off on the wrong foot, likely because we both were pretty passionate about what we thought. In any case. . .the strace is running, and I'm tailing strace.output, but my question is, nothing was immediately returned, and I was unsure by the way you proposed if you were asking me to allow the strace to run for a specific period of time or not. That said, I backgrounded it, and am tailing it's output, just in case that's what you meant. Is there any specific length of time you suggest running it for, or should I have seen an immediate result (based on the presence of connections)? Also. . .bad news. . .the CloudLinux machine that I was waiting on access to does not have the file in question, so that machine is ruled out -- that is the only other CloudLinux machine I had access to, as all the others are Cent, Fedora, RHEL, and Debian. Stranger yet is that I have not found the file on the Fedora, RHEL, or Debian machines -- only on Cent machines. So I think we can draw some reasonable conclusions in that case, but before doing so, if anyone else has access to Fedora, RHEL, or Debian machines, maybe we can get some additional feedback to see if this is only affecting CentOS machines.

Posted by egillette, 02-17-2013, 11:33 AM
That I can agree with for sure, because 98% of servers under my management have a non-standard SSH port, including my own.

Posted by egillette, 02-17-2013, 11:37 AM
Yes, I'm aware of that and exactly what I want/need -- and the majority of machines under my management run maldet, so I was looking to see if maybe I can push out an update to all of them so that rather than checking each machine one by one, I can instead have them notify me. That might sound a bit lazy, but there are hundreds of machines under my management, and it's going to take awhile to verify/check them all one by one if you know what I mean.

Posted by Steven, 02-17-2013, 11:44 AM
Run the strace, and then login from another terminal. It should occur right on login. It was actually a member of the cpanel security team that brought that to light to me. However, after my post to you, I found more variations of the file.. theres lots of different md5sums possible for this so its possible that some files may not have that IP in it as the IP is now 'dead'. Search for port 53 connections, perhaps they changed it on the file you have on that server which would be useful to know. They do a dns lookup it appears, but without a working server, we cannot run the lookup to see what it returns.

Posted by FastServ, 02-17-2013, 11:47 AM
I just checked about a dozen high traffic shared servers (some with thousands of accounts and frequent issues with php shells) and can't find the exploit on any of them. All servers are CentOS5/6 +WHM and Ksplice installed. IF Ksplice has played any part in protecting them, this might not be a 'new' vulnerability.

Posted by egillette, 02-17-2013, 11:50 AM
On a side note. . . Has anyone else been following this thread?? https://news.ycombinator.com/item?id=5230262

Posted by Steven, 02-17-2013, 11:51 AM
I have some people who came to me that have servers with ksplice with this problem..

Posted by Steven, 02-17-2013, 11:52 AM
Yep.. Many in this thread, but there is skeptics as to if this is the cause of the problem. We don't know if RHEL is vulnerable to it yet. For reference: https://access.redhat.com/security/cve/CVE-2013-0871

Posted by egillette, 02-17-2013, 12:01 PM
Steven, Got it. Logged in from the another terminal to the machine 1 (Plesk 9.5.x, CentOS 5, 32-bit), and see myself in the trace: But no weird port 53 connections outside of those that I set in /etc/resolv.conf (OpenDNS): Grabbing -o output from the other machine now. . .

Posted by Steven, 02-17-2013, 12:03 PM
Can you provide the md5sum of the libkeyutils.so.1.9 file in those server(s) please? One more test, can you run: ls -l /lib64/libkeyutils.so.1

Posted by egillette, 02-17-2013, 12:10 PM
Steve, Similar output from the second machine, except this time the output is a bit different like you had described That IP address is either the IP address you posted before, or is similar to it. . .I'm about to check your previous post.

Posted by egillette, 02-17-2013, 12:20 PM
Server 1 (Plesk 9.5.x, CentOS 5.9, 32-bit): Server 2 (cPanel 11.34.x, CentOS 6.3, 64-bit): ^^^ Server 2 is the one that just showed the IP you mentioned in your first post by the way. . . Last edited by egillette; 02-17-2013 at 12:22 PM. Reason: Corrected output

Posted by hb9aj4fn, 02-17-2013, 12:21 PM
egillette, it is the same ip address, I blocked it the first time Steven posted it, and ... Last edited by hb9aj4fn; 02-17-2013 at 12:24 PM.

Posted by Steven, 02-17-2013, 12:21 PM
Yeah.. I think that IP was some kind of call out to the attackers. On the first server can you run ls -l /lib/libkeyutils.so.1 I want to see if you are linked to /lib/libkeyutils.so.1.9

Posted by Steven, 02-17-2013, 12:29 PM
Could you toss up a strings output for the library on the 'mail' server? I dont have that hash.

Posted by egillette, 02-17-2013, 12:42 PM
Here's the output: It's linked alright. And strings output on that file (/lib/libkeyutils.so.1.9): This server has SSH firewalled off, which may be why there are no connections, as you had suggested prior. The other server also has no SSH connections, but I'm gonna check a few other things as well. . .namely the processes through lsof output -- will post my findings. Last edited by egillette; 02-17-2013 at 12:43 PM. Reason: Defined which "file" I was referring to.

Posted by egillette, 02-17-2013, 12:48 PM
Just doing some digging. . . Turns out the IP in question is a Bellsouth residential DSL line, and strangely enough, purports to be hosting a website called "googletrafficbots.com" http://awesomescreenshot.com/044xu5b33 Should we inform Mr. Meresiev that his IP is being used as part of an exploit being deployed on a substantial number of servers (assuming Mr. Meresiev is unaware of such exploit -- for all we know he manufactured it, and maybe wasn't smart enough to hide his tracks better with a public whois record. LOL! Just speculating. . .) http://awesomescreenshot.com/0faxu5he0

Posted by Steven, 02-17-2013, 12:51 PM
That is indeed interesting eric.. That strings out is VERY different from other libraries.

Posted by egillette, 02-17-2013, 12:56 PM
Is it possible that is the case because it's a 32-bit lib as opposed to the 64-bit ones we've all been seeing?

Posted by egillette, 02-17-2013, 01:00 PM
Luckily, I have most of these machines clustered in one way or another using CSF, so I just pushed out the deny request to all of them from my main server. My main server neither pushes requests nor receives requests from the other servers, but I do have it set as the master, so I can push out config or allow/deny requests easily for cases like this. Last edited by egillette; 02-17-2013 at 01:02 PM. Reason: Demonstrated output

Posted by Steven, 02-17-2013, 01:01 PM
It would be a little different, but you don't have the external port 53 call either so its different in that respect aswell.

Posted by egillette, 02-17-2013, 01:05 PM
Yeah, I see what you mean. . .this fact also leads me to suspect as I did previously, that these libs *may* have started out as legit libraries, but then may have later been used for this specific exploit. Either way, it definitely seems we may be getting closer to figuring this all out. . .

Posted by FastServ, 02-17-2013, 01:08 PM
Looks like it's transmitting your SSH password via UDP 53. http://forums.cpanel.net/f185/sshd-rootkit-323962.html

Posted by Steven, 02-17-2013, 01:22 PM
Yep. Me and jeff were talking / testing things when this came to light. Its really interesting. I am intrigued that eric has a library that is not doing it. Last edited by Steven; 02-17-2013 at 01:25 PM.

Posted by egillette, 02-17-2013, 01:31 PM
Likely because of the way I have my machines firewalled as you had pointed out prior. In addition, the IP in question was also blocked implicitly in the firewall by CSF. Maybe that's why it never got to transmit the password. . . Still, I want to find out how this happened in the first place.

Posted by Steven, 02-17-2013, 01:33 PM
It would still transmit regardless of firewall rules, it just would never make it to its destination. The fact remains is, your library is different, it physically is not making the call. They question is why..

Posted by egillette, 02-17-2013, 02:20 PM
The question of the hour for sure. Let me know if I can be of assistance. In the interim, I whipped up a quick and dirty little bash script that others here or reading this thread may be able to use to remove the file from their affected machines. Instruction on how to use this script: 1) Login to your server as root, or login to your server and use sudo or su to gain root access. 2) Watch output from the script to tell you if the file is removed. 3) No *output* from locate is an acceptable result. Disclaimer: This is a quick and dirty "for now" solution, if anyone has any input to add, feel free to do so -- specifically until we establish how the file ended up on the affected systems in the first place, there is the possibility it could somehow come back. Note: If you receive an error when running the final command, it's possible you don't have mlocate installed, in which case you should run the code below and then run the the code from Step 2: Last edited by egillette; 02-17-2013 at 02:22 PM. Reason: Correct one of the commands. . .

Posted by egillette, 02-17-2013, 02:29 PM
Steven, I deleted the file on the system in question, but made a copy of it, in case you'd like to scrutinize it further. Lemme know if you'd a copy of it, and I'll put in a publicly accessible directory. I want to figure out how this happened, because even deleting the file for now is fine, but if we don't figure out how it got there in the first place, I suspect we'll possibly be seeing it return at some point.

Posted by Steven, 02-17-2013, 02:33 PM
Yep. I would love a copy

Posted by egillette, 02-17-2013, 02:47 PM
Here you go sir: http://www.gsol.us/libkeyutils.so.1.9 This is from the 32-bit system by the way. Last edited by egillette; 02-17-2013 at 02:48 PM. Reason: Describing the file.

Posted by Steven, 02-17-2013, 03:19 PM
Eric, For your script, you may want to add something to kill existing sshd connections. If there are any open connections sending spam restarting sshd only will not stop that.

Posted by egillette, 02-17-2013, 03:56 PM
Yeah, that's a good point. But then, if I kill all SSH processes, wouldn't that terminate the session the user is running my script with too?? Maybe I should have them use nohup so that it runs even after they get disconnected?? Or. . .would the -t switch work with ps to find kill the SSH processes but leave the user connected??

Posted by FastServ, 02-17-2013, 04:01 PM
Full reboot would be in your best interest, especially given what little we know about these libs or how they got there.

Posted by Steven, 02-17-2013, 04:02 PM
Well, the connections this problem brings don't launch with a pts.

Posted by Steven, 02-17-2013, 04:02 PM
True to a point. Good time to ensure all updates are applied.

Posted by egillette, 02-17-2013, 04:03 PM
Another good point. . . So the question is. . .do I reboot the user's machine in my script, or advise them to reboot by echoing such to the terminal? What do you guys think??

Posted by FastServ, 02-17-2013, 04:27 PM
Have it echo a comment suggesting that a yum update and reboot should be done. Auto-reboot might not be a good idea.

Posted by egillette, 02-17-2013, 04:42 PM
Yeah, I figured that would be the common consensus. It's done, I modified it and added the following:

Posted by Losvre, 02-17-2013, 04:52 PM
This is getting more and more interesting as well as scary for someone who is not that expert in server management.. Thank you all for your efforts and for getting it to public

Posted by egillette, 02-17-2013, 05:06 PM
Yes, and while we haven't yet identified how the file is being created on compromised systems, at the very least, you can execute my script and follow the directions to help limit your exposure to the exploit. See this post: http://www.webhostingtalk.com/showth...21#post8561821

Posted by Losvre, 02-17-2013, 06:59 PM
Hi egillette, Thanks a lot for your efforts.

Posted by themedia, 02-17-2013, 07:40 PM
Did anybody find this on CentOS 6.3 machines 64 bit with all updates applied? I wasn't able to find it on any of the boxes i manage ~15 but then again i haven't had an intrusion in a few years now.

Posted by Steven, 02-17-2013, 09:01 PM
Redhat is worthless yet again https://bugzilla.redhat.com/show_bug.cgi?id=911937

Posted by kaniini, 02-17-2013, 09:45 PM
Yes, and the data send to that port 53 connection is not a normal DNS packet as far as I can tell.

Posted by iseletsk, 02-17-2013, 10:35 PM
The more I think about it, the more I think it is some daemon exploit and not a privileged escalation via kernel. Given that some boxes running CageFS were exploited -- if exploit would be delivered via end user account, /lib & /lib64 wouldn't be available to attacker (it would be a copy of those directories instead). So, unless hacker explicitly made a work around to deal with CageFS (which probably possible with ptrace kernel exploit, but highly unlikely), that library would never make it to /lib & /lib64.

Posted by jalapeno55, 02-17-2013, 11:02 PM
I'm leaning towards this as well, as the PoC wasn't really replicable, and considering RH's response (or lack thereof)

Posted by Steven, 02-18-2013, 12:09 AM
Lets talk about daemons here. Typical configurations SMTP: 1.) Cpanel - Exim 2.) Directadmin - Exim 3.) Plesk - Qmail OR Postfix (default on new install) POP3 1.) Cpanel - Dovecot OR Courier 2.) Directadmin - Dovecot 3.) Plesk - Courier FTP: 1.) Cpanel - Pure-FTP OR Proftpd ( I have seen pureftp servers compromised, so we can't blame proftp here). 2.) Directadmin - Proftpd 3.) Plesk - Proftpd And then there is webservers (internet facing -- in a reverse proxy situation) 1.) Litespeedtech -- I have seen litespeedtech servers compromised. 2.) Apache -- I have seen apache servers compromised 3.) Nginx -- I have not seen any of these, limit sample. --- Now, I wonder how many of these compromised servers have mysql internet facing, and if they are... what version are they running. I guess it can be done local aswell. There is always the chance that ssh itself has an exploit.. but that is kind of hard to believe for servers which do not have a standard ssh port that is compromised. There is also the possibility of bind. There are not many options for daemon exploits, due to lacking common denominator. Last edited by Steven; 02-18-2013 at 12:19 AM.

Posted by Steven, 02-18-2013, 12:36 AM
Has anyone seen this on something -not- centos, redhat, or cloudlinux? Example, ubuntu or debian?

Posted by RRWH, 02-18-2013, 01:08 AM
Since it appears that there are at least several variants of this trojanned library out there, rather than just delete it, to quarantine and fingerprint each version found along with some basic system info: IE grab an md5sum of the file, uname -a output as a minimum. Maybe rather than deleting, to move it and change Permissions (and attributes) to preserve any/all evidence. So far, it appears that there are several variations/versions floating around - and mybe by preserving there might be a clue within one of these version that lead to the how Sure - run a quick scan to determine if the file exists, remove the symlink to it and re-create the symlink to the real 1.3 version of the library - which by all accounts does not look to be removed from affected systems. So far there has been some great work done and reported here, so rather than blindly suggesting deleting the file, how about a nice set of instructions on how to go about gathering as much info as possible and hopefully find the how.

Posted by BreakFree, 02-18-2013, 01:41 AM
We have a little over 12 servers online with CentOS 6.x+, CPanel, and CSF. SSH on non-standard, Port 22 is blocked completely, though with at that, we use keys to access the servers over passwords just for security sake and SSH access isn't provided to clients. I've checked each of the servers and none of them show signs of the file(s) being discussed, nor the processes being ran, so it may be with the way we have the boxes setup. I do have 2 Ubuntu KVM VPS's online as well, though they're not showing anything out of the ordinary. They're mainly used for testing, though public sites are on them and we do have a few clients that use them for testing as well. I've not used Debian in quite some time, more so because our clients want a control panel (such as CPanel or DA). I'll apologize in advance if I missed anything obvious in this thread, I skimmed over it and passed through the original misunderstanding that took place. Just grabbed the security related info and verified ours against it.

Posted by tomfrog, 02-18-2013, 02:49 AM
You have cpanel and plesk, so smtp daemon is out? You have dovecot and courier, so pop3 daemon is out? You have both ftp deamons, so FTP is out? You have Litespeedtech and Apache, so web server is out? You have mysql port blocked in csf, so mysql is out, except local, but also on simple servers without relevant local accounts...?! Can anyone confirm that a non centos, fedora and red hat server was hacked? ssh and bind?

Posted by Steven, 02-18-2013, 03:31 AM
Tomfrog, What I'm trying go say... no common denominator if people have has plesk and direct admin hit. Only ssh bind and mysql are independent from the panels. At least for daemons.

Posted by Zimple, 02-18-2013, 04:35 AM
I have one got attacked which use NSD.

Posted by tomfrog, 02-18-2013, 04:59 AM
I've read what you wrote and also Igor's posts. Great work, Steven! Just trying to understand your point of view. If it only affects all-fedora servers, then it's probably a kernel vulnerability, right? But then you still have the local account hack? Open source scripts like wordpress and joomla? Any server hacked where a local account hack is unlikely?! NSD hacked. Just as you wrote, Steven: ssh and mysql daemons?! Maybe my thoughts can help others understand what's going on. I will leave the hard work to Steven and others...

Posted by The3bl, 02-18-2013, 07:32 AM
Don't run yum update until the system has been rebooted or all services that use the libkeyutils is killed and restarted. Yum itself uses libkeyutils. Running will show you that yum and lots of services including httpd named exim etc.. may have processes running in memory that are connected to the hacked libkeyutils.so.1.9. In fact if you have a compromised system change all passwords on everything there is no telling what passwords were sent to the hacker. This is just a sample from a hacked server we investigated.

Posted by unSpawn, 02-18-2013, 07:42 AM
Like I said in the other thread: I should point out the overarching problem here: replacing root-owned items in a root-owned directory means the perp had root rights already. Trying to "fix things" by removing the offending library may mitigate symptoms but it does not address the cause. See the standard literature on the 'net on how to deal with a breach of security properly (the SANS Reading Room for example) or, since it is a root compromise, draw the only logical conclusion possible.

Posted by The3bl, 02-18-2013, 09:04 AM
Yep gotta agree you can put a band aid on it but it is still bleeding some where. Thing is until we know how it got there in the first place an OS reload may not be enough.

Posted by mattmackman, 02-18-2013, 09:20 AM
Well, Me got hacked again I have deleted the file libkeyutils.so.1.9 before 2 days ago, but Today Its placed in /lib folder again. I doubt its 0 day openssh exploit. I have installed the snoopy as per the Steven advice, I can see the hacker running the following commands in server. I have parsed the strace output, but I don't see any outbound connections to unknown ips. Researching more about it.... Last edited by mattmackman; 02-18-2013 at 09:27 AM.

Posted by Majester, 02-18-2013, 09:28 AM
I'm curious, has anyone else seen alterations in their /etc/pam.d/sshd files on the compromised machines? In the machines I've found to be compromised, they have very different (and usually matching) files with various modules removed or set to not required.

Posted by The3bl, 02-18-2013, 09:52 AM
Is this your IP? egrep -vi|

Posted by egillette, 02-18-2013, 09:55 AM
No problem -- all I did was write up a simple shell script to remove the files. The real folks to thank would be Steven, serve-you, and others who have spent time investigating the issue. Also, I know it might sound like I'm beating a dead horse, but please keep in mind that my script will only remove the files, it doesn't "fix" the lack of security that may have allowed the compromised file to be implanted in the first place. Currently, none of us are really sure how the file is placed, so once that is discovered, the results will be posted here I'm sure.

Posted by egillette, 02-18-2013, 09:59 AM
Checked Debian machines and Ubuntu/Gentoo machines -- nothing. I think you were correct in your original assessment, that whatever this is, seems to be limited to RHEL/CentOS, et al.

Posted by mattmackman, 02-18-2013, 09:59 AM
No... "" is the IP address of the machine that is ssh'd from

Posted by egillette, 02-18-2013, 10:05 AM
I like this idea, and I suspect others here will too. That said, I will modify my script to do the following: 1) Remove -x permission 2) Create a directory in /root/ called "exploit-evidence-" 3) Move the respective file (32-bit or 64-bit version) to that directory, rather than deleting it. I think your idea was a really good suggestion RRWH, since we can then later call on who has used my script to send copies of the file in the above directory to Steven or anyone else who is researching further to discover its origin. Should have the script modified in 15 minutes, I'll post an update here once I've done that.

Posted by egillette, 02-18-2013, 10:09 AM
Yet another good point. . .I'll mention that in my script.

Posted by tomfrog, 02-18-2013, 10:11 AM
Is tempp a user? local account hack -> kernel vulnerability > root ? why cat /var/log/cron?

Posted by egillette, 02-18-2013, 10:12 AM
I agree with both of you guys. My script is simply designed to check for/remove the file -- it is not designed to fix a lack of security that may have allowed the file to be placed in the first place, although we're not even sure yet how that occurrence has taken place. In the meantime, it might still be "bleeding" per se, but the band-aid is helpful at least, if you know what I mean.

Posted by FastServ, 02-18-2013, 10:18 AM
1.. Wild guess, some code in cron to ensure the server remains compromised. Last edited by FastServ; 02-18-2013 at 10:21 AM.

Posted by mattmackman, 02-18-2013, 10:19 AM
tomfrog No user with the name "tempp". Also its not a local privilege escalation vulnerability, only root user and other default users in this server. So I don't think its a escalation vulnerability.

Posted by egillette, 02-18-2013, 10:30 AM
Ok guys, the script has been modified to address latest concerns. 1) Instead of being deleted, it is now being "quarantined" into a specific directory in /root/exploit-evidence/`date +%h-%d`/ -- I figure that should randomize it enough so that it can't find it's way back out, but more importantly will let anyone running the script be able to provide the specific date that the file was removed, in case it returns, they can see how long it took, etc. 2) I am now recommending that the person reboot their machine, and *then* run yum update to ensure it happens after reboot. Incidentally. . .on my machine specifically, the only file that was still using the library was httpd -- courier, proftpd, etc all started using the symlinked library on their own, and even Apache started using the other library after I restarted it.

Posted by tomfrog, 02-18-2013, 10:33 AM
I read this at cpanel. I thought I should share: SSH 5.3 remote root 0day exploit http://pastebin.com/WgXrebLq

Posted by mattmackman, 02-18-2013, 10:35 AM
Have you tested it?

Posted by tomfrog, 02-18-2013, 10:41 AM
No. I just notified openssh.

Posted by Patrick, 02-18-2013, 10:41 AM
http://www.tamonten.com/how-not-to-exploit-a-box Heh. It's not even a real exploit... it appears to rm -rf your box. Last edited by Patrick; 02-18-2013 at 10:45 AM.

Posted by jalapeno55, 02-18-2013, 10:43 AM
Jeff from cPanel said this is not a real exploit and will probably delete everything if you run it as root.

Posted by mattmackman, 02-18-2013, 10:48 AM
Those are not legitimate exploits. The exploit file contains this, which should be suspicious:

Posted by tomfrog, 02-18-2013, 10:48 AM
Delete everything on the local box... Or to put it another way "/bin","/rm","-r","-f","/"

Posted by Steven, 02-18-2013, 10:50 AM
mattmackman, Can you get me a copy of that new library?

Posted by Patrick, 02-18-2013, 10:53 AM
Ignoring the fact that someone claims a box was compromised with Qmail, I would bet money that it's related to Exim. You have to look at the common denominators: BIND. MySQL. cPanel. Exim. The developers behind BIND really cleaned their asses up after all of the exploits in the early 2000's. Ignoring some DoS and memory corruption attacks, there isn't anything there in any prior change logs to indicate a possible exploit. The same with MySQL, however, there were a couple possibilities if they already had local access to MySQL to do a privledge escalation... but that's unlikely here. As for cPanel, I don't know but my gut is saying no based on what I see for Exim: https://lists.exim.org/lurker/messag...b9147b.en.html Given that there was a ./exim exploit the other year, it's totally plausible that another one exists exploiting the DKIM overflow. Like I told Steven, a lot of exploit writers look at change logs and bug reports for 'flaws' already discovered by someone else or the software's own developers. It's totally possible, almost even likely that someone figured out how to exploit that DKIM vulnerability. ... then again, it could also be OpenSSH but like BIND their developers have really made security a priority so I don't know.

Posted by Steven, 02-18-2013, 10:55 AM
mattmackman, Did you change the root password for the server AFTER you removed it the first time?

Posted by Steven, 02-18-2013, 11:00 AM
I wonder what role this setting has in this...why are they checking to see if it is disabled?

Posted by mattmackman, 02-18-2013, 11:12 AM
Yes, I have changed the root password of the server after removed the library, also I have reinstalled the keyutils-libs. Check you PM box for the library

Posted by FastServ, 02-18-2013, 11:17 AM
1, do not attempt to compile that code. It's a trick.

Posted by Rubas, 02-18-2013, 11:17 AM
Dont use it .. http://forums.cpanel.net/f185/sshd-r...ml#post1324841

Posted by Rubas, 02-18-2013, 11:26 AM
http://cpanel.net/exim-remote-code-e...cve-2012-5671/ The default settings disable DKIM key verification ...

Posted by Patrick, 02-18-2013, 11:33 AM
Then non of this makes any sense! lol.

Posted by RH & Co IT Services, 02-18-2013, 11:41 AM
There's an update for CSF which fixes this issue. See changelog for 5.76 http://www.configserver.com/free/csf/changelog.txt =>Added new LF_EXPLOIT check SSHDSPAM to check for the existence of /lib64/libkeyutils.so.1.9 or /lib/libkeyutils.so.1.9

Posted by chirpy, 02-18-2013, 11:48 AM
It does no such thing. It simply looks for the file in /lib and /lib64 and alerts you if found. You have to investigate further from there.

Posted by Steven, 02-18-2013, 11:58 AM
Chripy thanks for this. Is this a default setting that will get applied on the next update or will people need to enable it for themself.

Posted by chirpy, 02-18-2013, 12:00 PM
It is enabled by default when they upgrade to v5.76

Posted by Steven, 02-18-2013, 12:08 PM
Good to know.

Posted by Majester, 02-18-2013, 12:09 PM
Would you mind utilizing inotify-tools instead of snoopy? I've been told it is better than snoopy as snoopy only shows commands ran whereas inotify runs against the entire machine. https://github.com/rvoicilas/inotify-tools/wiki That is, only if you still have the machine online and they exploit it again after removal.

Posted by Steven, 02-18-2013, 12:13 PM
It would be best to use both. Inotify deals with filesystem events while snoopy monitors the commands.

Posted by Steven, 02-18-2013, 12:15 PM
mattmackman, Was that the 'entire' log?

Posted by mattmackman, 02-18-2013, 12:23 PM
Yes Steven, he is only using that mentioned commands.

Posted by jalapeno55, 02-18-2013, 12:41 PM
Which directories do you have inotify monitoring?

Posted by Cyclon, 02-18-2013, 12:44 PM
Thanks. This is nice addition, but I think that it's not really a great check, since attackers are probably following this thread and will just start linking to a different file like libkeyutils-1.8.so, or any other file. It would be nice if following files could be added to md5sum checks in LFD... /lib/libkeyutils.so.1 /lib64/libkeyutils.so.1 ... so any change in the linked file would send a report. Maybe that is already possible to configure, but I am not sure if there is a way to customize md5sum file list that LFD is checking, or that is hardcoded. Is it maybe enough to just add mentioned files and their md5 hashes to csf.tempint, or that is not a proper way to do it? Thank you.

Posted by ifastnet, 02-18-2013, 01:18 PM
what versions of httpd / what httpd modules are loaded on the affected servers ? httpd -M httpd -V might be interesting.. : edit : sorry just seen Steven ... Another place to touch base, servers with both apache and litespeedtech were hit.

Posted by Steven, 02-18-2013, 01:21 PM
All up2date cpanel servers will have this for the 2.2 branch: Server version: Apache/2.2.23 (Unix) Server built: Jan 9 2013 15:42:47 Cpanel::Easy::Apache v3.16.6 rev9999 Server's Module Magic Number: 20051115:31 Server loaded: APR 1.4.6, APR-Util 1.4.1 Compiled using: APR 1.4.6, APR-Util 1.4.1 Architecture: 64-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/usr/local/apache" -D SUEXEC_BIN="/usr/local/apache/bin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="logs/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" This however, is from an affected server: Loaded Modules: core_module (static) authn_file_module (static) authn_default_module (static) authz_host_module (static) authz_groupfile_module (static) authz_user_module (static) authz_default_module (static) auth_basic_module (static) include_module (static) filter_module (static) deflate_module (static) log_config_module (static) logio_module (static) env_module (static) expires_module (static) headers_module (static) setenvif_module (static) version_module (static) proxy_module (static) proxy_connect_module (static) proxy_ftp_module (static) proxy_http_module (static) proxy_scgi_module (static) proxy_ajp_module (static) proxy_balancer_module (static) ssl_module (static) mpm_prefork_module (static) http_module (static) mime_module (static) status_module (static) autoindex_module (static) asis_module (static) info_module (static) suexec_module (static) cgi_module (static) negotiation_module (static) dir_module (static) actions_module (static) userdir_module (static) alias_module (static) rewrite_module (static) so_module (static) bwlimited_module (shared) cloudflare_module (shared) php5_module (shared) Syntax OK

Posted by ifastnet, 02-18-2013, 01:30 PM
do all affected servers allow local access to untrusted users ? (just trying to narrow down possible vectors)..

Posted by matbz, 02-18-2013, 01:36 PM
Trying to catch up with this lengthy thread. Were all those compromised running cPanel?

Posted by ifastnet, 02-18-2013, 01:39 PM
directadmin is also mentioned in the thread as being used on a comp'd system .

Posted by jalapeno55, 02-18-2013, 01:42 PM
Plesk was mentioned as well.

Posted by tomfrog, 02-18-2013, 02:00 PM
Steven, have you seen a plesk or directadmin server hacked? Is there any forum activity at directadmin or plesk regarding hacked servers? http://forum.directadmin.com/showthr...ghlight=hacked nothing at plesk When can we say that plesk and directadmin have also been hacked? If I were to report a plesk server, why should you consider my report valid? I don't have more than a few posts. Any non red hat servers hacked?

Posted by Steven, 02-18-2013, 02:07 PM
I have personally only seen Cpanel servers. Eric has seen some plesk and direct admin

Posted by jestep, 02-18-2013, 02:11 PM
Just finished reading through this whole thread. Conclusion based on comments here is that it's red hat only, and both 32 and 64 distributions. There are reports of plesk, cpanel, directadmin, and ISP config control panels but what might be considered an actual confirmed report is probably debatable. I'm just going to continue to watch because red hat seems to be the only identified commonality. Of my 3 red hat servers, none of them have been infected. Running Centos - 5.9, and 2 with 6.3 all 64 bit.

Posted by aeris, 02-18-2013, 02:20 PM
Out of curiosity, I went looking for this file on the ~40 servers I have, and none of them had it. They are a mixed bunch of 64-bit CentOS 5/6 with installations up to two years old. No panel, no other local users than me, and the only services exposed to public network are Apache, Varnish, nginx and sshd - so I doubt it's any of those.

Posted by Orien, 02-18-2013, 02:24 PM
I wrote up a quick thread summary at the beginning of the thread... let me know if I was completely off-base or if anything else in the thread should be added. Thanks!

Posted by Ceetoe, 02-18-2013, 02:35 PM
Perhaps someone can throw up some kind of web/wiki page(github ?) so the noise on the thread can be reduced? As the forum comments increase, it is difficult for readers(old and new) to summarize what has been found so far. We're still looking for common denominators here folks. Concerned members here should be worried about spamming threads which is a common tactic to dilute focus or create reader fatigue. Veracity of compromises by anonymous users(your trust factor should be almost non-existent right now) is difficult to sort through. How do we address that? Companies or organizations are welcome to verify here. Steven shared an observation some pages back that to me is troubling: "They are not logging in with root, nor are they even spawning a bash process." Is this still accurate? If so, then I'll play the devil's advocate and disregard if irrelevant or only also adds noise: Are we now worried about build environments where we're getting things? I'm not talking about package managers necessarily but even so far as repositories and/or build environments. Have we confirmed a RHEL compromise? How clean are Centos kernels? I understand the intent of Centos to be 100% binary compatible but...? Last edited by Ceetoe; 02-18-2013 at 02:43 PM.

Posted by ifastnet, 02-18-2013, 02:42 PM
uid:0 shows root user .

Posted by Steven, 02-18-2013, 02:44 PM
Initially the servers I saw were not spawning bash during spam. They are now. A few pages back I confirmed it with snoopy. Away from my desk atm. Will try to answer the rest of the questions when I get back Last edited by Steven; 02-18-2013 at 02:48 PM.

Posted by kaniini, 02-18-2013, 02:45 PM
That is fake, it invokes /bin/bash on the local system using system(), and makes bogus claims like "requiring root to use raw sockets", which it does not use raw sockets.

Posted by ifastnet, 02-18-2013, 03:15 PM
synopsis : 1) uid 0 on ssh is gained by attacker (before or initial exploit?) 2) a shell // tty is gained by attacker as uid 0 3) attacker alters pam.d/sshd to allow easy re-access to ssh 4) attacker injects a replaced lib that is linked to sshd to callback pasword changes. 5) the attacker seems very interested in the verion of 'redhat' , and the UseLogin option in sshd (which is widley know from 200x to have a previous exploit) http://www.sans.org/security-resourc...h_uselogin.php (are the sshd configs being altered / any new local users being added ???) 6) only redhat derivatives are affected 7) as we dont know 'how' the initial vector occured, then any affected box should be considered comprimised /rootkit'd and should be isolated sooner rather than later, any packet captures / ram dumps from affected servers could be very usefull .

Posted by bloodyman, 02-18-2013, 03:23 PM
When I search google for libkeyutils.so.1.9 I have found some evidence that this lib exist on servers without symlink patch. Also there are people who ask about this libkeyutils.so.1.9 since January 2013

Posted by matbz, 02-18-2013, 03:30 PM
This made me feel very uncomfortable... On a cPanel/Centos server with cPHulk Brute Force Protection disabled. And does not have the libkeyutils.so.1.9 file present.

Posted by M Bacon, 02-18-2013, 03:35 PM
Possible root compromise: File /lib64/libkeyutils.so.1.9 exists. For more information see: http://www.webhostingtalk.com/showthread.php?t=1235797 The file says that it's been on my system since libkeyutils.so.1.9 33.8k 22-Jun-2012 02:20:37 root(0)/root(0) 755 delete rename copy move June 22nd of 2012?

Posted by SoftDux, 02-18-2013, 03:35 PM
Ok, so it's probably safe to say the hacker now simply use another file name.... which also means all automated scans for the original file name will be void soon.

Posted by jalapeno55, 02-18-2013, 03:40 PM
I believe that file is normal, I have it as well:

Posted by Patrick, 02-18-2013, 03:43 PM
Yeah it's legit. I think people need to stop grasping at straws and coming to conclusions without knowing what they are looking at to make such a conclusion.

Posted by matbz, 02-18-2013, 03:45 PM
If so, what is your output of strings /lib64/security/pam_hulk.so please?

Posted by ifastnet, 02-18-2013, 03:46 PM
Pam_hulk is fine that is cpanel's brute force module, which is not installed via yum so will give a warning.

Posted by Patrick, 02-18-2013, 03:47 PM
What's interesting is that the RPM was built on that day: http://rpmfind.net/linux/RPM/centos/....el6.i686.html

Posted by Patrick, 02-18-2013, 03:47 PM
Same as yours: IO7+ __gmon_start__ _init _fini __cxa_finalize _Jv_RegisterClasses pam_hulk_version openlog vsyslog closelog pam_get_item readline fileno read feof get_hulk_key fopen fclose disconnect_hulk memset sigfillset sigaction alarm socket memcpy connect snprintf strlen write tell_hulk strncmp hulk_enabled pam_sm_authenticate pam_get_user time pam_sm_setcred libc.so.6 __xstat libpam.so.0 _edata __bss_start _end LIBPAM_1.0 GLIBC_2.2.5 ATSubH cphulk_version_2.7 PAM-hulk timeout while connecting to cphulkd /var/cpanel/cphulkd/keys/pam error getting hulk key error setting sigalrm opening socket /var/run/cphulkd.sock failed to connect stream socket error reading welcome from stream socket Unexpected welcome messge from cphulkd: %s AUTH pam %s error reading auth status from stream socket error logging into hulkd: %s PAM_AUTHENTICATE system %s %s 1 %ld 0 0 %s PAM_SETCRED system %s %s 1 %ld 0 0 %s err writing to stream socket error reading PAM_? from stream socket Brute force detection active: %s /var/cpanel/hulkd/enabled /var/cpanel/cphulk_enable crond

Posted by iseletsk, 02-18-2013, 03:49 PM
I think pam_hulk is part of cPanel hulkd, and not a hack at all.

Posted by tomfrog, 02-18-2013, 03:50 PM
I believe that file exists on previous hacked servers. Please confirm.

Posted by matbz, 02-18-2013, 03:51 PM
thnx, confirmed on 6 other boxes.

Posted by TravisT-[SSS], 02-18-2013, 03:53 PM
We can confirm that the pam module used by CPanel is not been exploited. We have the same strings match up on a clean box not being used to do spam nor have we seen any activity hinting to an exploit. I think it would be safe if we don't jump to conclusions here without posting verbose logs from snoopy ect.. to backup findings. On a sidenote, I think it would be a good idea if everyone was running snoopy till we find the root cause. On a random note: I think it is safe to say though we should expect the attackers to start randomizing the file at some point to attempt to remain more hidden or change their tactics up some. Last edited by TravisT-[SSS]; 02-18-2013 at 03:58 PM.

Posted by kaniini, 02-18-2013, 04:02 PM
pam_hulk.so is installed regardless of it being enabled or not. that setting is enforced elsewhere in the stack (specifically by hulkd not running).

Posted by Cyclon, 02-18-2013, 04:04 PM
mattmackman Did you check the cron log? Attacker checked the cron log 3 times like he was waiting for something to happend in that specific hour... # cat /var/log/cron | egrep -i Feb 18 07 What was it?

Posted by Steven, 02-18-2013, 04:04 PM
Normal file, part of cpanel cpkhulkd bruteforce protection. It is NOT part of a rpm.

Posted by SoftDux, 02-18-2013, 04:14 PM
Can I add: If the server doesn't have the original /lib/libkeyutils.so.1.3 or /lib64/libkeyutils.so.1.3 (CentOS 5.9 has /lib/libkeyutils.so.1.2?) then SSH, FTP, wget, in fact anything that requires this login library will fail and the server owner won't be able to login to his server again. I haven't tested this with console access but remote access will be "broken". So your script may want to check for the original file first and probably re-install the keyutils-libs-1.2-1.el5 or (corresponding for the OS in question) library.

Posted by LeadDogGraphicStudio, 02-18-2013, 04:17 PM
Hey Travis, so far our servers seam ok, no hacks, but I do have one server on CentOS 5.9 x64 that SSH wont respond, and wont restart, although the service shows as up and running in WHM under service status, any thoughts on that? Adding snoopy sounds like a good idea, any special instructions in setting it up, or is it an install and it works sort of thing? Also I wanted to thank the community of brilliant minds here working on this as well as the help of the cPanel, Cloud Linux and CSF staff, but my question is where the f*#k is Red hat on this? It seams like it has been narrowed down to their distro only and they are AWOL in these discussions.

Posted by iseletsk, 02-18-2013, 04:19 PM
I cannot blame RedHat here, as I am pretty sure no one here licenses OS directly from them. CloudLinux & CentOS are not RHEL. So, we cannot really blame them for not acting.

Posted by LeadDogGraphicStudio, 02-18-2013, 04:29 PM
I didn't blame them, I thanked them for being here. I asked where was Red Hat in this, you are right, no one licenses from them, however they are the ones capable of plugging the hole, so yes they need to be involved.

Posted by Patrick, 02-18-2013, 04:36 PM
The PTRACE flaw is probably unrelated to all of this, and as mentioned earlier, for that flaw to work there has to be ideal timing conditions within the kernel and it doesn't appear to be easily exploitable. Yes it's a flaw, but it doesn't look like a priority right now based on the degree of difficulty to make it work... so I kind of understand RedHat not dropping what they are doing to fix it.

Posted by Ceetoe, 02-18-2013, 05:24 PM
Confirmation(yea or nay) if all compromised systems were serving(not just installed) PHP?

Posted by Patrick, 02-18-2013, 05:31 PM
Why does it matter? PHP alone isn't going to lead to a root compromise.

Posted by NetworkPanda, 02-18-2013, 05:51 PM
Congratulations to the CSF team for the fast response, but CSF 5.76 caused us huge network issues, although our servers were absolutely clean and not infected by this exploit. It was blocking network connections from IP addresses not existing in the firewall rules, rendering our DNS cluster unusable, bringing down our billing and monitoring systems and blocking traffic from a lot of visitors. Fortunately we had kept the installation files of version 5.71 so we uninstalled the new version 5.76 and reinstalled 5.71. Sorry but we will not use CSF for scanning the exploit discussed in this topic. Rendering an entire network of servers unreachable to 50% of the world, to protect from an exploit is not a solution. Thanks Last edited by NetworkPanda; 02-18-2013 at 06:03 PM.

Posted by Steven, 02-18-2013, 05:52 PM
They released a new one that appears to have fixed it.

Posted by Ceetoe, 02-18-2013, 05:59 PM
It may not matter. Are you vouching for the correct configuration on all compromised servers? I'm certainly not just as I would never vouch that there are no vulnerabilities with installed packages. At this stage we just don't know.

Posted by NetworkPanda, 02-18-2013, 06:06 PM
After what happened, we will not take the risk with any new CSF version, we will keep 5.71 which has been working fine for us for ages. Let's hope their CXS malware scan tool gets updated soon to scan for the exploit, as we are also using it and it is proved to be better than CSF when used for malware detection. Last edited by NetworkPanda; 02-18-2013 at 06:11 PM.

Posted by Patrick, 02-18-2013, 06:14 PM
It doesn't matter, no vulnerability in PHP alone is going to lead to a root compromise therefore making it 100% irrelevant.

Posted by Dathorn-Andrew, 02-18-2013, 06:15 PM
Your issue with CSF had nothing to do with the malware scanning portion of it, it just happened to be included in the same update. The problem was the attempt at switching to conntrack which has since been effectively reverted in 5.78. Obviously it would have been better had this not occurred but refusing to update going forward is a bit much. If anything, disable automatic updates and test them in a pre-production environment before deploying them.

Posted by Cyclon, 02-18-2013, 06:23 PM
PHP alone probably not, but what about suPHP? Is there any compromised server that was not running suPHP?

Posted by servermanaged, 02-18-2013, 06:24 PM
Hello folks. I'm lucky, I manage 40+ servers and no one is affected by this issue. Thanks to Steven for full disclosure about this exploit. Curiosity: While searching on Google for libkeyutils.so.1.9 I've noticed a website that came with a symlink to /lib64: flyulendo.com/sym/root/lib64

Posted by spendergrsec, 02-18-2013, 06:25 PM
Hi guys, I've taken a preliminary look at two backdoor samples that were provided to me a couple minutes ago. You're missing out on a considerable amount of information by just running strings against the binary. The versions of the backdoor I received are presumably 1.0.4 and 1.2.1. They use a simple obfuscation of XORing each byte of a string with 0x81. Here's a simple program you can run against the binaries to naively deobfuscate the strings (it's suitable for running strings against the output): The code makes use of shared memory (IPC) with a built in key, perhaps unique to each version. The size of the shared memory varies between version as well. Strings against the "deobfuscated" binaries I've received produce: Will update as I find more. -Brad Last edited by spendergrsec; 02-18-2013 at 06:26 PM. Reason: formatting

Posted by egillette, 02-18-2013, 06:27 PM
That's a good thing. . .I guess it's also good that all my boxes and client boxes are set to update CSF/LFD automatically.

Posted by Patrick, 02-18-2013, 06:32 PM
With the way the PHP handlers work, I can't see any realistic attack vectors that would allow someone to get them to execute commands as root. A flaw in a daemon is much, much more probable at this point. (Sure - anything is possible, but it's extremely unlikely to be related to PHP and it's handlers.)

Posted by Patrick, 02-18-2013, 06:33 PM
That's the well documented symlink exploit on a server that is already backdoored. Nothing really there that's helpful outside of looking at what else they have to determine what OS + services are installed.

Posted by Ceetoe, 02-18-2013, 06:35 PM
Good to have you here and look forward to your input!

Posted by weredigital, 02-18-2013, 06:48 PM
I have benn monitoring this all day.I am finding very conflicting info and advice. Should We be doing any thing at this point or waiting for a solid cause/fix ? I complete re install is pointless until the fix is in place Im assuming ? The fixes here are re breached almost immediately to my understanding I believe waiting on the fix is my only option at this point. Or does any one have any other advice at this moment

Posted by Patrick, 02-18-2013, 06:53 PM
Outside of normal security precautions there isn't anything any of us can tell you to safe guard against this. We're all just speculating. The only thing 'extra' I would suggest is to lock down SSH and limit connections to particular IP addresses but as mentioned, who knows... it could be something else.

Posted by weredigital, 02-18-2013, 06:55 PM
that's what I was assuming thanks will keep monitoring

Posted by Ceetoe, 02-18-2013, 07:11 PM
FYI...I've submitted this to the Netsec subreddit for additional eyes on this. http://www.reddit.com/r/netsec/comme.../sshd_rootkit/ Also for members not aware, user spendergrsec is Brad from grsecurity . I'm sure others will be joining in as well. Much appreciation to all giving their time to look into this.

Posted by egillette, 02-18-2013, 07:38 PM
Just a note to let everyone know. . . The file is BACK on my Plesk 9.5.x box. I'm going to delete it and then touch the file + chmod 000 the file, and then I'm going to chattr +i on it as well to see what happens.

Posted by Steven, 02-18-2013, 07:43 PM
Plesk 9.5.x has had massive vulnerabilities. It also is lacking security enhancements as to how passwords are handled that 10.x and 11.x has. You can also root a plesk server by setting up a root cronjob via the admin panel. Because of that I'm not confident that plesk is vulnerable to the same exploit if its not latest updated.

Posted by LeadDogGraphicStudio, 02-18-2013, 07:47 PM
Steven, were you able to de-obfuscate any of your samples based on the instructions a page back by brad from GRSec?

Posted by Steven, 02-18-2013, 07:58 PM
Not yet. The samples he de-obfuscated were libraries I sent him. I am going to do some more soon, checking out some servers right now + other work.

Posted by egillette, 02-18-2013, 07:58 PM
Steven. . . It *just* happened on my Plesk 11 box too. It also happened on 2 of the 64-bit cPanel boxes as well. I am taking the steps on the files I mentioned previously so at least they cannot come back. I'm modifying the script I wrote up to do the same, since this is obviously something makes itself come back.

Posted by M Bacon, 02-18-2013, 08:03 PM
Well, whatever you do don't remove the file. It will seriously mess your server up. Reinstall the required modules instead.

Posted by egillette, 02-18-2013, 08:12 PM
SoftDux, yes that's possible, but I've confirmed CentOS 5.x also contains the same library in at least 98% of instances, so we should be ok. Also, for anyone that has run my script previously, I've updated it 3-4 times over the last 24 hours as new evidence discovered by Steven, myself and others has come to light. The latest version is here: http://www.ericgillette.com/clients/exploit-cleanup To execute the script, perform the following actions after logging in to your server with root priveleges, or after using sudo or su to gain root privileges: The script will work on both 32-bit and 64-bit systems. Disclaimer: You should reboot your system after running the script, and then run yum update all afterwards as well. In addition, keep in mind that evidence from this thread suggests that the attackers may simply rename the file, or use a file with a different name, so please keep in mind my script is only designed to help remove the original file in question, and *possibly* prevent it from coming back (since we don't know how it was put in place yet). Take care folks. . .I'll continue to post my findings as time wears on -- this proves very elusive, but we do know that with SSH firewalled off, they don't seem to be able to connect to SSH in the first place at least.

Posted by ErnieQ, 02-18-2013, 08:12 PM
Hey Eric : I see you're trying what our tech Kris came up with and posted at Reddit. Let me know how it goes. Anyone who's interested in the script: http://hudsonvalleyhost.com/stoplib - You shouldn't have problems with 'seriously messing your server up' as we haven't seen any servers where the legitimate module uses this file name.

Posted by Steven, 02-18-2013, 08:36 PM
ernie, You need a ldconfig otherwise your going to symlink'd to a illegitimate file after you touch it empty. /lib64/libkeyutils.so.1 is linked to /libkeyutils.so.9 for example

Posted by TravisT-[SSS], 02-18-2013, 08:39 PM
Just checked reddit. This was found: CVE mentioned: https://bugzilla.redhat.com/show_bug...=CVE-2012-5671 EXIM from on a server: root@server25 [~]# exim --version Exim version 4.80 #2 built 13-Nov-2012 13:23:41 Not sure if this is related or not.

Posted by Steven, 02-18-2013, 08:41 PM
That ip was found by brad @ grsecurity already and posted. That vulnerability is not present according to http://cpanel.net/exim-remote-code-e...cve-2012-5671/

Posted by ErnieQ, 02-18-2013, 08:45 PM
Noted. I'll past this on to kris. This should immunize the server, if something's symlinked, it should be reinstalled once the file's removed. This simply wipes the file and puts a read-only in place so it can't reinfect the machine, if it's being done with automation & with a static filename, as it seems to be doing.

Posted by jalapeno55, 02-18-2013, 08:51 PM
Anyone know what that audit command/script is?

Posted by Patrick, 02-18-2013, 08:58 PM
http://www.webhostingtalk.com/showpo...&postcount=284 ./audit input output

Posted by Patrick, 02-18-2013, 09:00 PM
Here is a strings output from a compromised system I randomly came across: The biz / info / net has me curious...

Posted by jalapeno55, 02-18-2013, 09:10 PM
That a different version # than the one posted on reddit:

Posted by Patrick, 02-18-2013, 09:22 PM
Probably an exploit pack... which makes me think there is more to it than that individual lib but what...

Posted by Steven, 02-18-2013, 09:34 PM
IDK.. There is variations in the strings. Eric has a library that does not do outbound port 53. I suspect pretty soon we may have a new version, that calls home to a new IP soon.

Posted by Steven, 02-18-2013, 09:39 PM
Any of you that have compromised machines using, ASL (atomic security linux) with the grsecurity kernel?

Posted by LeadDogGraphicStudio, 02-18-2013, 09:39 PM
This is just an idea but that maybe from them using various URL's to replace the IP they have been using, if they use an array of different URLs it will allow them to phone home using any one of them.

Posted by mattmackman, 02-18-2013, 09:43 PM
Steven, No. Did you get my exploited library?

Posted by Steven, 02-18-2013, 09:44 PM
Yes I did. Anyone running a NON stock centos / cloudlinux kernel?

Posted by TravisT-[SSS], 02-18-2013, 09:46 PM
We have vanilla kernels patched with GRSec if that means anything.

Posted by TheVisitors, 02-18-2013, 09:46 PM
Well doesn't it figure.... I install a fresh copy of CentOS 6.3 with cPanel current and in less than 2 hours.... BAM I had changed Ssh ports and made a pass code over 400 letters, numbers, and symbols long.... Still got despite all this. Little "tool" keeps changing my ssh log-in now. Can't turn it off, we need ssh.

Posted by LeadDogGraphicStudio, 02-18-2013, 09:51 PM
can any hosting providers setup a couple honeypot boxes with different settings to see maybe which ones get hacked and which ones do not?

Posted by Steven, 02-18-2013, 09:51 PM
Is there anything on the server or is it plain? as in no accounts?

Posted by TheVisitors, 02-18-2013, 09:53 PM
I just spent a few days trying to properly restore my site because we moved host. So yes, my site is there. Site just went live about an hour ago.

Posted by matbz, 02-18-2013, 09:54 PM
Did you try disabling password authorisation and use ssh keys (delete the private key once downloaded)? Just wondering if this is happening on boxes that have password authorisation enabled (even if still using keys).

Posted by Steven, 02-18-2013, 09:54 PM
were they compromised?

Posted by TravisT-[SSS], 02-18-2013, 09:55 PM
We haven't seen any compromises on them at this time. EDIT: We also run Debian. No compromises on them either. Last edited by TravisT-[SSS]; 02-18-2013 at 10:02 PM.

Posted by TheVisitors, 02-18-2013, 09:58 PM
No keys. Just a very long pass code. Was the pass word (no longer)

Posted by matbz, 02-18-2013, 10:02 PM
On all our boxes running cPanel/Centos we have disabled password auth, use keys only, deleted private keys from servers and not a single compromise as detailed here. I think password auth is the issue imho.

Posted by TheVisitors, 02-18-2013, 10:03 PM
Thats going to be a problem for me. Will wait until there is a fix until I attempt to clean up this mess.

Posted by Steven, 02-18-2013, 10:07 PM
Seems odd that they are changing your password now..

Posted by lbeachmike, 02-18-2013, 10:08 PM
Does anybody know of reports of compromised boxes that had SSH completely uninstalled? In other words, let's say cpanel with WHM and SSH entirely disabled ... Also, for an already-compromised box, is it known if disabling SSH eliminates the bad behavior? Thanks for everybody's efforts on this.

Posted by TheVisitors, 02-18-2013, 10:13 PM
If it wasn't for my VPS Control Panel ..... I'd be screwed. I've found myself "locked out" a few times now. I change it again. Log-in.... log out.... find that I need to change it again.

Posted by spendergrsec, 02-18-2013, 10:16 PM
Hi guys, I was busy for a bit (dinner, etc). Looking further into the backdoor, it's doing GOT modifications of the sshd it gets loaded into in order to hijack particular functions. For instance, it hooks: syslog __syslog_chk write audit_log_user_message audit_log_acct_message Presumably in response to receiving a specific password/username, for the one backdoor this is "XXXdYZulavB", it will temporarily disable logging via syslog, __syslog_chk, audit_log_user_message, and audit_log_acct_message. The write() hook will also prevent logging via stderr. The two I looked at were sending login credentials to UDP port 53 on I need to investigate further to see what exactly that includes, but it's clearly at least sending the login name/UID, and hostname connected to. The attacker has three commands available: Xver, Xcat, and Xbnd. Xver displays the backdoor version, Xbnd causes the connect() hook in the backdoor to bind to a specific address before performing the connect. The Xcat command involves the shared memory in some way. Of interesting note is that the backdoor would crash as-is with the PAX_MPROTECT feature in grsecurity enabled. If the system wasn't enforcing PaX flags with RBAC, they could just disable the feature on sshd, however. For code hooking in several locations, the region involved has its protections changed to read/write/execute -- something disallowed on a grsecurity kernel and optionally logged. The write following the RWX mprotect would fail, causing a crash of sshd. If anyone has a 32bit version of the backdoor they could mail me, it would speed up analysis a bit as I'm doing it all statically. -Brad

Posted by Steven, 02-18-2013, 10:16 PM
Do you have /lib64/libkeyutils.so.1.9 on the box? If so can you get it to me asap?! We need to know if they changed the call back to port 53.

Posted by Adam-AEC, 02-18-2013, 10:23 PM
Here is one that Eric posted a few pages ago.

Posted by jalapeno55, 02-18-2013, 10:26 PM
Preferabily with a different single port open on each.

Posted by MaB, 02-18-2013, 10:27 PM
Does anyone with a hacked server have 'PasswordAuthentication no' set in /etc/ssh/sshd_config?

Posted by jalapeno55, 02-18-2013, 10:28 PM
Do you have SSH limited to only specific ips?

Posted by TheVisitors, 02-18-2013, 10:29 PM
My IP changes often.... Daily sometimes hourly.... So thats not possible.

Posted by jalapeno55, 02-18-2013, 10:31 PM
You can restrict it to just a block of IPs.

Posted by TheVisitors, 02-18-2013, 10:32 PM
I don't stay within any single block.

Posted by bloodyman, 02-18-2013, 10:35 PM
Did you read entrie of this thread? There were information that servers without SSH enabled was also hacked. I don't think it is related to password authentication. Using long password + host access control + ssh on different port + disabled root login = good security for SSH.

Posted by MaB, 02-18-2013, 10:35 PM
I would suggest port knocking. But that's off topic. Are there any non-RHEL/CentOS boxes that have been compromised? Maybe it's time to start digging through the RedHat openssh SRPMs and check out the dozens of patch files that redhat applies to the openssh source ? That would explain why we see problems with RHEL/CentOS 5/6 but not other distros running openssh

Posted by matbz, 02-18-2013, 10:38 PM
Actually yes I have! And I haven't found a single reference to password auth being disabled prior to a compromise as detailed here. If I've missed it, please point it out

Posted by bloodyman, 02-18-2013, 10:40 PM
Steven - can you confirm that only boxes with password authentication were hacked?

Posted by matbz, 02-18-2013, 10:42 PM
I forgot to mention, all boxes have port 22 closed so ssh access is allowed by IP address. Anyway, the question for us is what is creating the file more than what it does right now.

Posted by NetworkPanda, 02-18-2013, 10:46 PM
The fact that systems where the default SSH ports are closed or restricted to specific IP addresses or the SSH password is too large to be guessed, are still being infected, means that the exploit gets in the system using a vulnerable service other than SSH? Just thinking... Maybe posting which versions of Exim and Bind do the infected systems run, could help?

Posted by Steven, 02-18-2013, 10:47 PM
So I am sitting here thinking, and I am positive I am going to get a bunch of people disagreeing but it has not been said. How many of these hacked servers that you encountered were customer servers? How many do only YOU access? Do you have staff? Before you guys dismiss this.. think really hard on this one. 1.) Flash 2.) Java 3.) Acrobat Reader What do these all have in common? Exploits in the last month. Food for thought, how secure are you personally.. do you reuse passwords? The FIRST server I encountered had a non standard ssh port..so either they are looking for that specifically or they already know your port via one of those exploits. And for those of you thinking.. yeah but what about ip restricted servers.. one phrase -- back connect best part is... would look like you doing it. Last edited by Steven; 02-18-2013 at 10:54 PM.

Posted by matbz, 02-18-2013, 10:53 PM
My next question was going to be... what are those with infected boxes using to connect via SSH? Because our worry is - what is creating the file?

Posted by jalapeno55, 02-18-2013, 10:53 PM
Good point. For the server getting hacked repeatedly it may be a good idea to create a blank file and then watch it with auditd: (Someone double check my syntax pls) Last edited by jalapeno55; 02-18-2013 at 10:57 PM.

Posted by TheVisitors, 02-18-2013, 10:57 PM
To connect to ssh I was using 2 things putty filezilla (for sftp)

Posted by Steven, 02-18-2013, 10:58 PM
On top of this -- think about the rootkit.. it was sending your password (suspected) to the 3rd party. What if you cleaned your computer.. it would be a second way back in.

Posted by spendergrsec, 02-18-2013, 10:59 PM
Have you tried: auditctl -w /lib64/libkeyutils.so.1.9 -p w -k backdoor auditctl -w /lib/libkeyutils.so.1.9 -p w -k backdoor then: ausearch -w /lib/libkeyutils.so.1.9 ausearch -w /lib64/libkeyutils.so.1.9 -Brad

Posted by jalapeno55, 02-18-2013, 11:02 PM
Will try that thx. Last edited by jalapeno55; 02-18-2013 at 11:05 PM.

Posted by mattmackman, 02-18-2013, 11:06 PM
the lates libkeyutils.so.1.9 is not creating any outbound to unknow ips, so they might be changed the script.

Posted by matbz, 02-18-2013, 11:07 PM
filezilla please nooooooooooo lol, sorry had more clients with problems with filezilla than it has rained on me (well almost). Do you log in as root with Filezilla? WinSCP is good We use Putty and WinSCP but we are using a fresh installation on a single machine (it's like 2 weeks old) so all software was only installed between then and now. It wouldn't the be the 1st, 2nd, 3rd, etc, etc time that a piece of software has been compromised with an exploit.

Posted by The3bl, 02-18-2013, 11:10 PM
How many of these servers hacked had ruby on rails running?

Posted by dclardy, 02-18-2013, 11:11 PM
Do all of these boxes have mod_ruby enabled? With all the issues that Ruby had last month and their repos being hacked, it might make sense for it to be so wide spread.

Posted by mattmackman, 02-18-2013, 11:12 PM
Yes its running on my server..

Posted by Ceetoe, 02-18-2013, 11:12 PM
Excellent points Steven. Solid shell's blog (http://blog.solidshellsecurity.com/2...yutils-so-1-9/) mentioned seeing the earliest exploit in late December. Does that affect the plausibility for one of the 3 exploits you mentioned?

Posted by Steven, 02-18-2013, 11:13 PM
To add to my theory. I have 1 customer with 4 compromised servers, out of all the servers I manage. The first server was compromised: Change: 2013-01-17 06:23:59.000000000 +0000 Provided me with an updated password list on the 15th, and each other server was compromised after that date a few days apart.

Posted by dclardy, 02-18-2013, 11:13 PM
We are thinking the same thing. I went back to read Steven's Apache stack post before I posted it. I didn't see anything in there about it though. It just seems really odd with all the Ruby trouble that this is happening now. Ruby Gems were hacked January 30th.

Posted by Steven, 02-18-2013, 11:13 PM
Not really.. all 3 softwares have horrible track records over the last year.

Posted by Steven, 02-18-2013, 11:14 PM
I pursued this.. Nothing.

Posted by Ceetoe, 02-18-2013, 11:16 PM
So maybe "TheVisitors" are you updated on your Windows box? You seem a good candidate to explore since you mentioned just having a fresh install with an immediate compromise thereafter.

Posted by Steven, 02-18-2013, 11:16 PM
Food for thought: http://www.cvedetails.com/vulnerabil...sh-Player.html

Posted by Ceetoe, 02-18-2013, 11:17 PM
Yeah, just the mention of Adobe gives me nausea.

Posted by TravisT-[SSS], 02-18-2013, 11:18 PM
I've considered what Steven mentioned about the exploits being local. I think this could be very possible. In fact, we ourselves have seen local businesses call us in when their boxes got destroyed from fake adobe files so this very well could be true. As far as our blog post goes, it was more of a summary of this thread taking bits and pieces of the major stuff to use to share with some professionals and get some others to check their servers to help us narrow down some entry points.

Posted by NetworkPanda, 02-18-2013, 11:20 PM
putty is fine but stay away from FileZilla It stores all passwords in plain text, unencrypted (even the SSH/SFTP passwords) in its bookmarks and sitemanager settings file. Anyone who can read files on your system (locally or remotely via an exploit) can get your root password if you connected to your server using Filezilla and SFTP.

Posted by Steven, 02-18-2013, 11:23 PM

Posted by jalapeno55, 02-18-2013, 11:25 PM
To update Brad's command could anyone that's been hacked please run: If your on a 32-bit system remove 64. Then to check it afterwards, if you've got rehacked run: Last edited by jalapeno55; 02-18-2013 at 11:28 PM.

Posted by matbz, 02-18-2013, 11:27 PM
That's really interesting because on a few of our boxes several files in /lib* were updated that day during an overnight cPanel update. On that box that you refer to right there, what time is cPanel Update set to run?

Posted by Hellsheep, 02-18-2013, 11:31 PM
Hey guys, I just wanted to chime in here. I have been reading this thread from the start and decided to create an account to discuss the issue with. I have a box which is compromised. The server is running the following kernel version: 2.6.32-279.14.1.el6.x86_64 It's a CentOS 6.3 Box. Confirmed with lsof | grep libkeyutils that the /lib64/libkeyutils.so.1.9 library appears to be being used by sshd, python, cpdavd, cpsrvd-ss, mailmanct however pure-ftpd and pure-auth are still okay with libkeyutils.so.1.3 libkeyutils.so.1 has a symlink to the 1.9 one. I only ever log into the box myself, it's got cPanel/WHM latest stable build 11.34.1 (build 7) I SSH to the server using keys, however password auth was enabled still but never used. The change date on the file is 10-12-2012. I have firewalled the IP address earlier in this thread as when I did a strace on sshd saw a connection to it. I have also disabled password auth now as it's not really needed for me. I have not seen any spam go out at all from the server. I also have a fully patched windows 7 box which I connect to it from using PuTTY. Adobe Reader and Flash and java I check manually for updates daily on this win 7 computer. If you guys need me to test anything let me know happy to do what I can to help. Also running auditctl -w /lib64/libkeyutils.so.1.9 -p war -k backdoor now.

Posted by Steven, 02-18-2013, 11:34 PM
Some exploits have been out for a long time before patching by vendor

Posted by matbz, 02-18-2013, 11:35 PM
May I ask is this a dedicated server or VPS? If I am right on something then it is a dedicated server.

Posted by Ceetoe, 02-18-2013, 11:36 PM
Follow-on idea: Perhaps we can have someone donate a tester box or vps for TheVisitors to do a fresh install. This might help substantiate a possible local pc compromise?

Posted by jalapeno55, 02-18-2013, 11:36 PM
Thanks! That file should be removed though as well, if you haven't already. Could someone list the proper steps for that?

Posted by Hellsheep, 02-18-2013, 11:36 PM
Sorry matbz, it's a VPS. Also mod_ruby is enabled. I haven't removed the file yet actually. Would appreciate someone listing the appropriate steps as I know a lot is going on and there was a script a few pages back to remove it. Just want to confirm what the most updated steps to remove it are. Last edited by Hellsheep; 02-18-2013 at 11:40 PM.

Posted by Steven, 02-18-2013, 11:40 PM
They may be looking at specific IP so it may not be instant.

Posted by Ceetoe, 02-18-2013, 11:43 PM
Thanks for joining and welcome! Do you log in to this box ONLY from that Windows 7 computer?

Posted by Hellsheep, 02-18-2013, 11:45 PM
That's correct Ceetoe. It's never ever logged into by anything except that Windows 7 machine at home.

Posted by dclardy, 02-18-2013, 11:45 PM
I have been searching around on that IP to see what I can find, and the pmadison12@gmail.com account has a lot of unusual domains attached to it. Here are all the IPs that I could find. Host: fx-in-f99.1e100.net That same email address can be found on several other whois with a name of Peter Madison. Info at this address: http://www.affiliateguarddog.com/for...-t3782-p2.html There are a bunch of random letter domain names that come up as well, but I doubt this helps much.

Posted by Gogax | Simon, 02-18-2013, 11:48 PM
Those are the IP address used to call home? If so, i think it would be a good "temporary" solution to nullroute these ip from our networks!

Posted by Ceetoe, 02-18-2013, 11:48 PM
1e100.net is google.

Posted by matbz, 02-18-2013, 11:58 PM
I think, two ways this file is being created.. either all compromised boxes have a certain (infected) rpm/package/module installed or local software infection I only say this because we are concerned with what is creating the exploit file libkeyutils.so.1.9 Maybe time to compare installed packages? We don't have any of the compromises on our boxes but I do see lots of files in /lib or /lib64 updated on 17/01 just not the file that everyone who has the exploit has.

Posted by Gogax | Simon, 02-19-2013, 12:01 AM
I think its local software infection or something simillar. I have a copy of my prod cpanel servers with no clients on it, i just checked it and its completely clean.

Posted by Patrick, 02-19-2013, 12:02 AM
Null route ... that IP is def linked to the attacker and calling back to them. It appears in every backdoored lib so far.

Posted by Steven, 02-19-2013, 12:05 AM
More food for thought -- facebook was hacked how? Not saying its definitely not problem on the server.. but there is too many conflicting stories, and to many contradictions on server setup. Two things remains constant -- people & software vendors delaying updates. Some flash updates took half a year to get out.. and just came out in the last two weeks.

Posted by matbz, 02-19-2013, 12:06 AM
I asked and thought it relevant to cPanel updates because of the dodgey file's timestamp (the time, not date, pretty much correlates to when cPanel Updates run - in the middle of the night, local time of course)

Posted by matbz, 02-19-2013, 12:09 AM
same way their share price was hacked? Seriously though... what other packages have the same date in the timestamp and almost same time as the exploited libkeyutils file on the infected boxes?

Posted by TravisT-[SSS], 02-19-2013, 12:11 AM
Just to prove Steven's theory is right, I'm going to contact some clients who were compromised and then placed on new servers and see if they got hit yet again. If not I'll ask if they did any local updates or changed anything.

Posted by Gogax | Simon, 02-19-2013, 12:11 AM
any command to grep and sort that stat data automatically?

Posted by NetworkPanda, 02-19-2013, 12:12 AM
But servers with DirectAdmin and (I think) Plesk were also reported been infected. So maybe something common all these control panels recently downloaded, could be the reason of the infection.

Posted by Hellsheep, 02-19-2013, 12:13 AM
stat /lib64/* | grep 'Change: 2012-12-10' Change: 2012-12-10 20:08:12.606379504 +0930 Change: 2012-12-10 20:08:12.541379501 +0930 That gives me 2 files that were changed, one is the libkeyutils but is there any way to figure out what the other one is?

Posted by matbz, 02-19-2013, 12:14 AM
rpm -qa | sort

Posted by matbz, 02-19-2013, 12:18 AM
well you have something in common with some of our boxes there, which is why, if we have do not have the infected file... it may be a certain installed package that is at fault and planting this bloody exploited file during an update

Posted by jalapeno55, 02-19-2013, 12:18 AM

Posted by RWJKM, 02-19-2013, 12:19 AM
Unfortunately, the timestamps on libkeyutils.so.1.9 probably won't help. If you look back at the snoopy log posted by mattmackman (it's on the top of page 14 of this thread), one of the last things done while injecting the malicious .so is to set it's times to those of the real libkeyutils-1.2.so: Last edited by RWJKM; 02-19-2013 at 12:23 AM.

Posted by Hellsheep, 02-19-2013, 12:22 AM
Thanks for that, this is the output: stat /lib64/* | grep 'Change: 2012-12-10' -B6 File: `/lib64/libkeyutils.so.1' -> `libkeyutils.so.1.9' Size: 18 Blocks: 0 IO Block: 4096 symbolic link Device: fd00h/64768d Inode: 2361693 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-02-18 20:55:03.464880930 +0930 Modify: 2012-06-22 15:50:37.000000000 +0930 Change: 2012-12-10 20:08:12.606379504 +0930 -- File: `/lib64/libkeyutils.so.1.9' Size: 32200 Blocks: 64 IO Block: 4096 regular file Device: fd00h/64768d Inode: 2361660 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-02-18 20:55:03.464880930 +0930 Modify: 2012-06-22 15:50:37.000000000 +0930 Change: 2012-12-10 20:08:12.541379501 +0930

Posted by matbz, 02-19-2013, 12:23 AM
actually... rpm -qa | sort > packages.txt because the list will be long for most of us this will create a text file (packages.txt) in the root folder I feel particular (almost) that for old installs (i.e. not fresh today) there was an update on 17th January which (on compromised boxes) downloaded/installed the infected file)

Posted by matbz, 02-19-2013, 12:29 AM
I wasn't referring to new/fresh installs, just old existing intalls (fresh installs will no doubt run an update so there will be a different file timestamp compared to old installs). The file timestamp correlates with automated updates (i.e. the early hours of the morning)

Posted by Steven, 02-19-2013, 12:30 AM
Depends on timezone. My timestamp is time in portugal.

Posted by RWJKM, 02-19-2013, 12:33 AM
Edit: I take this back, only the `modify' time appears to be faked on the malicious file; the `change' time appears to still be accurate. My mistake. Last edited by RWJKM; 02-19-2013 at 12:36 AM.

Posted by matbz, 02-19-2013, 12:37 AM
Not at all, filestamps will be in your local time (a long as you have set it to use your local time) There is no universal file timestamp, it's relevant to the box.

Posted by Steven, 02-19-2013, 12:40 AM
I am aware, but I can't get the words to come out the way I want them.. For what its worth. The time for upcp which would be automatic updates is 1AM on that box.

Posted by Steven, 02-19-2013, 12:41 AM
The change time is when the metadata changes, the modify time is when the contents change. The modify time is what shows up when you ls -l the file.

Posted by matbz, 02-19-2013, 12:44 AM
stat /lib64/* | grep 'Change: 2013-01-17' -B6 might be more interesting on compromised boxes that are not fresh installs

Posted by RWJKM, 02-19-2013, 12:49 AM
Thanks for that, Steven. My confusion came from thinking that the touch call could update the ctime, as well as the mtime and atime, which some furious googling quickly cleared up for me.

Posted by matbz, 02-19-2013, 12:49 AM
Oh sorry, of course. But I hope you see where I am coming from? It's not the first time a package has been infected. And timestamp may not exactly correlate with upcp start time. The more I read the more I am convinced it is an infected package, innocently installed during the overnight control panel updates (not just cPanel). Regrettably I do not have the upcp log files for that time period. Deleted

Posted by Hellsheep, 02-19-2013, 12:51 AM
That outputs nothing on my old install unfortunately.

Posted by Steven, 02-19-2013, 12:53 AM
var/log/yum.log* Have that?

Posted by matbz, 02-19-2013, 12:54 AM
I really think it is time to compare packages (I hope no one takes that as an innuendo)... rpm -qa | sort > packages.txt

Posted by matbz, 02-19-2013, 01:00 AM
Sorry for the delay, I should have been asleep 5 hours ago, but for the intrigue! Of course on a default install /var/log/yum.log exists, so yes I do. But what does yours say compared to mine on an 'uncompromised' box?

Posted by Steven, 02-19-2013, 01:02 AM
here you go

Posted by Hellsheep, 02-19-2013, 01:03 AM
In case anyone is interested, I have attached the results of this years yum.log and also attached rpm -qa | sort Attached Files packagelist.txt (15.6 KB, 35 views) yum.txt (1.3 KB, 25 views)

Posted by matbz, 02-19-2013, 01:07 AM
VPS or dedicated? Could the exploit have altered/suppressed that logging at all? If this is a compromised box what was the change date of the infected file? a dedicated of ours... Jan 03 00:28:52 Updated: gtk2-2.10.4-23.el5_8.x86_64 Jan 03 00:28:53 Updated: gtk2-2.10.4-23.el5_8.i386 Jan 03 00:28:59 Updated: gtk2-devel-2.10.4-23.el5_8.x86_64 Jan 08 00:29:38 Updated: 30:bind-libs-9.3.6-20.P1.el5_8.6.x86_64 Jan 08 00:29:39 Updated: 30:bind-9.3.6-20.P1.el5_8.6.x86_64 Jan 08 00:29:39 Updated: 30:bind-utils-9.3.6-20.P1.el5_8.6.x86_64 Jan 08 00:29:40 Updated: 30:caching-nameserver-9.3.6-20.P1.el5_8.6.x86_64 Jan 08 00:29:43 Updated: 30:bind-devel-9.3.6-20.P1.el5_8.6.x86_64 Jan 08 00:29:43 Updated: 30:bind-libs-9.3.6-20.P1.el5_8.6.i386 Jan 08 00:29:44 Updated: 30:bind-devel-9.3.6-20.P1.el5_8.6.i386 Jan 17 01:10:50 Updated: libgcc-4.1.2-54.el5.x86_64 Jan 17 01:11:09 Updated: glibc-common-2.5-107.x86_64 Jan 17 01:11:14 Updated: glibc-2.5-107.x86_64 Jan 17 01:11:15 Updated: zlib-1.2.3-7.el5.x86_64 Jan 17 01:11:15 Updated: nspr-4.9.1-6.el5.x86_64 Jan 17 01:11:15 Updated: popt- Jan 17 01:11:15 Updated: e2fsprogs-libs-1.39-35.el5.x86_64 Jan 17 01:11:15 Updated: nss-3.13.5-8.el5.x86_64 Jan 17 01:11:15 Updated: sqlite-3.3.6-6.x86_64 Jan 17 01:11:15 Updated: gdbm-1.8.0-28.el5.x86_64 Jan 17 01:11:15 Updated: cpio-2.6-25.el5.x86_64 Jan 17 01:11:15 Updated: libstdc++-4.1.2-54.el5.x86_64 Jan 17 01:11:16 Updated: lvm2-2.02.88-10.el5.x86_64 Jan 17 01:11:16 Updated: libvolume_id-095-14.29.el5.x86_64 Jan 17 01:11:16 Updated: kpartx-0.4.7-54.el5.x86_64 Jan 17 01:11:17 Updated: device-mapper-multipath-0.4.7-54.el5.x86_64 Jan 17 01:11:19 Updated: e2fsprogs-1.39-35.el5.x86_64 Jan 17 01:11:20 Updated: diffutils-2.8.1-16.el5.x86_64 Jan 17 01:11:21 Updated: gtk2-2.10.4-29.el5.x86_64 Jan 17 01:11:24 Updated: file-4.17-28.x86_64 Jan 17 01:11:25 Updated: tcl-8.4.13-6.el5.x86_64 Jan 17 01:11:25 Updated: libgomp-4.4.7-1.el5.x86_64 Jan 17 01:11:25 Updated: logrotate-3.7.4-14.x86_64 Jan 17 01:11:26 Updated: 2hadow-utils-4.0.17-21.el5.x86_64 Jan 17 01:11:27 Updated: pam- Jan 17 01:11:28 Updated: udev-095-14.29.el5.x86_64 Jan 17 01:11:28 Updated: kbd-1.12-22.el5.x86_64 Jan 17 01:11:29 Updated: gawk-3.1.5-16.el5.x86_64 Jan 17 01:11:29 Updated: 2:vim-minimal-7.0.109-7.2.el5.x86_64 Jan 17 01:11:29 Updated: kernel-headers-2.6.18-348.el5.x86_64 Jan 17 01:11:30 Updated: glibc-headers-2.5-107.x86_64 Jan 17 01:11:30 Updated: centos-release-notes-5.9-0.x86_64 Jan 17 01:11:30 Updated: crontabs-1.10-11.el5.noarch Jan 17 01:11:30 Updated: hwdata-0.213.28-1.el5.noarch Jan 17 01:11:30 Updated: nash- Jan 17 01:11:31 Updated: gnutls-1.4.1-10.el5.x86_64 Jan 17 01:11:31 Updated: cpp-4.1.2-54.el5.x86_64 Jan 17 01:11:31 Updated: procps-3.2.7-22.el5.x86_64 Jan 17 01:11:32 Updated: selinux-policy-2.4.6-338.el5.noarch Jan 17 01:11:33 Updated: cvs-1.11.22-11.el5_8.1.x86_64 Jan 17 01:11:35 Updated: gtk2-devel-2.10.4-29.el5.x86_64 Jan 17 01:11:35 Updated: 1:quota-3.13-8.el5.x86_64 Jan 17 01:11:35 Updated: ftp-0.17-38.el5.x86_64 Jan 17 01:11:35 Updated: psmisc-22.2-11.x86_64 Jan 17 01:11:35 Updated: strace-4.5.18-18.el5.x86_64 Jan 17 01:11:36 Updated: sysstat-7.0.2-12.el5.x86_64 Jan 17 01:11:36 Updated: iproute-2.6.18-15.el5.x86_64 Jan 17 01:12:02 Updated: selinux-policy-targeted-2.4.6-338.el5.noarch Jan 17 01:12:02 Updated: 10:centos-release-5-9.el5.centos.1.x86_64 Jan 17 01:12:02 Updated: glibc-devel-2.5-107.x86_64 Jan 17 01:12:02 Updated: pam-devel- Jan 17 01:12:02 Updated: grub-0.97-13.10.el5.x86_64 Jan 17 01:12:03 Updated: libstdc++-devel-4.1.2-54.el5.x86_64 Jan 17 01:12:03 Updated: gdbm-devel-1.8.0-28.el5.x86_64 Jan 17 01:12:03 Updated: e2fsprogs-devel-1.39-35.el5.x86_64 Jan 17 01:12:04 Updated: zlib-devel-1.2.3-7.el5.x86_64 Jan 17 01:12:04 Updated: logwatch-7.3-10.el5.noarch Jan 17 01:12:04 Updated: 50:aspell-en-6.0-3.x86_64 Jan 17 01:12:05 Updated: glibc-2.5-107.i686 Jan 17 01:12:05 Updated: zlib-1.2.3-7.el5.i386 Jan 17 01:12:05 Updated: e2fsprogs-libs-1.39-35.el5.i386 Jan 17 01:12:06 Updated: pam- Jan 17 01:12:06 Updated: nspr-4.9.1-6.el5.i386 Jan 17 01:12:06 Updated: libvolume_id-095-14.29.el5.i386 Jan 17 01:12:06 Updated: gdbm-1.8.0-28.el5.i386 Jan 17 01:12:06 Updated: libgcc-4.1.2-54.el5.i386 Jan 17 01:12:06 Updated: libstdc++-4.1.2-54.el5.i386 Jan 17 01:12:06 Updated: nss-3.13.5-8.el5.i386 Jan 17 01:12:08 Updated: gtk2-2.10.4-29.el5.i386 Jan 17 01:12:08 Updated: gnutls-1.4.1-10.el5.i386 Jan 17 01:12:08 Updated: libgomp-4.4.7-1.el5.i386 Jan 17 01:12:08 Updated: popt- Jan 17 01:12:08 Updated: sqlite-3.3.6-6.i386 Jan 17 01:12:09 Updated: tcl-8.4.13-6.el5.i386 Jan 17 01:12:09 Updated: glibc-devel-2.5-107.i386 Jan 17 01:12:10 Updated: libstdc++-devel-4.1.2-54.el5.i386 Jan 17 01:12:11 Updated: gdbm-devel-1.8.0-28.el5.i386 Jan 17 01:12:11 Updated: pam-devel- Jan 17 01:12:11 Updated: e2fsprogs-devel-1.39-35.el5.i386 Jan 17 01:12:11 Updated: zlib-devel-1.2.3-7.el5.i386 Jan 17 01:12:12 Updated: gcc-4.1.2-54.el5.x86_64 Jan 17 01:12:13 Updated: gcc-c++-4.1.2-54.el5.x86_64 Jan 17 01:12:13 Updated: python-2.4.3-56.el5.x86_64 Jan 17 01:12:14 Updated: hal- Jan 17 01:12:14 Updated: rpm-libs- Jan 17 01:12:15 Updated: iscsi-initiator-utils- Jan 17 01:12:16 Updated: rpm- Jan 17 01:12:18 Updated: python-libs-2.4.3-56.el5.x86_64 Jan 17 01:12:18 Updated: pm-utils-0.99.3-14.el5.x86_64 Jan 17 01:12:20 Updated: gnome-vfs2-2.16.2-10.el5.x86_64 Jan 17 01:12:20 Updated: rpm-python- Jan 17 01:12:20 Updated: yum-metadata-parser-1.1.2-4.el5.x86_64 Jan 17 01:12:20 Updated: tkinter-2.4.3-56.el5.x86_64 Jan 17 01:12:20 Updated: libuser-0.54.7-3.el5.x86_64 Jan 17 01:12:21 Updated: gdb-7.0.1-45.el5.centos.x86_64 Jan 17 01:12:21 Updated: rpm-build- Jan 17 01:12:22 Updated: mkinitrd- Jan 17 01:12:22 Updated: kudzu- Jan 17 01:12:23 Updated: python-devel-2.4.3-56.el5.x86_64 Jan 17 01:12:24 Updated: m2crypto-0.16-9.el5.x86_64 Jan 17 01:12:24 Updated: hal-devel- Jan 17 01:12:24 Updated: python-iniparse-0.2.3-6.el5.noarch Jan 17 01:12:24 Updated: yum-3.2.22-40.el5.centos.noarch Jan 17 01:12:24 Updated: gnome-vfs2-devel-2.16.2-10.el5.x86_64 Jan 17 01:12:25 Updated: python-tools-2.4.3-56.el5.x86_64 Jan 17 01:12:26 Updated: hal- Jan 17 01:12:26 Updated: gnome-vfs2-2.16.2-10.el5.i386 Jan 17 01:12:26 Updated: mkinitrd- Jan 17 01:12:27 Updated: python-devel-2.4.3-56.el5.i386 Jan 24 00:29:45 Updated: tzdata-2012j-1.el5.x86_64 Jan 24 00:29:45 Updated: 12:dhclient-3.0.5-33.el5_9.x86_64 Jan 24 00:29:47 Updated: kernel-headers-2.6.18-348.1.1.el5.x86_64 Feb 01 00:30:27 Updated: nspr-4.9.2-2.el5_9.x86_64 Feb 01 00:30:27 Updated: freetype-2.2.1-32.el5_9.1.x86_64 Feb 01 00:30:28 Updated: freetype-2.2.1-32.el5_9.1.i386 Feb 01 00:30:28 Updated: nspr-4.9.2-2.el5_9.i386 Feb 01 00:30:28 Updated: nss-3.13.6-3.el5_9.x86_64 Feb 01 00:30:29 Updated: freetype-devel-2.2.1-32.el5_9.1.i386 Feb 01 00:30:30 Updated: freetype-devel-2.2.1-32.el5_9.1.x86_64 Feb 01 00:30:30 Updated: nss-3.13.6-3.el5_9.i386 Feb 05 00:29:14 Updated: kpartx-0.4.7-54.el5_9.1.x86_64 Feb 05 00:29:16 Updated: device-mapper-multipath-0.4.7-54.el5_9.1.x86_64

Posted by Steven, 02-19-2013, 01:09 AM
Impossible to know if the logging was suppressed. Its a compromised machine. Change: 2013-01-17 06:23:59.000000000 +0000 its a centos 6 dedicated server.

Posted by matbz, 02-19-2013, 01:11 AM
yum log output is as previous package output is attached (I hope) Attached Files packages.txt (13.4 KB, 25 views)

Posted by matbz, 02-19-2013, 01:16 AM
I'll give you the outputs from the other boxes tomorrow if you don't mind. knackered.com

Posted by camt, 02-19-2013, 01:22 AM
Just adding a bit to the thread. I'm running 5 VPS's across 3 nodes. All running Centos 6.3 64bit with cpanel, SSH is using p22 and a password on each VPS. 2 of the VPS's were exploited and both of them are running a drupal site and not really anything else on the VPS. The first VPS has a very large and active drupal site running on it and then a small private buddypress site whose home directory is password protected with cpanel and the site isn't really used. The second VPS was setup about a week about and is a carbon copy of the large drupal site, this VPS is used for development and testing of this drupal site only. Nothing else is running on this VPS.

Posted by lbeachmike, 02-19-2013, 01:44 AM
Realizing that this has not been limited to only cpanel boxes, I can't help but loosely correlate the current RPM line-of-thinking to cpanel's migrating to RPM-based packages, and adding RPM-checking functionality which first complained about Munin's RPM on my boxes January 31. Late Feb 1, I subsequently ran a YUM Update which resulted in numerous package updates. My change file date for libkeyutils.so.1.9 is then just about four hours later on Feb 2 around 3:30 AM server time and just a few minutes apart for both boxes. Where there's smoke there's a fire ... ?

Posted by Hellsheep, 02-19-2013, 01:50 AM
I have some output from auditctl, doing an ausearch results in the following: ausearch -f /lib64/libkeyutils.so.1.9 ---- time->Tue Feb 19 13:40:19 2013 type=PATH msg=audit(1361247019.623:341282): item=0 name="/lib64/libkeyutils.so.1.9" inode=2361660 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 type=CWD msg=audit(1361247019.623:341282): cwd="/root" type=SYSCALL msg=audit(1361247019.623:341282): arch=c000003e syscall=191 success=no exit=-61 a0=7fff8f8c2120 a1=7f8df2603f60 a2=7fff8f8c20e0 a3=14 items=1 ppid=453 pid=29279 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=54223 comm="ls" exe="/bin/ls" key="backdoor" ---- time->Tue Feb 19 13:40:19 2013 type=PATH msg=audit(1361247019.623:341283): item=0 name="/lib64/libkeyutils.so.1.9" inode=2361660 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 type=CWD msg=audit(1361247019.623:341283): cwd="/root" type=SYSCALL msg=audit(1361247019.623:341283): arch=c000003e syscall=192 success=no exit=-61 a0=7fff8f8c2120 a1=7f8df2a232d9 a2=238b1c0 a3=ff items=1 ppid=453 pid=29279 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=54223 comm="ls" exe="/bin/ls" key="backdoor" ---- time->Tue Feb 19 13:40:19 2013 type=PATH msg=audit(1361247019.623:341284): item=0 name="/lib64/libkeyutils.so.1.9" inode=2361660 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 type=CWD msg=audit(1361247019.623:341284): cwd="/root" type=SYSCALL msg=audit(1361247019.623:341284): arch=c000003e syscall=192 success=no exit=-61 a0=7fff8f8c2120 a1=7f8df23fedb7 a2=0 a3=0 items=1 ppid=453 pid=29279 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=54223 comm="ls" exe="/bin/ls" key="backdoor" ---- time->Tue Feb 19 13:40:19 2013 type=PATH msg=audit(1361247019.624:341285): item=0 name="/lib64/libkeyutils.so.1.9" inode=2361660 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 type=CWD msg=audit(1361247019.624:341285): cwd="/root" type=SYSCALL msg=audit(1361247019.624:341285): arch=c000003e syscall=192 success=no exit=-61 a0=7fff8f8c2120 a1=7f8df23fed88 a2=0 a3=0 items=1 ppid=453 pid=29279 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=54223 comm="ls" exe="/bin/ls" key="backdoor"

Posted by Steven, 02-19-2013, 01:53 AM
Hell sheep... are you doing an ls on the binary?

Posted by Hellsheep, 02-19-2013, 02:08 AM
Hey Steven, it's possible. See below: history | grep ls 1023 auditctl -w /lib64/libkeyutils.so.1.9 -p war -k backdoor 1024 ausearch -w /lib64/libkeyutils.so.1.9 1025 ausearch -W /lib64/libkeyutils.so.1.9 1029 ausearch /lib64/libkeyutils.so.1.9 1048 ausearch -f /lib64/libkeyutils.so.1.9 1056 ls -l /lib64 - This one was done by me 1062 stat libkeyutils.so.1.9 1063 stat /lib64/libkeyutils.so.1.9 1085 ausearch -f /lib64/libkeyutils.so.1.9 1086 ls - This one was done as a test after you asked if i was doing an ls on it 1087 ausearch -f /lib64/libkeyutils.so.1.9 1088 ls /lib64/libkeyutils.so.1.9 - This also done after you asked me if i was doing it as a test 1089 ausearch -f /lib64/libkeyutils.so.1.9 1090 history | grep ls 1095 history | grep ls The two testing ones I did were to confirm I get the same result and sure enough I do. However unless doing stat does an ls then no only one of the ls is explainable. Edit: To make a bit more sense for you, the first ls was done by me and was after i started the audit. The other two ls's are both after you replied to this thread asking if I had ls'd the binary. What is weird is even if I did do it all 3 times there is 4 entries in the audit and only 3 ls's. Does the command generate multiple entries in the log? Last edited by Hellsheep; 02-19-2013 at 02:18 AM.

Posted by Flegmaa, 02-19-2013, 05:30 AM
I have found out that removing /lib/libkeyutils.so.1.9 breaks python (therefore yum too) and named server. After i put back lib file, it started ok and python/yum started to work normally. So is this a false positive notification from CSF or?

Posted by Cyclon, 02-19-2013, 05:44 AM
I am just wondering if there is any server that have this file added and that had a pasword login disabled in sshd? Maybe I am wrong, but I somehow concluded that all compromised servers had password login enabled...?

Posted by Cyclon, 02-19-2013, 05:49 AM
It't not a false positive. You should reinstall keyutils... # yum reinstall keyutils-libs Remove file /lib/libkeyutils.so.1.9 and reboot after that.

Posted by SoftDux, 02-19-2013, 05:49 AM
You should have reinstalled keyutils-libs in order to replace the missing library first.

Posted by Hellsheep, 02-19-2013, 06:25 AM
So now that nothing was really happening since I disabled SSH access without public keys I decided to reinstall the library and quarantine the infected one. Installing an updated kernel and doing a cPanel update to 11.36. Going to keep an eye on the server and monitor it closely over the next few days to see if a re-infection occurs. If all goes well and no re-infection I'll rebuild the server with the same patches applied and leave it running a few days to ensure it doesn't get infected, then restore from backup for customer data.

Posted by EthernetServers, 02-19-2013, 07:06 AM
cPanel 11.36 shouldn't be used on production servers yet. It's a CURRENT release and you should use RELEASE at minimum. http://docs.cpanel.net/twiki/bin/vie...elease%20tiers

Posted by NetworkPanda, 02-19-2013, 07:20 AM
A command that could help finding exploits like this, even with new filenames (libkeyutils.so.1.9 or any other name) will be the following For 64 bit For 32 bit If you don't get any result, then you are not infected by this exploit. If you get something like "xyz-filename.so is not owned by any package" then you need to check it This command scans all files in /lib or /lib64 folders and if it finds any libraries not used by a package (which is the case with the exploit discussed here) it prints their file names. Last edited by NetworkPanda; 02-19-2013 at 07:29 AM.

Posted by constantine, 02-19-2013, 08:04 AM
I deleted libkeyutils.so.1.9 from the server . It caused a problem with our services ( such as apache, ssh , ... ) . When I restart the apache it shows below error :Do you have any idea ?

Posted by EthernetServers, 02-19-2013, 08:06 AM
You need to reinstall libkey-utils. yum -y reinstall keyutils-libs

Posted by atrocity, 02-19-2013, 08:08 AM
same problem here, i re-installed the keyutils-libs, moved the /lib/libkeyutils.so.1.9 to /root/ and rebooted, like Cyclon wrote, but now everything is down (apache, ssh, ...) I can't retry to re-install the keyutils-libs ... Anybody have a idea ?

Posted by EthernetServers, 02-19-2013, 08:10 AM
32-bit: ls -lah /lib | grep libkey 64-bit ls -lah /lib64 | grep libkey What does that output?

Posted by NetworkPanda, 02-19-2013, 08:11 AM
In your /lib or /lib64 folder what is the target of the libkeyutils.so.1 symlink? You check it by running the ls -l or ll command

Posted by atrocity, 02-19-2013, 08:18 AM
Here is the same output as on a other not compromized server, because i don't have ssh, i'm on console ... 64-bit ls -lah /lib64 | grep libkey -rwxr-xr-x 1 root root 7.1K Jan 6 2007 libkeyutils-1.2.so* lrwxrwxrwx 1 root root 18 Feb 19 12:36 libkeyutils.so.1 -> libkeyutils.so.1.9*

Posted by atrocity, 02-19-2013, 08:23 AM
found a post on cpanel forum, seems to work ... 1. SSH to the server 2. cd /lib64/ 3. rm libkeyutils.so.1.9 4. rm libkeyutils.so.1 5. ln -s libkeyutils.so.1.2 libkeyutils.so.1 ! i have libkeyutils.so.1.2, but there can also be libkeyutils.so.1.3 ... 6. Restart ssh 7. yum update kernel and Reboot to close any active connections Last edited by atrocity; 02-19-2013 at 08:32 AM.

Posted by atrocity, 02-19-2013, 08:26 AM
YES, server rebooted and is running .... Now, will have to fix all this stuff. Do anybody know what to do to repair and clean such a server ?

Posted by weredigital, 02-19-2013, 08:38 AM
Atrocity a complete system re-install would be needed as this is a root exploit. But the cause and fix has not been totally confirmed as of yet. Right now all we can do is wait for the fix as new installs get the exploits quickly. There are some helpful tips in the thread for increased security you may consider. Hope that helps

Posted by pipindopal, 02-19-2013, 08:40 AM
Is there any system affected that had sshd not open to public network?

Posted by atrocity, 02-19-2013, 08:53 AM
i did a mistake, i have : /lib64/libkeyutils-1.2.so and NOT libkeyutils.so.1.2 Please use this if you are also on 1.2.so : 1. SSH to the server 2. cd /lib64/ 3. rm libkeyutils.so.1.9 4. rm libkeyutils.so.1 5. ln -s libkeyutils-1.2.so libkeyutils.so.1 6. Restart ssh 7. yum update kernel and Reboot to close any active connections

Posted by serve-you, 02-19-2013, 09:03 AM
For all you new folks coming into this thread, please note that this is not a *fix* at all. We have already seen this lib pop back up on "cleaned" servers within hours. So continue monitoring this until the source has been found.

Posted by Flegmaa, 02-19-2013, 09:26 AM
I couldn't reinstall it since YUM wasnt working, so i put the file i backuped. Now i reinstalled, but CSF keeps popping up Also, this is output that gives me "for i in `ls /lib/ | grep -v '@'`; do rpm -qf /lib/$i | grep ot owned by any package'; done" command:

Posted by jalapeno55, 02-19-2013, 10:04 AM
Someone on the cPanel forums mentioned a Curl exploit. http://curl.haxx.se/docs/adv_20130206.html http://secunia.com/advisories/52103/ Does anyone that got hacked have an affected version of Curl? 7.26.0 through 7.28.1

Posted by bloodyman, 02-19-2013, 10:09 AM
cPanel servers use Curl 7.24.0

Posted by jalapeno55, 02-19-2013, 10:10 AM
Mine does not. Last edited by jalapeno55; 02-19-2013 at 10:15 AM.

Posted by Steven, 02-19-2013, 10:26 AM
One thing you guys need to keep in mind. It does not matter what you do to ssh at the server level. If someone can access your WHM, it won't matter. 1.) Start ssh on default settings: 2.) You can modify your hosts.deny/allow file 3.) You can disable csf

Posted by egillette, 02-19-2013, 10:52 AM
I personally only use PuTTY and WinSCP, and even when I use WinSCP I *never* login using root. Most of the boxes I connect to using PuTTY, I login as an unprivileged user and then use sudo or su to root. Although on 3 of the boxes that had that file present, 2 of them allow root login.

Posted by unSpawn, 02-19-2013, 10:54 AM
And FWIW there is no need to use 'rpm -qf': libkeyutils' latest official release is 1.5.5 of 30-Nov-2011 and IIGC the current version CentOS 6.3 provides is keyutils-libs-1.4-4.el6 of 22-Jun-2012. The latter has an actual file name of libkeyutils.so.1.3 so any minor version purporting to be greater than .3 will be a fake anyway.

Posted by egillette, 02-19-2013, 10:55 AM
All 3 of the boxes I had that contained this exploit, were dedicated servers -- not a one was a VPS. In fact none of the VPS' I manage showed any sign of the file.

Posted by atrocity, 02-19-2013, 11:00 AM
we have about 20 VPS under CPanel, and 4 are affected. All our servers have curl 7.15.5 Is this version affected or no, i really don't know, Secunia say : The vulnerability is reported in versions 7.26.0 through 7.28.1. Strange ... We have disabled SSH to all our CPanel servers, using a whitelist, hoping this can help until there a fix. PS to whatevercomputes : Also IF we reformat all the affected servers, reinstall everything and restore the last backups, there is NO way to be sure that this "new" servers will not anymore be affected, until we know what's going on ...

Posted by jalapeno55, 02-19-2013, 11:02 AM
Reasons #573-575 why cpanel should give an alternate user for WHM.

Posted by Cyclon, 02-19-2013, 11:09 AM
So, is there any server that contains this exploit with disabled password login in sshd?

Posted by FastServ, 02-19-2013, 11:10 AM
I'm sitting here twiddling my thumbs without a single infected box to play with. Many of them have password auth, some have been online for years, some just days. Most use external MTA's for spam filtering and do not use domain keys with Exim. Maybe a long shot, but what do the repo settings look like on the affected boxes? Is there some commonality in the location of the affected boxes, such as provider or geolocation? Yum does a very rough geolocation and is accurate within 500 miles in most cases, and some providers force yum to use onsite repos. There could be a rouge repo out there. Most CentOS repos are donated and many are poorly managed. Last edited by FastServ; 02-19-2013 at 11:20 AM.

Posted by Steven, 02-19-2013, 11:11 AM
Word of advice -- Those of you that have removed the file. All of your ssh private keys inside ~/.ssh folders... you should replace them. (id_rsa, id_dsa, etc)

Posted by egillette, 02-19-2013, 11:12 AM
My script *attempts* to put a 'band-aid' on the issue. It quarantines the affected file, symlinks to an alternate lib file, and also touches a new file with the same name, assigns the immutable and undelete bits to try and prevent it from coming back. Like serve-you stated -- there is no "fix" per se, but this should help while the origin of the file is established. If you prefer to automate the process of a removal a little. . .

Posted by egillette, 02-19-2013, 11:13 AM
Yes, I did that myself and second what Steven is suggesting. I went one step further and deleted them, and re-created them from scratch to be safe.

Posted by iseletsk, 02-19-2013, 11:30 AM
Standard kernel, grsec kernel? It is not a repo, as it is not enough to hack into repo, you also need a key to sign RPMs.

Posted by gigatux, 02-19-2013, 11:33 AM
FWIW, we noticed a hack against a SLES kernel (3.1.0-1.2-xen) so it's not just the CentOS kernels that are affected.

Posted by Steven, 02-19-2013, 11:35 AM
Was this a vps or a node? What services are running on this server?

Posted by gigatux, 02-19-2013, 11:39 AM
It's a VPS running the standard cPanel services. The node is unaffected. upcp seems to be run each day on the VPS so it should have been kept reasonably up to date. Apart from Varnish, I can't think of anything else exotic on there.

Posted by bloodyman, 02-19-2013, 11:39 AM
Can anyone reply - does it affects only servers with password authentication turned on?

Posted by jalapeno55, 02-19-2013, 11:42 AM
Apearently affects servers with it off as well.

Posted by lbeachmike, 02-19-2013, 11:46 AM
It's clear that this is not a fix and the point of initial entry is unknown. The point that I think many of us non-expert level users are unclear on is whether or not this band-aid followed by entirely disabling SSH mitigates anything or eliminates re-infection. I'm not sure if this answer is yet known. I know that the band-aid + leaving SSH enabled allows reinfection. Is it known whether or not disabling SSH prevents reinfection? Thanks.

Posted by matbz, 02-19-2013, 11:49 AM
Not so sure I believe that. Only one person claims to have had 1 box affected with passwd auth disabled so far, but I am still not convinced.

Posted by jalapeno55, 02-19-2013, 11:50 AM
Appearently it does not prevent it.

Posted by Steven, 02-19-2013, 12:02 PM
Anyone who keeps getting reinfected interesting trying some alternate ssh rpms for me?

Posted by FastServ, 02-19-2013, 12:02 PM
Stock kernel + ksplice on all machines. Password auth enabled. Some very busy and visible with 1000's of accounts. Some with a history of backdoors and symlink attacks (although kspliced the whole time). No infections.

Posted by lbeachmike, 02-19-2013, 12:14 PM
Hi Eric - After your script runs, there is still some ssh cleanup to do if I understand and recall properly details Steve mentioned earlier on in this thread. I believe it was found that particular ssh components had been modified. Is it possible to enhance the script so as to restore those things as well or to at least have it inform the end user of the necessary subsequent steps? Thanks.

Posted by tech1ne, 02-19-2013, 12:14 PM
It would be great if anyone infected had remote logging setup. For servers that are getting re infected remote logging is a good idea. Additionally everyone with an infected server should reviewing these logs just in case any evidence was left behind. grep libkey /usr/local/apache/logs/* grep libkey /var/log/* grep libkey /root/bash_history

Posted by SafeSrv, 02-19-2013, 12:15 PM
This only affects redhat based systems and its all done via the ptrace kernel exploit as far as i know, this has been ready available in malware since august last year. I dont think its bypassing cagefs machines either but i think these CageFS machines are being exploited at root level maybe throwing in some confusion thinking its dameon based, SSHD etc Its also worth noting that CURL and dovecot at the same time have some holes circulating i haven't fully checked yet but when i do and dovecot works al be heading over to claim the 1k http://www.dovecot.org/security.html

Posted by Steven, 02-19-2013, 12:17 PM
You are aware that servers with courier have been hacked too right?

Posted by SafeSrv, 02-19-2013, 12:24 PM
Yes, i can see that but im not saying its solely down to dovecot am i, i don't even know that works !

Posted by asimzeeshan, 02-19-2013, 12:25 PM
Thank you very much Steven, it helped a LOT

Posted by FastServ, 02-19-2013, 12:26 PM
Any 3rd party repos on these machine? Has there been a PLAIN cpanel install compromised (without *ANY* 3rd party plugins or addons)? For the servers that get compromised quickly after install, is your host's installer using a random password each time?

Posted by FastServ, 02-19-2013, 12:30 PM
Is public facing IPMI present on these repeatedly infected machines?

Posted by Steven, 02-19-2013, 12:31 PM
Yes many with just centos repos installed.

Posted by Steven, 02-19-2013, 12:37 PM
Some not all. Some vps were compromised aswell.

Posted by apocas, 02-19-2013, 12:40 PM
Hi guys, I think we found something. We have a cluster of similar servers ~100 (hosting, cpanel, ...). From these only 8 were "infected", in the last days we were crawling everything trying to find a common thing between these. We were curious, since these servers were not the most problematic we have. We found that these 8 servers had ssh password based login enabled and were almost the only ones to have it enabled. Was there anyone "infected" with key-based auth? Last edited by apocas; 02-19-2013 at 12:44 PM.

Posted by Steven, 02-19-2013, 12:47 PM
Right now, myself and others are pursing this. I think password auth has a lot to do with it. Look what happens now with some new openssh rpms: Feb 20 03:37:41 cluster01 sshd[10066]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= user=root Feb 20 03:37:43 cluster01 sshd[10066]: Failed password for root from port 51314 ssh2 Feb 20 03:37:45 cluster01 sshd[10066]: Failed password for root from port 51314 ssh2 Feb 20 03:29:51 cluster01 sshd[8100]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= user=root Feb 20 03:29:53 cluster01 sshd[8100]: Failed password for root from port 50635 ssh2 Feb 20 03:29:55 cluster01 sshd[8100]: Failed password for root from port 50635 ssh2 whereas before, there was no password attempts occuring.

Posted by mattmackman, 02-19-2013, 12:57 PM
Steave. old openssh rpm also checking the password in my server(exploited two times)

Posted by FastServ, 02-19-2013, 01:00 PM
On affected systems... Is /bin/login or /bin/nologin modified? What's /var/log/notify.log -- this log is not present on any server I checked. What's in various cron tabs (/etc/cron.d/*, crontab -l)?

Posted by egillette, 02-19-2013, 01:01 PM
Well, the file has been shown to come back on some of the affected machines, so I modified my script so that it prevents the file from simply recurring. I tested this on one of the original 32-bit and 64-bit boxes I had that were affected, and they both do not show any sign of the original file again, even though last time the file was back within less than 24 hours. So I think the "band-aid" solution is a reasonable measure for now while it is determined how the file was created in the first place. In addition, Steven recommended replacing SSH keys -- I would write this into my script, however, I'd rather leave something like that up to the end-user, since all servers are configured differently and have different users/customers to contend with and such. But anyone executing my script will achieve the following at a minimum: 1) Original file "quarantined" in a new, date specific directory, so that it can be studied later by Steven or others upon request. 2) New file created in its place without sufficient permission to be overwritten, in an attempt to prevent the file from recurring. 3) Symlink to alternate library created with ldconfig executed and ssh restarted. 4) Both 64-bit and 32-bit library files addressed. Outside of that, as has been recommended by others, after the script runs, you'll want to reboot your machine (so that other daemons like httpd, courier, etc. that might be using the affected library are forced to terminate), and then run yum update. Hope that helps for now. Last edited by egillette; 02-19-2013 at 01:04 PM. Reason: Supplied actual script again.

Posted by mattmackman, 02-19-2013, 01:04 PM
bin/login or /bin/nologin is not modified in my server. /var/log/notify.log-- its my custom logfile, the hacker is opned the file, its clear from snoopy log.

Posted by Steven, 02-19-2013, 01:04 PM
Are you sure that is a ip involved in this and not a bruteforce type of ip?

Posted by mattmackman, 02-19-2013, 01:06 PM
That is bruteforce ip.

Posted by Steven, 02-19-2013, 01:08 PM
What I mean is, is it an ordinary bruteforce or do you have evidence its involved in this attack? The ips I provided are involved in it.

Posted by mattmackman, 02-19-2013, 01:15 PM
oh.. I thought, that you have provided ip address is just a bruteforce ip. I am really Sorry for that. This is the IP address of the hacker SSHd from. I have already sent an abuse mail to ovh team regarding this.

Posted by jestep, 02-19-2013, 01:18 PM
Does anyone have a compromised server that is not running a control panel? It would seem we would be seeing compromised servers of all types if this were specifically ssh related and it appears that everything is running a control panel of some type.

Posted by iseletsk, 02-19-2013, 01:18 PM
We put up an advisory for now, and will update it as we have more info. It has scripts that should help with cleanup if anyone needs it: http://www.cloudlinux.com/blog/clnews/sshd-exploit.php

Posted by Steven, 02-19-2013, 01:19 PM
The problem is, many people don't even know about this yet. Its possible we have a large number of servers that are compromised but don't even know.

Posted by jalapeno55, 02-19-2013, 01:20 PM
For what its worth and one of the IPs Steven gave: are cPanel servers (both in Spain), which have likely been hacked. Last edited by jalapeno55; 02-19-2013 at 01:25 PM.

Posted by mattmackman, 02-19-2013, 01:22 PM
Yes, they are using this servers to hack other ones.

Posted by mattmackman, 02-19-2013, 01:24 PM
well its also an OVH IP {}.

Posted by Steven, 02-19-2013, 01:27 PM
Most of the ips appear to be ovh in the attack.

Posted by ModelWebHost, 02-19-2013, 01:30 PM
I received a notification about this today. Have anyone found permanent solution to this?

Posted by jalapeno55, 02-19-2013, 01:30 PM
This may be related: http://arstechnica.com/tech-policy/2...-34m-annually/

Posted by hb9aj4fn, 02-19-2013, 02:03 PM
I know DirectAdmin has also been reported in this thread, but now the first user that is reporting he is infected by this has posted in DAs forum: http://forum.directadmin.com/showthr...018#post234018 - lets hope he give the output of the versions he is running.

Posted by bloodyman, 02-19-2013, 02:06 PM
OK, so conclusion for now is that you suspect password authentication is the source of this attack. Second question for people who has been hacked: 1. did you have firewalled WHM access only for specified IPs/IP range 2. did you have hosts.allow / hosts.deny policy active? 3. did you allow direct root logins?

Posted by Mat Sumpter, 02-19-2013, 02:50 PM
I'd be interested to know what version of curl software has been compiled against on compromised servers. I only have access to a few cpanel boxes still and they are all clean but have older versions of curl than 7.26.

Posted by kmike, 02-19-2013, 02:50 PM
If they are using the login credentials stolen from your local machine, *and* if the snoopy log you've posted is complete, you should see the attackers' login entry in /var/log/secure, since they didn't bothered to cover their tracks in the logs.

Posted by jalapeno55, 02-19-2013, 02:59 PM
I second this.

Posted by kmike, 02-19-2013, 03:00 PM
Another idea to test the theory that the attackers are using the server credentials stolen elsewhere - set up a remote syslog logging, or add a script that will email you the new /var/log/secure entries immediately. That way you'll get notified of the illegitimate ssh accesses at once (provided the attackers would come back to re-infect server). Increasing LogLevel in the sshd config file would probably help, too.

Posted by ciprianbalus, 02-19-2013, 03:00 PM
Hello I have many servers none of them hacked. i run CentOS release 5.9 (Final) CentOS release 6.3 (Final) also CloudLinux Server release 5.9 (Sergey Oleynikov) CloudLinux Server release 6.3 none of my cPanel,Cloudlinux,CSF,Softaculous,CageFS are broken maybe attackers check for port 22 or scripts check for port 22 to be open before attacking. SO PLEASE ADVISE YOUR CUSTOMERS TO CHANGE DEFAULT SSHD PORT to something else root@server [~]# cat /etc/ssh/sshd_config | grep Port Port 22002 I`m using another port for more then 15 years and didnt have any problems! Changing default ssh port can help your machine be 30% less visible on the internet. No one has the time to scan over 22001 ports, not even hackers. Ciprian BALUS!

Posted by jalapeno55, 02-19-2013, 03:05 PM
They've hacked servers not using port 22 as well.

Posted by Steven, 02-19-2013, 03:06 PM
The very first server two weeks ago did not have a standard port.

Posted by matbz, 02-19-2013, 03:07 PM
In one instance that I know of curl-7.19.7-26.el6_2.4.x86_64 but that is also in use on machines of ours that are not infected.

Posted by Flegmaa, 02-19-2013, 03:10 PM
I've changed SSH port, blocked port 22, i'm using duosecurity (when some1 logs in to SSH notification is pushed to my iphone and there i can allow it or disallow) and file is again there after running your script and rebooting the machine. It is VPS, running cPanel/WHM, CentOS 5.9.

Posted by matbz, 02-19-2013, 03:14 PM
Has anyone who's got the compromise used Kippo?

Posted by Mat Sumpter, 02-19-2013, 03:15 PM
Well cpanel typically compiles their own version of curl on a server. So the typical CentOS RPM will exist along with the custom version. The version PHP is compiled against would be used by other cPanel services. I was going to say that as well but I don't think sshd is the original vector. But at this point I don't have much to go on to really exclude sshd. Kippo would certainly give us an idea of what else happens once the server has been backdoored.

Posted by afletch, 02-19-2013, 03:17 PM
Security through obscurity is not security at all.

Posted by LeadDogGraphicStudio, 02-19-2013, 03:19 PM
Besides the current situation, just for general security, I like that idea of using duosecurity, nice service and it's free for under 10 users which is probably most of us admins. Anyone suggest an alternate or better service with the same feature?

Posted by Digivity, 02-19-2013, 03:20 PM
I always restirct ssh access with this setting on my servers. Does it protect my servers against this threat? AllowUsers root@xxx.xxx.xxx.xxx

Posted by Mat Sumpter, 02-19-2013, 03:26 PM
At this point it's too early to say exactly what the vector of the attack is. But it does seem that having a firewall restrict connections to port 22 is the best method to control it. So using a software or hardware firewall will provide the most protection versus a SSH configuration option.

Posted by matbz, 02-19-2013, 03:29 PM
Totally agree, and opening any port to public access to SSH is not such a great idea, although not as much of a bad idea as allowing passwd auth for root. We just close port 22 and allow by IP address.

Posted by Disconnected, 02-19-2013, 03:30 PM
I strongly suspect this is caused by a password compromise likely due to malware on the administrator's machine. I've seen a OpenVZ VPS node infected with this.

Posted by Steven, 02-19-2013, 03:32 PM
I mentioned this as a possible vector last night, but many people disagree.

Posted by Flegmaa, 02-19-2013, 03:32 PM
Yes, duosecurity is fine, free for 10 users and it supports many systems. But even with duosecurity enabled (there has never been a situation someone else requested ssh login to my machine) how is it possible for a hacker to put back infected files after i cleaned them?

Posted by notsosuree, 02-19-2013, 03:32 PM
It's quite clear now that a wide array of web-based exploits are being used, targeting servers all over the place, to install this rootkit for SSHD. The question which remains at this point is which exploits are being used. There's one obvious way to figure this out. You can capture the 0day's being used by using tcpdump/wireshark/etc to capture all HTTP traffic coming into your web servers. What I propose is that one of you which is reliably exploited even after fixing your systems (probably because your restoring your server to the same state as it was with the same vuln's in place) restore your system again with packet capturing in place. Once your system is root'd upload your captures and let some of us examine them for web requests which stand out. With pcap files we could figure out the 0days being used pretty quick. Make sure you start capturing packets before exposing the system to the net, that way there's no chance of the exploit going in before the capturing begins. There's also a small chance some of your HTTP error logs might contain 'bad requests' if multiple exploits are attempted against your websites, check them for odd entries, post anything you see which looks suspicious, it may lead to clues about the 0day's being used. -- In addition, it seems odd that your servers are being rooted to send out spam, seems like a pretty poor strategy. Of all the types of systems which could be hijacked for a spam network you'd think web servers are the worst. I seriously doubt in general admins would ignore a sudden increase in load, so detection within a short period of time is practically guaranteed. Something smells fishy there. Last edited by notsosuree; 02-19-2013 at 03:41 PM. Reason: oddity

Posted by tarsiran, 02-19-2013, 03:35 PM
I received a notification about this today. Have anyone found permanent solution to this? i have a directadmin and has infected by this how can i fix it?

Posted by matbz, 02-19-2013, 03:39 PM
Here is a packages-comparison against the newest of our servers which is running... cPanel/WHM 11.34.1 (build 7) CENTOS 6.3 x86_64 standard – ruby Not much jumped out at me, except you seem to have an older version of libssh2 which I wouldn't have expected. Attached Files compared-packagelistHELLSHEEP-packagesMINE-ws51com.txt (47.3 KB, 31 views)

Posted by Steven, 02-19-2013, 03:41 PM
One version of the libssh2 is not os provided.

Posted by matbz, 02-19-2013, 03:42 PM
It's distinctly possible, have you run malware scans, etc? Tried running something like TCPView?

Posted by matbz, 02-19-2013, 03:45 PM
Which one? The top one is on the compromised machine (whenever there are two files listed the first one is always the first machine - the compromised one)

Posted by egillette, 02-19-2013, 03:49 PM
Hey all, Someone mentioned to me on Facebook that they were concerned that if this is something that is affecting the machine of the administrator, that it could possibly modify the hosts file to change it so that my domain is re-routed to another IP on their Windows machine. That being the case, I also have an SSL certificate which is an EV SSL (green address bar) for those who are concerned, and that should help establish my identity and my company: View the script using this URL instead, to ensure you aren't being redirected elsewhere (your address bar should turn green, and you should see my company name "GSolutions Online LLC" in the address bar for verification): https://www.ericgillette.com/clients/exploit-cleanup Assuming that works as intended, I think it would be safe to assume your hosts file hasn't been modified before you use the script. Also, keep in mind that it is your server grabbing the script, not your Windows machine, so even if your Windows hosts file *IS* modified, it shouldn't affect your ability to execute the script on your server.

Posted by edlmkts, 02-19-2013, 03:51 PM
I ran the "Band Aid" script, which includes ldconfig after ln statements, and it linked back to the original file (ldconfig thinks its the latest library version presumeably) Also a reboot causes ldconfig to run during startup in most systems, so that has the same effect.

Posted by egillette, 02-19-2013, 03:52 PM
Have you run the latest version of the script?? The latest version should prevent the original file from being linked to again.

Posted by matbz, 02-19-2013, 03:53 PM
Your script does not check sub-folders for files that are not owned by a package

Posted by tarsiran, 02-19-2013, 03:58 PM
dear egillette i had tried your script on directadmin server but after resboot server all plugins on directadmin panel has disappeared

Posted by egillette, 02-19-2013, 04:02 PM
Nope, just the base directories (i.e. /lib on 32-bit machines and /lib64 on 64-bit machines). It isn't designed to be an "end-all fix-all" just to help you identify the file, and 'quarantine' it if possible.

Posted by matbz, 02-19-2013, 04:04 PM
Oh I can see it wont fix anything But I meant this is misleading... It does not scan the sub-directories, does it? You need to rewrite that part

Posted by egillette, 02-19-2013, 04:05 PM
Did you receive any error messages while running the script?? Also, did you run yum update as well, so that any appropriate libraries could be re-installed??

Posted by egillette, 02-19-2013, 04:08 PM
Well, the issue is that the sub-directories in that directory could and do vary, so rather than running an intensive scan with my script, I'd rather those executing the script use a tool designed for that purpose. But what you point is still a good point. . .that said I will inform the script executor that this is not a FULL scan of their machine, and that they should conduct one.

Posted by tarsiran, 02-19-2013, 04:09 PM
no i did't receive any message and working fine after reboot i have enter yum update all but show me no package to update.. after that all plugin on directadmin going hidden and also CSF problwm: Firewall Status: Enabled and Running (lfd restart request pending) please help me dude

Posted by Steven, 02-19-2013, 04:09 PM
Except, many people ended up with broken servers after reboot for not running ldconfig.

Posted by egillette, 02-19-2013, 04:12 PM
Based on matbz's feedback. . . The latest version of the script is available here: https://www.ericgillette.com/clients/exploit-cleanup

Posted by egillette, 02-19-2013, 04:14 PM
Do this for me on paste the results here (I'm assuming your machine is 64-bit):

Posted by huck, 02-19-2013, 04:18 PM
I just like to urge a word of caution as I've seen some other "fix scripts" floating on the net. Do not run anything untrusted as root. Verify checksums before executing files - even if they are from a trusted source. @egillette can you publish a MD5 sum for your script? I appreciate your hard work but can see someone trying to take advantage of this situation by pushing a nefarious script. Also, perhaps this was mentioned but in one case we have seen /etc/pam.d/* files changed.

Posted by Steven, 02-19-2013, 04:18 PM
The one that says 'rf' in the name.

Posted by matbz, 02-19-2013, 04:19 PM
They aren't packages? Scanning those sub-folders would be something so far from "an intensive scan". I seem to recall that you incorporated this from a command someone else ran. I think it would be more useful to see various outputs from the individual commands run on compromised systems for the guys here to consider. It would help narrow things down perhaps. With all due respect, I think your script is counter-productive to be frank. A list of commands for others to run and see their individual output would be more useful for all.

Posted by tarsiran, 02-19-2013, 04:22 PM
root@xlhost csf]# ls /lib64 dbus-1 libkeyutils.so.1.9 device-mapper libkrb5.so.3 ld-2.12.so libkrb5.so.3.3 ld-linux-x86-64.so.2 libkrb5support.so.0 libacl.so.1 libkrb5support.so.0.1 libacl.so.1.1.0 liblber-2.4.so.2 libaio.so.1 liblber-2.4.so.2.5.6 libaio.so.1.0.0 libldap-2.4.so.2 libaio.so.1.0.1 libldap-2.4.so.2.5.6 libanl-2.12.so libldap_r-2.4.so.2 libanl.so.1 libldap_r-2.4.so.2.5.6 libasound.so.2 libldif-2.4.so.2 libasound.so.2.0.0 libldif-2.4.so.2.5.6 libattr.so.1 liblvm2app.so.2.2 libattr.so.1.1.0 liblvm2cmd.so.2.02 libaudit.so.1 libm-2.12.so libaudit.so.1.0.0 libmount.so.1 libauparse.so.0 libmount.so.1.1.0 libauparse.so.0.0.0 libm.so.6 libblkid.so.1 libncurses.so.5 libblkid.so.1.1.0 libncurses.so.5.7 libBrokenLocale-2.12.so libncursesw.so.5 libBrokenLocale.so.1 libncursesw.so.5.7 libbz2.so.1 libnih-dbus.so.1 libbz2.so.1.0.4 libnih-dbus.so.1.0.0 libc-2.12.so libnih.so.1 libcap-ng.so.0 libnih.so.1.0.0 libcap-ng.so.0.0.0 libnl.so.1 libcap.so libnl.so.1.1 libcap.so.2 libnsl-2.12.so libcap.so.2.16 libnsl.so.1 libcidn-2.12.so libnspr4.so libcidn.so.1 libnss_compat-2.12.so libcom_err.so.2 libnss_compat.so.2 libcom_err.so.2.1 libnss_dns-2.12.so libcrypt-2.12.so libnss_dns.so.2 libcryptsetup.so.1 libnss_files-2.12.so libcryptsetup.so.1.1.0 libnss_files.so.2 libcrypt.so.1 libnss_hesiod-2.12.so libc.so.6 libnss_hesiod.so.2 libdb-4.7.so libnss_nis-2.12.so libdbus-1.so.3 libnss_nisplus-2.12.so libdbus-1.so.3.4.0 libnss_nisplus.so.2 libdevmapper-event-lvm2mirror.so libnss_nis.so.2 libdevmapper-event-lvm2raid.so libpamc.so.0 libdevmapper-event-lvm2snapshot.so libpamc.so.0.82.1 libdevmapper-event-lvm2.so.2.02 libpam_misc.so.0 libdevmapper-event-lvm2thin.so libpam_misc.so.0.82.0 libdevmapper-event.so.1.02 libpam.so.0 libdevmapper.so.1.02 libpam.so.0.82.2 libdl-2.12.so libparted-2.1.so.0 libdl.so.2 libparted-2.1.so.0.0.0 libdmraid-events-isw.so libpci.so.3 libdmraid-events-isw.so.1 libpci.so.3.1.4 libdmraid-events-isw.so.1.0.0.rc16 libpcre.so.0 libdmraid.so libpcre.so.0.0.1 libdmraid.so.1 libplc4.so libdmraid.so.1.0.0.rc16 libplds4.so libe2p.so.2 libply.so.2 libe2p.so.2.3 libply.so.2.0.0 libexpat.so.1 libply-splash-core.so.2 libexpat.so.1.5.2 libply-splash-core.so.2.0.0 libext2fs.so.2 libpopt.so.0 libext2fs.so.2.4 libpopt.so.0.0.0 libfipscheck.so.1 libproc-3.2.8.so libfipscheck.so.1.1.0 libproc.so libfreebl3.chk libpthread-2.12.so libfreebl3.so libpthread.so.0 libgcc_s-4.4.6-20120305.so.1 libreadline.so.6 libgcc_s.so.1 libreadline.so.6.0 libgcrypt.so.11 libresolv-2.12.so libgcrypt.so.11.5.3 libresolv.so.2 libgio-2.0.so.0 librt-2.12.so libgio-2.0.so.0.2200.5 librt.so.1 libglib-2.0.so.0 libSegFault.so libglib-2.0.so.0.2200.5 libselinux.so.1 libgmodule-2.0.so.0 libsemanage.so.1 libgmodule-2.0.so.0.2200.5 libsepol.so.1 libgobject-2.0.so.0 libss.so.2 libgobject-2.0.so.0.2200.5 libss.so.2.0 libgpg-error.so.0 libthread_db-1.0.so libgpg-error.so.0.5.0 libthread_db.so.1 libgssapi_krb5.so.2 libtinfo.so.5 libgssapi_krb5.so.2.2 libtinfo.so.5.7 libgssrpc.so.4 libudev.so.0 libgssrpc.so.4.1 libudev.so.0.5.1 libgthread-2.0.so.0 libutil-2.12.so libgthread-2.0.so.0.2200.5 libutil.so.1 libidn.so.11 libuuid.so.1 libidn.so.11.6.1 libuuid.so.1.3.0 libip4tc.so.0 libwrap.so.0 libip4tc.so.0.0.0 libwrap.so.0.7.6 libip6tc.so.0 libxtables.so.4 libip6tc.so.0.0.0 libxtables.so.4.0.0 libipq.so.0 libz.so.1 libipq.so.0.0.0 libz.so.1.2.3 libiptc.so.0 rsyslog libiptc.so.0.0.0 rtkaio libiw.so.29 security libk5crypto.so.3 tls libk5crypto.so.3.1 libkeyutils.so.1 xtables libkeyutils.so.1.3 [root@xlhost csf]#

Posted by egillette, 02-19-2013, 04:22 PM
Good point. . .

Posted by matbz, 02-19-2013, 04:23 PM
That would be on our unaffected machine. Let me go check some others and see if I can find why that one was installed. It was built in August 2012.

Posted by egillette, 02-19-2013, 04:26 PM
Understood, however, I know plenty who would disagree with that assessment, including a few datacenter techs. In any case, all the script does is simply attempt to identify the file, and then attempt to quarantine it so that if requested later, it can be provided for review -- try getting a bunch of users to do that *without* a script, especially considering some of them aren't as a technical as we are. For you, it may be counter-productive, but for those less technically-inclined, it's pretty helpful as I've been told quite a few times today and yesterday. In addition, most of the commands that would be run are in the plain text of the script: https://www.ericgillette.com/clients/exploit-cleanup

Posted by NetworkPanda, 02-19-2013, 04:27 PM
Hello everyone, I see that some of you requested a modification of the command I posted earlier today, so that it scans not only files, but also the sub-folders in /lib and /lib64 So here is the updated commands for scanning the sub-folders too. For 64 bit systems For 32 bit systems I did not find the command increasing the server load on any of the servers we used it and usually it completes within seconds.

Posted by matbz, 02-19-2013, 04:29 PM
Danger Will Robinson!

Posted by egillette, 02-19-2013, 04:30 PM
Try this: Sorry I didn't mention this prior -- was on the phone when I posted it.

Posted by Martin-D, 02-19-2013, 04:30 PM
Just to throw something in the mix here that I don't think has been identified/mentioned. I've come across a machine that was affected but there was no lib64/libkeyutils.so.1.9. The only files existing where lib64/libkeyutils.so.1.3 and the link lib64/libkeyutils.so.1

Posted by tarsiran, 02-19-2013, 04:33 PM
root@xlhost csf]# ls -lah /lib64/libkeyutils.so.1 -rw-r--r-- 1 root root 34K Dec 7 2011 /lib64/libkeyutils.so.1

Posted by matbz, 02-19-2013, 04:33 PM
That's very good of you. But may I respectfully suggest to readers that they run these commands on their boxes and post the results back here so the guys/community here can use it to narrow down the possibility of any other infected/exploit files, or confirm any false-positives?

Posted by TravisT-[SSS], 02-19-2013, 04:33 PM
Nice update NP! That will help because we are seeing other files named other then the 1.9 now being placed on boxes.

Posted by matbz, 02-19-2013, 04:36 PM
I think the right command is actually... ls -lah /lib64 | grep libkey AND ls -lah /lib | grep libkey

Posted by egillette, 02-19-2013, 04:37 PM
Fair enough. . .I'll include it in that case -- I just figured it would be a safer bet to have the user use tools that are better for this purpose. I want to make sure that we aren't giving folks a false sense of security -- which is why I mention it so many times and have posted disclaimers consistently. Anyway. . .some users directories may be larger than others, just keep that in mind everyone, and the possibility of false positive are also possible (i.e. pam_hulk.so which is cPanel's cPhulk library). Script modified to scan all directories as per NetworkPanda: I did however, leave the disclaimer in that it is *still* not a full scan of the user's machine, and they should perform one, since for all we know the file could originate from anywhere on the affected machine.

Posted by Martin-D, 02-19-2013, 04:38 PM
The only one we're finding on the only box that appears to have been affected. ....which of course is to be expected.

Posted by NetworkPanda, 02-19-2013, 04:40 PM
We are also getting this on all of our clean, not affected servers. So maybe this specific file is safe to ignore and does not mean that you are infected.

Posted by tarsiran, 02-19-2013, 04:41 PM
root@xlhost csf]# ls -lah /lib64 | grep libkey -rw-r--r-- 1 root root 34K Dec 7 2011 libkeyutils.so.1 -rwxr-xr-x 1 root root 10K Jun 22 2012 libkeyutils.so.1.3 ---------- 1 root root 34K Feb 19 21:45 libkeyutils.so.1.9 please help me how can i fix issue !

Posted by edlmkts, 02-19-2013, 04:41 PM
I have run your latest script, I have manually reproduced the effect of ldconfig in your script, see the way the links are different before and after running ldconfig? I reckon that's what happens in your script # ls -l libkey* lrwxrwxrwx. 1 root root 25 Feb 19 20:04 libkeyutils.so.1 -> /lib64/libkeyutils.so.1.3* -rwxr-xr-x. 1 root root 12592 Jun 22 2012 libkeyutils.so.1.3* ----------. 1 root root 34640 Feb 19 17:12 libkeyutils.so.1.9 # ldconfig # ls -l libkey* lrwxrwxrwx. 1 root root 18 Feb 19 20:35 libkeyutils.so.1 -> libkeyutils.so.1.9 -rwxr-xr-x. 1 root root 12592 Jun 22 2012 libkeyutils.so.1.3* ----------. 1 root root 34640 Feb 19 17:12 libkeyutils.so.1.9

Posted by Martin-D, 02-19-2013, 04:42 PM
Yep, that's part of cPanel's cphulk so it's a false positive. I just hit submit too early when responding

Posted by egillette, 02-19-2013, 04:42 PM
Unless I'm mistaken, it appears the file isn't symlinked on your machine. You may want to reboot your machine after this again. Let me know how that turns out for you.

Posted by matbz, 02-19-2013, 04:43 PM
It would be more helpful to the community to see the output of the various commands the community is putting up. It would aid in pinpointing other possibilities. It's great that you took so much time and effort to copy everyone's commands and put them together in a script, but it is clear (with the greatest respect) that you do not clearly understand all of them. And as we see with the chap who has Directadmin, it can bugger their boxes up. I hope you understand how useful it would be to get feedback from those who are infected.

Posted by matt1206, 02-19-2013, 04:45 PM
I've just ran the check command against my Cpanel VPS which doesn't appear to be affected, however, I've got two extra results as well as the pam_hulk.so Last edited by matt1206; 02-19-2013 at 04:49 PM. Reason: added code tags

Posted by jalapeno55, 02-19-2013, 04:46 PM
At this point I'm leaning with Steven however, Igor from CloudLinux seems to be leaning towards a possible OpenSSH exploit: http://www.cloudlinux.com/blog/clnews/sshd-exploit.php What's still missing is a spreadsheet with commonalities and a complete forensic investigation of some of the recently compromised hosts, particularly the ones getting reinfected.

Posted by egillette, 02-19-2013, 04:47 PM
So it seems that the actual symlink is being modified as well. I can put the immutable/undelete bits on that too, but honestly, I'm concerned that may not turn out so well -- especially when yum attempts to update the libraries later on. Anybody have good suggestions on how to address that issue -- I've executed the script more than 80 times now, and have had others execute it too, but this is the first time I've seen the issue that edlmkts is mentioning. Any ideas??

Posted by jalapeno55, 02-19-2013, 04:50 PM
And it would be better to watch that file with auditd

Posted by egillette, 02-19-2013, 04:52 PM
Actually the *only* commands I included were NetworkPanda's -- the rest of the script was written by me, and yes, I understand *all* of the commands -- I'm not sure how you arrived at the conclusion I didn't. My concern was not giving folks a false sense of security in running the script, since some will believe that the script will "fix" the issue, when in fact, many who have run previous versions of the script have been re-infected. As for the DirectAdmin guy, it seems in his case, a symlink was missing, which brings me back to what eldmkts mentioned about the symlinks being replaced so quickly (meaning the symlink might also be at risk of being replaced by the exploit). Let's stay on focus -- I agree with you that seeing the results that specific commands produce are helpful, and the script allows non-technical users to do exactly that, since they can cut/paste results here, instead of attempting to cut/paste commands they may or may not understand.

Posted by Orien, 02-19-2013, 04:54 PM
So libkeyutils.so.1.9 is not the only exploited file now?

Posted by jalapeno55, 02-19-2013, 04:54 PM
Steven have any of the servers you been watching personally get reinfected?

Posted by matbz, 02-19-2013, 04:55 PM
Anyone non-technical shouldn't be using SSH into a box

Posted by jalapeno55, 02-19-2013, 04:56 PM
That file doesn't exist the hacker is creating it.

Posted by TravisT-[SSS], 02-19-2013, 04:56 PM
We have seen other files with other name variants placed on servers.

Posted by Orien, 02-19-2013, 04:56 PM
That's what I meant.

Posted by egillette, 02-19-2013, 04:58 PM
See matbz -- this is *exactly* what I was afraid of with the false positives produced by running a full scan of the sub-directories. Now that I've made my point. . .I guess we'll just have to agree to disagree on the points. I'll leave the script as-is, however it should be noted to everyone running the script (or the commands separately as per matbz) that some files, though not belonging to any package aren't necessarily exploited.

Posted by matbz, 02-19-2013, 04:59 PM
That has always been quite particular, because no one has yet determined how it was placed there in the first place.

Posted by egillette, 02-19-2013, 05:00 PM
In theory that is true, but in practice, I think we both know the truth -- I've had non-technical users on facebook contacting me about this all day who are logged into SSH, even though some of them wouldn't know BASH if it hit them upside the head. LOL! So I agree with you mate, just trying to avoid creating false-positives for folks who are already nervous.

Posted by matbz, 02-19-2013, 05:03 PM
You haven't made any point. It's not for you alone to determine what is or is not a false positive. Without the feedback it can't be determined. No one should run your 'script'. What this community has been trying to do is determine the source of the infection. Please let it continue it's work? The source may not even be in those folders but it is not such a bad place to start.

Posted by JohnCrowley, 02-19-2013, 05:04 PM
Using the 32-bit /lib check command on our RHEL 5 servers (not infected as far as we can tell) returns the following: False positives?

Posted by NetworkPanda, 02-19-2013, 05:04 PM
That's right, so here are the updated commands to bypass this false positive (they ignore the cPanel cphulk library). For 64 bit For 32 bit

Posted by egillette, 02-19-2013, 05:06 PM
Agreed! Let's kill the chatter and stick to posts that are directed to an eventual solution/discovery. So for new folks. . .please PM me with questions about the script, or if you run into snags like our friend with DirectAdmin did a bit ago. Anyone else coming here for the first time, please see the script here: https://www.ericgillette.com/clients/exploit-cleanup Feel free to run those command individually as suggested by matbz, or run the script in its entirety from your server using the below commands: After downloading the script: Make sure the md5sum reads: 4a0e4c2fe094cd9944d52bf0366599d3 If it does, then do this: Monitor your server to see if the file libkeys.so.1.9 returns (you can also re-run the script periodically over the next few days to accomplish this). Post any results you get after executing the commands listed or running the script here so that others can use the information to try and determine how this exploit occurred.

Posted by TravisT-[SSS], 02-19-2013, 05:08 PM
What we need is less talk about band-aid scripting. It's a false sense of security anyway. You basically were rooted. EDIT: now we have people running stuff assuming if something is not owned by a package it is a threat..

Posted by egillette, 02-19-2013, 05:11 PM
NetworkPanda -- good call, however, I won't include this new set of commands, only because matbz's original point was to produce a FULL SCAN of the sub-directories. Given the stir this thread has created, and the fact that some have mentioned that the exploit has taken on new file names on occasion, even with the possibility of false-positives, I don't think it's prudent to exclude any results based on what we know thus far.

Posted by NetworkPanda, 02-19-2013, 05:12 PM
Nobody said this, but if somebody finds a *libkeyutils* (whatever version) module not owned by a package, then it is suspicious, as it appears on all affected systems.

Posted by TravisT-[SSS], 02-19-2013, 05:14 PM
I was talking more in terms of other packages that are not named *libkeyutils*

Posted by tarsiran, 02-19-2013, 05:15 PM
dudes if i try to delete these files: LIB64=/lib64/libkeyutils.so.1.9 LIB64_13=/lib64/libkeyutils.so.1.3 LIB64_12=/lib64/libkeyutils-1.2.so problem will be solved? please help

Posted by matbz, 02-19-2013, 05:15 PM
Hooray to that! It's what is needed though, people giving their output and allowing the community to see what may be amiss, and advising them accordingly.

Posted by Steven, 02-19-2013, 05:18 PM
Matbz, except people are not smart enough to know whats real or whats not real on their own and go into panic mode on wht.

Posted by matbz, 02-19-2013, 05:19 PM
It's not a full scan, stop saying that! Giving people a false sense of security. People should run the commands and post the feedback for advice! Not run anyone's 'script' and post the output that is written into the script. Your script doesn't give output from the box, it returns text strings that you wrote based on conditionals.

Posted by egillette, 02-19-2013, 05:20 PM
That was my point. Thank you for stating it so well Steven. It seems you and I are in total agreement about that.

Posted by james2398, 02-19-2013, 05:23 PM
Hello, I'm not sure if this helps but, I have been rooted as well. A couple of weeks ago, there were thousands of emails being sent out from the server every minute. I have a server administration company (no names, I'm not here to flame) that 'fixed' the solution for me, and stopped the spam. The servers SSH port was changed, only one IP allowed in, and things felt normal. However, I see they are not. I've tried the script that both cloundlinux and eric here have provided. The file /lib64/libkeyutils.so.1.9 gets immediately created. Running stat /lib64/libkeyutils.so.1.9 gives me the same modified, created, and accessed dates as all the same, time/date being seconds after the file is deleted. This box is running CentOS 64 with cPanel + WHM, Dovecot. I'm not sure what may be happening with the progress of a fix, but thank you all for your diligence. Let me know if I can provide any information to help

Posted by matbz, 02-19-2013, 05:23 PM
They are posting the outputs and seeking advice, that is the right thing to do surely.

Posted by egillette, 02-19-2013, 05:24 PM
I meant to say scan of the sub-directories under /lib or /lib64 specifically, sorry for the confusion with the term "full scan". As I mentioned prior, folks are free to review the commands in the script and run them individually as per your advice, or they can run the script -- how about we let them decide what they'd like to do instead of you trying to tell them what is best for them and their unique circumstances/conditions? And the script does give output, just so you know, the only time it doesn't is if it *DOES NOT* find the original file (libkeyutils.so.1.9) that started this whole thread, since if the file is not found, there is a pretty good chance the machine is not affected, yet still I advise them to scan their machines anyway even if the file isn't detected. I think that's about as good as it's going to get -- you arguing a point about running the commands individually, or as part of the shell script I created doesn't change anything. So take a chill pill. . .relax. Let people decide what they prefer to do based on their varying levels of Linux experience, and then post the output here for inquiring minds. Fair enough? Last edited by egillette; 02-19-2013 at 05:27 PM. Reason: Clarified the "file" being found.

Posted by matbz, 02-19-2013, 05:29 PM
None of these modules have ownership... file /lib/modules/r1soft/hcpdriver-cki-2.6.18-308.1.1.el5PAE.ko is not owned by any package file /lib/modules/r1soft/hcpdriver.o is not owned by any package file /lib/modules/r1soft/hcpdriver-cki-2.6.18-308.11.1.el5PAE.ko is not owned by any package file /lib/modules/r1soft/hcpdriver-cki-2.6.18-348.1.1.el5PAE.ko is not owned by any package file /lib/modules/r1soft is not owned by any package file /lib/modules/2.6.18-308.1.1.el5PAE.ksplice-updates is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.pcimap is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.dep is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.ieee1394map is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.usbmap is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.ccwmap is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.isapnpmap is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.inputmap is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.ofmap is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.seriomap is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.alias is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE/modules.symbols is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.pcimap is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.dep is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.ieee1394map is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.usbmap is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.ccwmap is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.isapnpmap is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.inputmap is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.ofmap is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.seriomap is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.alias is not owned by any package file /lib/modules/2.6.18-308.16.1.el5PAE/modules.symbols is not owned by any package file /lib/modules/2.6.18-308.11.1.el5PAE.ksplice-updates is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.pcimap is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.dep is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.ieee1394map is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.usbmap is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.ccwmap is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.isapnpmap is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.inputmap is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.ofmap is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.seriomap is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.alias is not owned by any package file /lib/modules/2.6.18-348.1.1.el5PAE/modules.symbols is not owned by any package On all my boxes that do not have the compromise this is also present... file /lib/device-mapper is not owned by any package Same on all my uncompromised boxes as well. So your output looks fine

Posted by egillette, 02-19-2013, 05:30 PM
Touch base with Steven -- I think he's been trying to find a machine currently running the exploit, so that he can dig a little deeper.

Posted by Cyclon, 02-19-2013, 05:30 PM
You have mentioned few times that you have seen some other files added now, but I didn't noticed that you provided any details about that. Can you add some more details... Did you find some completely different libraries now or just libkeyutils.so.1 was linked to some new files?

Posted by NetworkPanda, 02-19-2013, 05:31 PM
By the way, your point is correct, but since currently none of the known antivirus/antimalware utilities (ClamAv, AVG For Linux, CXS, maldet etc.) scan and clean a system from this malware, we need to find our own temporary solutions to protect our servers. Of course as soon as the antivirus utilities start detecting and cleaning this malware, there will be no other "band-aid" solutions.

Posted by nofz, 02-19-2013, 05:33 PM
I am following this news closely for 2 days. I am in Linux Security for almost 15 years now and none of my 200+ CentOS (EPEL) servers are infected (yet). We expose SSH/MySQL/httpd/tomcat ports publicly on some servers, some are firewalled and some are not. We own our own /23 and not IP's from public hosting companies that might have delayed it a bit or perhaps because we dont run any thing known on port 80. Its all custom build web applications (java). Personally I dont think its an 0day remote SSHD exploit either, but it is good target to infect to maintain access or log passwords after you entered the server. I would almost expect it to be something on port 80. Problem is now everybody use those scripts to fix the infected SSHD daemon, but in fact its just matter of time and the server is hacked again. And its already happening. And in the meantime you can expect new versions out in the wild already. libkeyutils.so is old news. So what is everybody running on port 80? I am going to check out some honeypots later today to see if they caught anything.

Posted by FastServ, 02-19-2013, 05:33 PM
I'm siding with malware on an administrator's PC and/or privilege escalation until a box under my control is compromised or someone proves it otherwise. This thread is becoming very tiresome.

Posted by james2398, 02-19-2013, 05:33 PM
Thank you Eric, I think I will see if he comes back to this thread, I know he must be busy with clients that are paying for his valuable services. I appreciate your referral. I was a little worried, since it seems that this file is re-creating itself immediately. Are there any suggestions anyone may have for a scenario such as this? Thank you James

Posted by matbz, 02-19-2013, 05:34 PM
Absolutely not. No one should run your script. Everyone should post the raw output of the commands that have so far been determined to be appropriate to use to find this certain file, or that the community updates in light of further detailed output. Until someone determines the source of the compromise, the one that creates that file. And until it is determined that indeed the hack has not been updated it is not safe to assume that is the only file infected.

Posted by pjf5000, 02-19-2013, 05:35 PM
Thanks Steven and egillette and everyone else for all the time and effort they have put into researching this nasty in the wild rootkit. It is appreciated. I just spent the last hour reading all 40 pages of this thread, my eyes are now bleeding, my mouth is dry and I should get up and urinate at some point shortly. But I can't... as I manage many CentOS 5/6 boxes, all up-to-date running cPanel, this is quite horrifying. after reading every post here and other sites.... There are really no steps to take to try and block this, I've read the post saying it is possible SSHD related and to block SSH access, anyone else running out and blocking SSH completely? Feels like it may be a waste of time until we know more... Thanks again y'all, finally signed up after lurking for 10 years on this one.

Posted by tarsiran, 02-19-2013, 05:36 PM
dudes if i try to delete these files: LIB64=/lib64/libkeyutils.so.1.9 LIB64_13=/lib64/libkeyutils.so.1.3 LIB64_12=/lib64/libkeyutils-1.2.so problem will be solved? please help

Posted by matbz, 02-19-2013, 05:37 PM
Good idea to run something like TCPView on an admin's PC that is communicating with a compromised box perhaps?

Posted by james2398, 02-19-2013, 05:38 PM
I can completely understand and relate to your situation as well, this is quite horrifying, given that the servers it is infecting seem to be coming from all backgrounds, regardless of the software being current on them. I myself am hoping for a solution to this scenario

Posted by matbz, 02-19-2013, 05:42 PM
There are no confirmed instances where passwd auth for root is set to no, i.e. it is disabled in favour of using ssh keys.

Posted by nofz, 02-19-2013, 05:44 PM
You should always prevent public access to SSH, but I doubt its a SSH exploit. If it is, it should be spreading way faster and more consistent. Some people are saying even hardware firewalled servers have been compromised. Too bad my honeypots havent caught anything yet.

Posted by Orien, 02-19-2013, 05:49 PM
Have there been any connections made between this and all the local Java exploits recently?

Posted by nofz, 02-19-2013, 05:50 PM
Still need a point of entry. And wasnt those java exploits only a problem on workstations?

Posted by Steven, 02-19-2013, 05:52 PM
Are you talking about on users desktops? if so I think flash is a bigger culprit: http://www.cvedetails.com/vulnerabil...sh-Player.html

Posted by Orien, 02-19-2013, 05:53 PM
Yeah, both Java/Flash on user computers. I believe that was your theory right?

Posted by tomfrog, 02-19-2013, 05:53 PM
Would they be targeting server admin / owners? No. Considering the number of infected servers in such a short period, wouldn't you expect a correlation with site hacks? There are more site admin / owners than server admin / owners. Have you seen a sudden increase in site hacks over this period or just the normal activity? Do you have to be root to send spam? No.

Posted by matbz, 02-19-2013, 05:53 PM
Not yet. Is there an SSH client that uses it though? Knowing what everyone who has rooted boxes use to connect via SSH or via browser whilst logging in as root would be useful though. Like seeing the output of Hijackthis perhaps?

Posted by tarsiran, 02-19-2013, 05:54 PM
Dear Steven if i try to delete these files: LIB64=/lib64/libkeyutils.so.1.9 LIB64_13=/lib64/libkeyutils.so.1.3 LIB64_12=/lib64/libkeyutils-1.2.so problem will be solved? please help

Posted by nofz, 02-19-2013, 05:55 PM
Even if those CPs uses java for a SSH terminal, you are still connecting to your own server. So unlikely.

Posted by james2398, 02-19-2013, 05:56 PM
Tarsiran, I've tried running the two scripts that have been provided here, one by Eric, and the other by cloudlinux, and it does not seem to resolve the issue (I believe it deletes the files). The lib64/libkeyutils.so.1.9 file re-creates itself immediately after being deleted on my server.

Posted by tarsiran, 02-19-2013, 05:59 PM
oh it's so danger so what should we do right now ?

Posted by egillette, 02-19-2013, 06:00 PM
Well, sorry but again, we'll just have to agree to disagree -- I have clients to answer to who in turn have their own clients to answer to, and I'd rather they be able to execute something and get some kind of feedback to me. In fact, I've already had clients attempt to cut and paste the commands from this very thread, and some of them have done so incorrectly causing further problems (one guy symlinked to the wrong file because he concatenated the last part of the command), after he rebooted his machine. So sorry, but I'd rather a non-technical user execute my script that will *definitely* run the proper commands, rather than risking them pasting the commands incorrectly or typing them incorrectly. Your problem is, you think everyone has the same level of technical expertise as we do, but as Steven pointed out, some people do not, and all it does is cause further panic and problems. So again, you do things your way, and I'll do things mine -- however, unlike you, I won't try to presume that everyone here is as technical as we are, and will give them the benefit of providing a script they can execute and provide feedback from. Ironically, you've also mentioned a dozen times now that they should "execute the scripts in this thread individually" and such, yet you provide no way for them to do so, so I'm assuming you expect them to read through all 40 pages. At least I've been helpful in that regard by putting all the commands somewhere for them -- what exactly have you done besides try to tell non-technical users what they should do, and encourage false-positive identification to cause more panic among said users? As Steven mentioned, NOT EVERYONE KNOWS what files can be safely ignored or not, and though NetworkPanda proposed excluding a specific cPanel file, I suggested we not go that route, since the whole idea was to show results for the sub-directories as well, irrespective if they are false positives so that users in turn at least have ALL the output. Yes, some will come here and panic based on your recommendation, but I believe that them having all the possible output they could have was a good idea, and agreed with you. Now you're saying non-technical users should not run a script (too late for that anyway since most of them already have), and instead try to cut/paste commands based on YOUR advice?? I think you were better off leaving it alone, and simply allowing users to do whatever they feel most comfortable with given their level of experience -- when you come across trying to tell people how they should do things irrespective of their level of experience, it actually can have the affect of creating bigger problems than the one we're attempting to solve here as in the instance I mentioned above. Again, remember, not everyone is highly technical -- some reading this thread are the total opposite, so your advice isn't prudent, has the potential to create other problems, or panic. Or perhaps as your original notion suggested (non-technical users shouldn't be logging to SSH), perhaps you'll also attempt to suggest that non-technical users shouldn't read this thread or participate in this forum too. Get real man. Let people decide what's best for them. Last edited by egillette; 02-19-2013 at 06:05 PM.

Posted by matbz, 02-19-2013, 06:01 PM
If you are being re-infected please talk to Steve about this, he has ideas about trying out different RPMs

Posted by FastServ, 02-19-2013, 06:01 PM
Why not? With the recent outbreak in Java and Flash exploits I'd say it's been easier than ever over the past couple months to launch an effective campaign. No increases over here. If you want to send spam completely under the radar bypassing logging and SMTP tweak restrictions in control panels like Cpanel, you surely want root.

Posted by tomfrog, 02-19-2013, 06:02 PM
Keel calm. Be patient. There's no definitive fix for this yet. Keep reading this thread.

Posted by egillette, 02-19-2013, 06:02 PM
James after you ran the script, did you immediately check for the file again, or did you reboot?? Can you describe your steps in detail starting from when you ran the script?? Times/Dates are helpful too if you have them. Also have you run the latest version of the script (today)??

Posted by matbz, 02-19-2013, 06:04 PM
Ugh! I'm not going to even bother reading that. What this community need sin order to help others is detailed output from the commands that it has collectively put together so far, based on input so far. Your script does not help. Maybe if you would contribute to helping others rather than focusing on encouraging people to run your script which does not return terminal output then we would progress. I have no intention of bogging this thread down with any more wasteful talk with you. Moving on.....

Posted by Darkimmortal, 02-19-2013, 06:05 PM
Can you elaborate?

Posted by tomfrog, 02-19-2013, 06:06 PM
Put your self in their place. You infect 200 000 computers. How many site owners? How many server owners? Why just dump the site owners info? Why not use it? And where's the increase in site hacks? It's missing. No? I have a open mind... But isn't it missing?

Posted by james2398, 02-19-2013, 06:06 PM
Hey Eric, Thank you for your help, as well as what you mentioned above about on behalf of some who may not be a tech-savy as others. I just ran the command, this is what I got: I am currently rebooting, and then will follow further instructions as detailed in your file and post results here

Posted by Steven, 02-19-2013, 06:07 PM
1.) You don't want to delete /lib64/libkeyutils.so.1.3 or /lib64/libkeyutils-1.2.so 2.) It will temporarily fix the problem. 3.) You will want to reinstall the OS and install clean backups eventually once we find the real cause.

Posted by PLE, 02-19-2013, 06:08 PM
Have someone already checked the cron jobs on one of the compromised servers?

Posted by Cyclon, 02-19-2013, 06:09 PM
Can you please add some details...

Posted by matbz, 02-19-2013, 06:10 PM
If you reply to a post that Steven has made he may be better placed to see your query. But this on the cPanel forums does describe steps on what to do if you find you have the infected file. Although it may be helpful to move the infected file to an empty folder, chmod it chmod 000 and set the immutable and undelete bits chattr +iu and let us know here so that some of the more technical guys here can look at it and see if it is mutating. Some may want to decompile it, some may want to see the strings output etc, etc.

Posted by Martin-D, 02-19-2013, 06:10 PM
nothing found and all scripts listed in cron were found to be clean. That said, the server in question hasn't been reinfected.

Posted by Arieh, 02-19-2013, 06:11 PM
I've tried to read as much as humanly possible in this thread but it's hard with all the fighting and endless rants. Maybe it is useful if someone could summarize what things are certainly not the cause and what options are still open, so we can all close those options and narrow it a bit down. What I've understood so far is that SSH is not the source of infection as people have it firewalled. Then the only options would be: - any of the public services: apache/ftp/mail - hacked sites executing scripts - ? Then there must be either a bug in the public daemons or what sites can access. I wonder for instance if maybe perl/python are available on infected machines? What php/apache mods: suphp/php-fpm/mod_ruid etc Also I was wondering if any of the people investigating right now are serious security specialists and one would wonder if RHEL/CentOS devs are on this as well? This all sounds very serious. Personally I'm only a small Debian user.

Posted by nofz, 02-19-2013, 06:14 PM
I havent checked out what those scripts do, but I advise to change root password after running it and reboot, apparently root password is sent using port 53.

Posted by jestep, 02-19-2013, 06:21 PM
The summary on the first page is still correct. Only RH systems affected. Likely not apache related since there's some reports of servers running lightspeed being compromised. There's also reports of fresh vanilla installs being compromised in a few hours, so not likely a hacked or malicious user / site. So far there hasn't been any technical evidence of how the system is being compromised in the first place, just what happens after it's compromised. Last edited by jestep; 02-19-2013 at 06:29 PM.

Posted by Hellsheep, 02-19-2013, 06:22 PM
Good morning! So here's the latest from my compromised server: Last night I decided to try band-aid the issue and see if the server gets re-infected. I did the whole process of deleting the libkeyutils symlinks, the dodgy library, etc etc. Works great of course to band-aid the issue. I updated the kernel to 2.6.32-279.22.1.el6.x86_64 Updated cPanel to 11.36 (yes I know this isn't meant for production servers but it has a a number of features which I wish to use) I can still see the exim version is 4.80. I disabled password auth in SSH. I changed all my SSH keys, made brand new ones, changed password for WHM. I deliberately left SSH open to the world to see what would occur. Enabled verbose logging for sshd, see this in /var/log/secure: Feb 20 02:57:19 sshd[26951]: Failed publickey for root from port 62814 ssh2 Feb 20 02:57:19 sshd[26952]: Connection closed by Feb 20 02:57:21 sshd[26950]: Connection closed by There is no trace of the infected files on the server after doing all the reboots and updates and security hardening however it's clear that they are trying to use the old ssh key. It would be safe to assume if you haven't changed your ssh keys and you've tried to clean the server, you've probably been re-infected for that reason. That IP address is also ovh. http://whois.domaintools.com/ Confirmed it's a cPanel box also, appears to be compromised.

Posted by nofz, 02-19-2013, 06:24 PM
egillette, can please also advise people in your script to change their root password after succesfully running your script and reboot?

Posted by matbz, 02-19-2013, 06:28 PM
Dude, don't use the script, it may bugger up your box.

Posted by Scott.Mc, 02-19-2013, 06:34 PM
I have been reading this thread and have remained silent because it's filled with many more posts that contribute nothing however I absolutely have to second this comment. @egillette you are actually becoming rather annoying in this thread constantly being involved in fights and continually writing essays to defend how much of an "expert" you are. It just goes to show how sub-par the technical expertise are around these parts. I could write an entire post dissecting your "cleanup script" and on how poor your scripting knowledge is - it's laughable at best and absolutely frightening that people will actually run this. Hint: Do not simply rename a library and expect to be "clean". Now specifically to @egillette I ask you stop this nonsense. The next time you start writing several paragraphs discussing someone else or your "expertise" simply close the tab before hitting reply. You are discouraging the people that actually can contribute from participating. Totally in agreement. Given the fragmented types of installations if there is no common daemon it is most likely password. The thing is around these parts it's always made incredibly complicated when it's mostly the simplest of sources that end up being the source. In recent history it was providers helpdesk (perldesk) and more specifically a budget provider basically having the passwords taken - yet this was "remote openssh vuln". Then it was WHMCS if you recall the "sanz" or whatever other ridiculous name the target was again that was "remote openssh vuln", in fact we even had hostgator chime in that they had "patched" this by disabling the aes*-cbc ciphers???? They really should have been shamed for this, why aes*-cbc be exploitable but not aes*-ctr ? This is another company you shouldn't trust, constantly trying to make headlines with patches for things they actually don't even the technical know-how to patch, they didn't even contact the openssh developers yet released a "patch" for what exactly????). You see it's not exactly unusual to have people making terrible predictions and assuming things are much more complicated than they actually are. Moving on if you have this problem, - Don't run some random script you see here - Don't rename a library and expect to be "clean" - If you are going to "clean" yourself replace libraries and binaries from the source repos with verified checksums. - Make a note of your setup and versions and simply post them which will allow for conclusions to be made - Check the logs, even a simple strings on the partition and match "sshd.*Accepted" to see if there was prior password/key logins.

Posted by Mat Sumpter, 02-19-2013, 06:35 PM
So I used Domaintools' reverse IP lookup and found two domains operating on that IP address and one of those happened to have a phpinfo() in the root of the domain: http://fiestas10.com/phpinfo.php Seems to be a CentOS 5 server running cPanel and has curl 7.24 which is within the exploited range.

Posted by matbz, 02-19-2013, 06:38 PM
Interesting, since if root passwd auth was disabled and an attempt was made then before a password prompt sshd would look for a private key on the system trying to log in, and if not there would simply close the connection and say "No supported authentication methods available"

Posted by streaky, 02-19-2013, 06:42 PM
Prohacking with the domaintools there. Seriously though this thread is a massive pile of wtfsauce...

Posted by gatorpatrick, 02-19-2013, 06:44 PM
Disabling those ciphers was for an unrelated SSH vulnerability from years ago (2008-2009 maybe?). If you are going to call us dishonest please at least be accurate in your accusations. If we are going to patch something, we'll perform the following (as many SW vendors will happily attest to) 1) Contact developer and provide our investigation & evidence 2) Create patch 3) Once developer has had a chance to distribute own patch we post to bugtraq to inform the community and provide our patch freely to all who ask. If there's anything in there that deem us untrustworthy, so be it. We are not the technical end-all be-all and there are many teams in the industry just as good (maybe better) than the HG team. However if we are going to release data on something and maybe put the script kids on the warpath we're going to do our damndest to at least make sure we're accurate and we understand the data -- and that releasing it will do more good than harm by getting the word out to our colleagues in the hosting community. Finally, to answer your technical question as to why CBC might be vulnerable but not other ciphers, simply read the following: http://lwn.net/Articles/307873/ EDIT: Once a server is rooted, no binary or log file on it may be trusted. As such, to be sure the servers integrity you need to do the following: Backup user data. OS Reload. Shore up firewall. Make sure system is updated & updating regularly. Restore user data. Ensure mod_sec or whatever equivalent you like to use is using an up to date ruleset. Watch the logs closely. Last edited by gatorpatrick; 02-19-2013 at 06:52 PM.

Posted by james2398, 02-19-2013, 06:45 PM
Eric, thank you but I think I will step away from this thread, honestly, like many have mentioned, it is ridiculous the way people have been behaving. Thank you to the community for your passion and dedication to the web hosting industry. Times have changed. I also thank my own server administrators, they know who they are, for their continued support. I hope that everyone in here comes together and comes up with a solution, I believe we are all united on that front. Or I would hope so. James

Posted by coldbeer, 02-19-2013, 06:47 PM
Hahaha +1 Lucky I am not infected.

Posted by Disconnected, 02-19-2013, 06:49 PM
Does anyone have a copy of the 32bit library or have we only seen 64bit versions infected?

Posted by coldbeer, 02-19-2013, 06:49 PM
Both are vulnerable

Posted by Scott.Mc, 02-19-2013, 06:53 PM
Actually this "patch" was released at the exact same time as the "sanz" thread. So I will certainly call it out because it does relate to this. It's always overblown with people who should know better trying to make headlines. For the certain thread I referenced even one of your own staff told me a few days later that it's likely Dave was told to come up with a "patch" for publicity and to save face for disabling ssh for all users for nothing and then looking stupid. Now I am sorry I am doing exactly what my main reason for posting here was and taking the thread on a side-track of unrelated facts. Feel free to have the last word and anything else we can take to PM or another thread.

Posted by Disconnected, 02-19-2013, 06:53 PM
If anyone has a copy of a 32bit infected file, I would like to examine it.

Posted by Orien, 02-19-2013, 06:56 PM
Let's keep discussion focused on figuring out the source of this exploit, please. If you have personal comments directed at a specific person that don't really benefit anyone else, PM them instead. Thanks.

Posted by Scott.Mc, 02-19-2013, 07:01 PM
For those that have been compromised what services do you use that store your root password/keys? Maybe there is a single source either currently in use by the same group or previously. Examples are things like billing software, remote server monitors, providers, management companies?

Posted by egillette, 02-19-2013, 07:04 PM
Here you go my friend. Unfortunately this was drowned out awhile back by un-necessary chatter from some as you have clearly seen: http://www.gsol.us/libkeyutils.so.1.9 That is a CentOS 5.x Plesk 9.5.x machine. There are Plesk 10-11.x machines I have that were infected too, however, they are both 64-bit systems. The DirectAdmin machine's lib was deleted before I had a chance to decide on quarantining it or not, early on. . .but the file hasn't come back on that machine at all. I took many of the actions described in this thread (changing SSH keys, firewalling SSH off, etc).

Posted by jalapeno55, 02-19-2013, 07:05 PM
Have we ruled out FastCGI? I've seen that a number of times now on exploited servers.

Posted by gscript, 02-19-2013, 07:05 PM
Summary of a system where I have experienced this: Linux kernel 2.6.32 CentOS 5.9 64-bit VPS cPanel/WHM 11.34.1 curl 7.15 OpenSSH_4.3p2 OpenSSL 0.9.8e-fips-rhel5 httpd 2.2.22 dovecot 1.2.17 exim 4.80 SSH passwd auth for root allowed Exposed to internet as web, mail, and name server Discovery: Notification from lfd including a link to this forum. No unusual outbound traffic. Response: Remove rogue file. Run updatedb. Reboot server. Disable SSH daemon. Change passwords. 24 Hours Later: No recurrence. I changed the passwords last, recognizing that this threat could capture the new passwords while still occupying libkeyutils. The 'last login' reported for root did not indicate an unknown login; I don't know much about the usefulness of this certain measure. I've read and followed this entire thread. I'm most suspicious of curl, sshd, and cPanel at this point (unfounded maybe but makes me feel better). In other words, I'm glad to have sshd turned off for the time being. Feel free to ask for any additional details. Last edited by gscript; 02-19-2013 at 07:18 PM.

Posted by matbz, 02-19-2013, 07:08 PM
I originally set the box up ... "Installing libssh2-1.2.2-7.el6_1.1.x86_64" A yum update installed though.. My mistake, this box was compiled 25th April 2012. This version is listed in the repo rpmforge and I have rpmforge.repo in /etc/yum.repos.d libssh2 is not even installed on any of our other boxes. *no compromised boxes here*

Posted by Phil @ NodeDeploy, 02-19-2013, 07:10 PM
Looks like it's not only centos specific... Debian Squeeze w/ Xen 4.0 root@vhost01 /lib64 # lsof libkeyutils.so.1.9 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 2637 root mem REG 253,0 27928 1052300 libkeyutils.so.1.9 named 16032 bind mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 18226 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 19164 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 21628 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 21629 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 25176 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 25177 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 26025 root mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 26558 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 26560 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 26564 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 apache2 28205 www-data mem REG 253,0 27928 1052300 libkeyutils.so.1.9 root@vhost01 /lib64 # locate libkey /lib/libkeyutils.so.1 /lib/libkeyutils.so.1.3 /lib/libkeyutils.so.1.9 root@vhost01 /lib64 # ls -l libkey* lrwxrwxrwx 1 root root 18 Nov 16 22:05 libkeyutils.so.1 -> libkeyutils.so.1.9 -rw-r--r-- 1 root root 8.4K Apr 3 2010 libkeyutils.so.1.3 -rw-r--r-- 1 root root 28K Apr 3 2010 libkeyutils.so.1.9 root@vhost01 /var/www # wget -qq -O - http://www.cloudlinux.com/sshd-hack/check.sh |/bin/bash The server is compromised, /lib64/libkeyutils.so.1.9 found Doesn't appear to be any traffic coming out of the machine that's abnormal though... anyone else?

Posted by jalapeno55, 02-19-2013, 07:13 PM
Was PHP running with FastCGI by any chance?

Posted by iseletsk, 02-19-2013, 07:15 PM
Phil, Could you provide version of OpenSSH on that server?

Posted by afletch, 02-19-2013, 07:16 PM
If you go back around page 20 or so, you'll see a couple of posts from Brad (grsec). He gives some code you can compile which will de-obfuscate the library and you can then run strings on it, to see if you are really running a compromised lib (since you say you see no unusual outbound traffic). Can you run us through what this Debian box does, what the SSH config is (password, keys, ports, firewalling, etc). Does it run public-facing websites? Control panels? Details all useful.

Posted by Phil @ NodeDeploy, 02-19-2013, 07:23 PM
We're running OpenSSH_5.5p1 Debian-6+squeeze2, OpenSSL 0.9.8o 01 Jun 2010 Linux vhost01 2.6.32-5-xen-amd64 #1 SMP Sun May 6 08:57:29 UTC 2012 x86_64 GNU/Linux The system is an xen hypervisor it runs cloudmin but it is not accessible to the public, you must vpn into the office to be able to get at it (we hardware firewalled it off to only the office IP) sshd auth is via public key only.

Posted by Petabyte, 02-19-2013, 07:24 PM
Anyone suspected WHMCS in this thread? Please somebody confirm that some of compromised servers didn't have access from WHMCS. You know, WHMCS is storing all root accesses for accounts setup: keys, ports, password etc.

Posted by Martin-D, 02-19-2013, 07:25 PM
No, it's not WHMCS related.

Posted by Hellsheep, 02-19-2013, 07:27 PM
I do use WHMCS and my box was compromised however I don't think it's WHMCS related.

Posted by gscript, 02-19-2013, 07:27 PM
FastCGI is not in use, or even installed as far as I can tell. PHP is in use with WordPress instances. PHP was configured to use suphp as handler and suexec was enabled, although I have since changed both.

Posted by FastServ, 02-19-2013, 07:28 PM
This narrows down the attack vector considerably. *going back to infected workstation theory We already know the infection involves key auth in some form (see previous log posts where a freshly installed system after infection shows key failures). Might want to start looking more closely at /root/.ssh/authorized_keys* on these infected boxes and taking network captures on all your workstations. Last edited by FastServ; 02-19-2013 at 07:34 PM.

Posted by Phil @ NodeDeploy, 02-19-2013, 07:30 PM
Doubtful.. we don't have any windows machines in the office, unless it's something mac/linux based? Edit: Plus it's only 2 people with ssh keys to this machine

Posted by serve-you, 02-19-2013, 07:31 PM
If it's related to something like java/adobe then these would exist potentially on any platform.

Posted by Patrick, 02-19-2013, 07:35 PM
Apple admitted they were owned recently with Flash/Adobe attacks and they were using... Macs, obviously. It was in the news earlier, go Google it.

Posted by Hellsheep, 02-19-2013, 07:37 PM
For what it's worth, yesterday after Steven had suggested the possibility of a locally compromised machine, I ran scans and this was detected on the machine I connect to my linux server from: http://www.microsoft.com/security/po...r:Java/Toniper It was removed/cleaned before I made any changes to the linux server, the libkeyutils.so.1.9 was removed last night after all that and some additional hardening (see previous post) and I have yet to be re-infected. Although there are still random ovh IP addresses (all appear to be previously listed in this thread) that were trying to ssh using public key's last night.

Posted by Zadmin, 02-19-2013, 07:38 PM
you can use this to de-obfuscate the library and check strings in it :

Posted by FastServ, 02-19-2013, 07:38 PM
FWIW, it would appear phil@nodedeploy uses WHMCS.

Posted by Steven, 02-19-2013, 07:38 PM
Phil @ NodeDeploy Still doubtful? http://www.cnn.com/2013/02/19/tech/w...ked/index.html

Posted by Phil @ NodeDeploy, 02-19-2013, 07:40 PM
I guess not

Posted by tarsiran, 02-19-2013, 07:42 PM
apple hacked by this !!!? http://www.cnn.com/2013/02/19/tech/w...ked/index.html

Posted by Phil @ NodeDeploy, 02-19-2013, 07:43 PM
Debian is most definitely affected also: %s %s %sg:%sgshd:1 %s %s %sssh:1 %s %s %s %dMD5_InitMD5_UpdateMD5_FinaloptionshostaddrdeflateInit_deflatedeflateEndinflateInit_inflateinflateEnd1.2.1.0audit_log_user_messageaudit_log_acct_mess ageabcdefghijklmnopqrstuvwxyzbiz.info.net

Posted by Patrick, 02-19-2013, 07:46 PM
Everyone is so focused on a complicated scenario here... even myself initially until Steven brought up local PC infection and then it all started to make sense. You guys have to think about it, if there were zero day exploits in OpenSSH or MySQL or BIND - the only daemons that most of those compromised servers had in common, there would be a crap load of more infections being reported. Keep It Simple Stupid... I could be wrong, we could all be wrong, but I think Steven is correct in assuming that this is a local PC infection with some malware stealing SSH credentials. There is just nothing to suggest right now that this is a local exploit or a remote exploit against a particular daemon. Sure it's very possible but so unlikely given the wide range of OS's and firewall settings and all that jazz. ... and no, it's not PHP and it's sure as hell isn't cURL nor is it WHMCS. People without a clue, while good intentioned, need to stop grasping at straws and making other people panic. Remember kids, panic kills! :X

Posted by egillette, 02-19-2013, 07:51 PM
Speaking of. . .I just updated my Adobe Acrobat Reader. . .FireFox complained about me using an outdated version that could be compromised after I updated FireFox a few minutes ago. But then the only thing that seems odd to me about that scenario is that why only 2-3 servers hacked with the exploit -- why not all 122 I manage in that case?? Just playing Devil's Advocate. I guess it could go the other way too -- because 2-3 servers out of 122 would be less noticeable perhaps?? I don't know. Just thinking about it as logical as I can.

Posted by Patrick, 02-19-2013, 07:54 PM
I don't know, maybe their malware sucks? None of this makes any sense and we can grasp at straws all we want, but until someone gets a packet capture of an attack and confirms something is 100% going down outside of a local PC infection... then we are just wasting our time.

Posted by dicoemil, 02-19-2013, 07:56 PM
If somebody has an infected machine, I would like a peek a bit too

Posted by The3bl, 02-19-2013, 07:57 PM
Just curious does anyone have a copy of the spam they are sending?

Posted by Scott.Mc, 02-19-2013, 07:58 PM
Maybe you have firewalls on those servers or the auth is different/not stored? Do you store the auth or have shared the auth for those servers anywhere else that you can think of such as billing systems / other vendors? If they are customer servers maybe the customer turn leaked the password (if they had it) or a system / vendor the end customer ? Given you have a larger number of infections it might be possible to narrow it down even more. If this thread is to be believed there appears to be too many different system types, versions and services to make this plausible to be anything but some sort of password/auth leak.

Posted by tarsiran, 02-19-2013, 08:02 PM
so now what is the best way to control or stop working temporary this exploit?

Posted by hb9aj4fn, 02-19-2013, 08:07 PM
They best advice may be that you first scan your home computer and remove any virus you may find, then take backup using DirectAdmin, then rebuild your server and import backup into DirectAdmin. No matter what you do, please let us know if you find any virus on your home computer.

Posted by Orien, 02-19-2013, 08:07 PM
Great question.

Posted by nofz, 02-19-2013, 08:07 PM
If everyone would post some basic info of their hacked server. For example; #netstat -lnptu tcp 0 0* LISTEN 10760/mysqld tcp 0 0* LISTEN 13204/nginx tcp 0 0* LISTEN 1273/master tcp 0 0* LISTEN 9249/sshd tcp 0 0 ::1:25 :::* LISTEN 1273/master tcp 0 0 :::22 :::* LISTEN 9249/sshd # rpm -q openssh-server curl exim dovecot httpd openssh-server-5.3p1-81.el6_3.x86_64 curl-7.19.7-26.el6_2.4.x86_64 package exim is not installed package dovecot is not installed httpd-2.2.15-15.el6.centos.1.x86_64 rpm -q should be extended with more suspected packages I seriously doubt about the local workstation theory. You cant hijack SSH session, so then what? Infect putty.exe? Keylog? Dont think so.

Posted by Hellsheep, 02-19-2013, 08:09 PM
Keylog the local workstation, wait for someone to type hostname/IP address and root username/password. Not too hard really.

Posted by Steven, 02-19-2013, 08:11 PM
http://en.wikipedia.org/wiki/Gumblar It existed for ftp passwords, plz explain why ssh is impossible.

Posted by Patrick, 02-19-2013, 08:11 PM
Key logger for password authentication... grab the keys from the hard drive for key auth. It's insanely plausible and frankly seems like the most likely scenario at the moment. Edit: Could even use the local PC as a remote desktop or something to connect to the servers... anything is possible.

Posted by FLDataTeK, 02-19-2013, 08:13 PM
Since Steve's theory is possible desktop compromise what about a Java exploit? Java runs on everything including Mac... And there have been major holes in Java lately. Also if its a java exploit it could be exploited on servers that run java. Just throwing that out there since even limited access machines have been compromised.

Posted by nofz, 02-19-2013, 08:14 PM
Dont want to go into details here, but doesnt make much sense to me. They could have just infected everyone and send spam from the workstation instead. And why only Linux?

Posted by Disconnected, 02-19-2013, 08:15 PM
Here's a sample I saw: ######### 049F From: Happy Bingo 037T To: "[[REDACTED]]" <[[REDACTED]]> 081 Subject: Hello [[REDACTED]]. Casino Action - CA$1250 FREE + 1 Hour of Free Play! #########

Posted by dicoemil, 02-19-2013, 08:16 PM
Until somebody does not share an abused machine with the people which can debug properly, is a total waste of time waiting for a miracle...

Posted by The3bl, 02-19-2013, 08:20 PM
Do you know what site it was directing people to visit? Was it happy-bingo.info?

Posted by Disconnected, 02-19-2013, 08:22 PM
Unfortunately, I don't have a copy of the body saved. But it looks like bazhenovaa@yahoo.com owns a large number of similar domains. bingo-club.info was another domain referenced in spam messages.

Posted by barry[CoffeeSprout], 02-19-2013, 08:22 PM
The Java exploits that are making the news lately are generally in the horrible broken Java plugin (the part that loads applets) These are generally not found in Java on the server side But yes, please turn off applets if you can

Posted by FLDataTeK, 02-19-2013, 08:22 PM
Here is the whois on that domain.. Was just registered recently. Domain ID48937054-LRMS Domain Name:HAPPY-BINGO.INFO Created On:14-Jan-2013 18:54:10 UTC Last Updated On:26-Jan-2013 22:20:25 UTC Expiration Date:14-Jan-2014 18:54:10 UTC Sponsoring Registrar:eNom, Inc. (R126-LRMS) Status:CLIENT TRANSFER PROHIBITED Status:HOLD Status:TRANSFER PROHIBITED Registrant ID:b9142a1fc06c6721 Registrant Name:Alla Bazhenovaa Registrant Organization:na Registrant Street1:28/ app 407 Iskrovskiy pr, Sankt-Peterburg, Registrant Street2:--- Registrant Street3: Registrant Cityankt-Peterburg Registrant State/Province:Russia Registrant Postal Code:193232 Registrant Country:RU Registrant Phone:+7.9218677014 Registrant Phone Ext.: Registrant FAX: Registrant FAX Ext.: Registrant Email:

Posted by zildjian42, 02-19-2013, 08:23 PM
Not the best way to send spam these days considering most DSL/Cable IP addresses are blacklisted. Hosted servers are always on and not in those IP address ranges given out by Verizon, Comcast, Cox, or other ISPs used in residential and small business.

Posted by plexy, 02-19-2013, 08:25 PM
Rather odd this one. A friend of mine is asking me to look over his machines as i know his infrastructure well. All of the FUD floating about this thread aside, here is the information I can provide at this stage. No speculation here, just hard facts. So heres his setup; Three VM's, virtualbox host, all centos 64 bit (6.3 final) and all are kernel 2.6.32-279.11.1.el6.x86_64. All on the same subnet (public addressable). All VM's run WHM and CSF/LFD with the atomic modsec ruleset. No WHMCS. No addons. Only extra is NRPE plugin. He does allow password based login via port 22, but uses prety good passwords. One other machine, an ubuntu box which is based in his home - with no external ports open/forwarded - has a SSH key to perform nightly rsync backups from all three virtual machines. Host machine unaffected. Host machine runs no publicly accessible ports and is only open to my friends home IP and my IP. Virtual machines Two of the three WHM VM's are unaffected. The third is affected. All have same service versions and ports open. No logs on the third machine to indicate any kind of SSH based access, nor did LFD email alert to say SSH access had taken place. No alerts to indicate any kind of brute force. The software versions all seem the same (i havent delved too deep yet, but a cursory glance indicates this to be so). The main difference that I can see is the php version on the infected machine is 5.3.21, with XCache and the other two VMs run 5.3.18 without XCache. The infected machine does have a wider vareity of PHP software on there, such as (an up to date) magento, wordpress etc. But with atomic modsec ruleset in place, I place the risk level here to be low (but certainly not impossible). Other than that, the machines look identical. My friend has logged into all three of the virtual machines several times in the past few days from the same Mac based terminal. He does have flash enabled in his browser. Something I did note. LFD began warning from around 12 noon of the existence of libkeyutils.so.1.9. My friend logged in at 16:47 GMT via SSH, and ran the cleanup script at 16:50. The script did not remove the file, but did restart SSHD. At the point SSHD restarted, libkeyutils.so.1.9 was re-written to disk, both in /lib64 and in /tmp: ----------. 1 root root 32200 Feb 19 16:50 libkeyutils.so.1.9 from secure (when cleanup script was ran). Feb 19 16:50:39 vs3 sshd[19060]: Received signal 15; terminating. Feb 19 16:50:39 vs3 sshd[21857]: Server listening on port 22. Feb 19 16:50:39 vs3 sshd[21857]: Server listening on :: port 22. upon inspection of the 'cleanup' script (ericgillette.com/clients/exploit-cleanup) it does not take into account that libkeyutils.so.1.9 was set as immutable. chattr -i libkeyutils.so.1.9 was required on this certain machine, then the correct linking and ldupdate was needed. What stands out here is that the libkeyutils.so.1.9 files (in both /tmp and /lib64) were re-written at the exact moment SSHD was restarted. The infected machine has had the library removed, ld linkingupdated and services restarted. Kernel was updated to 2.6.32-279.22.1.el6.x86_6. There is one red flag that my friend mentioned - he recently (about 10 days ago) used a third party contractor to perform some work as root. This contractor had the login details to the machine that is affected. This may be a red herring but that always concerns me, expecially as some have posted in this thread asking if a third party contractor has recently worked on infected machines. As I said, no FUD here so I wont speculate, I appreciate the efforts and professionalism of those in this thread eager to crack this problem and offer my help (17 years experience, I also work for a top security firm with links to global Govs and Mils). Please folks, try to keep the FUD and speculation to a minimum if possible this is a a serious threat. Ive set up a packet capture of all traffic and will await to see if our little friend comes back. If it does, ill analyse what is captured and report back.

Posted by dicoemil, 02-19-2013, 08:31 PM
And nothing at all in the logs of the abused machine? The file just poped there from nowhere?

Posted by tecnobrat, 02-19-2013, 08:32 PM
plexy, can you paste a list of installed php modules and apache modules and versions and report any inconsistencies between the machines?

Posted by matbz, 02-19-2013, 08:33 PM
Would you mind running this on each of the 3 servers and post the results for each?...

Posted by plexy, 02-19-2013, 08:34 PM
doh root@vs3 [/etc/httpd/logs]# rpm -q openssh-server curl exim dovecot pure-ftpd openssh-server-5.3p1-81.el6_3.x86_64 curl-7.19.7-26.el6_2.4.x86_64 exim-4.80-5.x86_64 dovecot-1.2.17-0cp.x86_64 pure-ftpd-1.0.36-1.x86_64 root@vs3 [/etc/httpd/logs]# httpd -v Server version: Apache/2.2.23 (Unix) Server built: Feb 8 2013 07:37:17 Cpanel::Easy::Apache v3.18.0 rev9999 The other machiens run the same (well, apache build time is slightly different, but same version). Something that stands out is pureftpd is a different version on the unaffected machines: pure-ftpd-1.0.36-3.cp1136.x86_64 @ matzb, technobrat, give me a few mins. will do. @dicoemil - yep. nothing in logs i can see.

Posted by plexy, 02-19-2013, 08:43 PM
matzb: (sorry i need more posts for links on this forum) Infected: pastebin.com/8WVZdzrC Not infected: pastebin.com/SH2REZSz Not infected2: pastebin.com/Y326Zt4v

Posted by Patrick, 02-19-2013, 08:47 PM
It has NOTHING to do with Apache or PHP. Next question.

Posted by plexy, 02-19-2013, 08:50 PM
matzb To avoid confusion about kernel version and presence of libpcap, this was the yum updates and installs logged on the infected machine. These were done by my friend and libpcap by me. Feb 19 16:51:45 Installed: mlocate-0.22.2-3.el6.x86_64 Feb 19 17:56:27 Installed: kernel-2.6.32-279.22.1.el6.x86_64 Feb 19 23:21:13 Installed: 14:libpcap-1.0.0-6.20091201git117cb5.el6.x86_64 Feb 19 23:21:14 Installed: 14:tcpdump-4.0.0-3.20090921gitdf3cb4.2.el6.x86_64

Posted by The3bl, 02-19-2013, 08:51 PM
Of the machines infected sending spam at the time discovered was there any strange files in /tmp around a meg or more?

Posted by suhailc, 02-19-2013, 08:53 PM
Hi, So has anyone had a server hacked which had SSH password authentication disabled? Does anyone have any idea of how the privileges are escalated? What script is being used to do this?

Posted by dicoemil, 02-19-2013, 08:53 PM
He has any ftp accounts which are allowed to execute something with ssh? I saw a spamwave interesting attack few days ago with a stolen FTP user/password executed thru ssh somewhere... Interesting fact the logs are clean, would be good if you knew the date when file was created to check against http access, but if your friend ran the script to delete the infected file... If the attack was not via http, then is not a stupid attack and yes is v. serious threat

Posted by nofz, 02-19-2013, 08:57 PM
Too bad we cant make any good assumptions about why 1 out of 3 almost identical server is hacked and not the other 2. Its possible that the "autorooter" isnt sweeping consecutive ip ranges to avoid suspicion.

Posted by Ceetoe, 02-19-2013, 08:58 PM
Steven has done a great amount of work here and one of the things posited last night was the possibility of a local machine compromise. I do not think this has gotten any significant sample testing to at least narrow down what we are dealing with: Assumption 1: Spam net: Primary purpose of the rootkit is for spam purposes. If this is a bot-net for spam then the idea is to infect a sizeable number of machines but NOT ALL. Why? To avoid detection for as long as possible. Spam nets are meant to be longterm operations. If this is for pure theft of credentials for sensitive information then this assumption is invalidated. Assumption 2: Local keylogger: Servers are being compromised by local connection pcs. We have had several individuals with presumed fresh re-installs compromised immediately. Last night from WHT member TheVisitors Well doesn't it figure.... I install a fresh copy of CentOS 6.3 with cPanel current and in less than 2 hours.... BAM I had changed Ssh ports and made a pass code over 400 letters, numbers, and symbols long.... Still got despite all this. Little "tool" keeps changing my ssh log-in now. Can't turn it off, we need ssh. My IP changes often.... Daily sometimes hourly.... So thats not possible. Assumption 3: Multi-pronged attack(local keylogger, server rootkit): With the large number of vulnerabilities in desktop pc software in January(Adobe Flash and Reader, Java browser plug-in, curl(less used), Ruby's yaml unsafe parsing(less used), it is likely the number of infected pcs greatly outnumbers the number of servers needed. Why? Typical pc owners probably do not have their own webhosting server. Couple this with the reduced need for a huge number of spam servers. Also/Or the keylogger might be using an algorithm to infect only particular servers(time-based, IP address pattern, staging for later infection as spam network capability is reduced, etc). So why not send spam from the local pc with the key-logger? The chances are pretty good the local machine is on a residential internet connection with port 25 blocked or closely monitored. The presence of an SSH key indicates a server and if an LSB-compliant Linux then probably contains a mail server. What we are lacking: -Example of the spam being sent. Who can submit some examples? -Curious to see a list of servers compromised. Any kind of pattern in IP addressing ? Probably not likely to be helpful and too low of a signal/noise indicator. -Testing of possible local compromise. Use same local IP address and same server IP address to do a fresh install of a server that has been compromised. If it works, this narrows down that fact the source is a local pc compromise. If it doesn't, we might still be dealing with a local pc compromise and the key-logger does in fact have a smart infection alogrithm. Last edited by Ceetoe; 02-19-2013 at 09:06 PM. Reason: Here be typos...

Posted by dicoemil, 02-19-2013, 09:02 PM
nofz autorooters don't leave clean logs... unless they use http access with wget blabla combination. Considering those spammers in general are specialized in spam and not in hidding, I would say the http was the entry point, but plexy said is unlikely and he's probably right considering the advanced backdoor which is not typical for this type of spam. Why would you risk a 0day for some spam...

Posted by jalapeno55, 02-19-2013, 09:04 PM
Presumably they are logging in directly as root with the root password.

Posted by jalapeno55, 02-19-2013, 09:04 PM
Until we know what it is anything is possible.

Posted by plexy, 02-19-2013, 09:05 PM
None that I can see. Well, the cleanup script was unable to delete the file. It seems the attacker got cleverer and tried to negate the script by making so.1.9 immutable. I had to chattr -i the file and then run the script (well, i ran the commands in the script by hand tbh). However, restarting SSH did overwrite the file even though it already existed (as seen 16:50) The LFD alarms started arriving at 12:42 GMT. I am currently analysing all HTTP logs for that time period. Does anyone know the time LFD takes to check for presence of the file - 5 mins? 30?

Posted by suhailc, 02-19-2013, 09:06 PM
You guys have seen this right? https://bugzilla.redhat.com/show_bug.cgi?id=911937 http://seclists.org/oss-sec/2013/q1/326 Can someone please confirm if CentOS 5/6 and CloudLinux 5/6 have patched this yet? What's the latest kernels available?

Posted by Scott.Mc, 02-19-2013, 09:06 PM
Can you do something as simple as strings /dev/sda |egrep "sshd.*Accepted" (where /dev/sda is your partition) and see what accesses there has been.

Posted by nibb, 02-19-2013, 09:07 PM
I posted that. And I just mentioned it, I don´t know if its related or not, but could be. Yes, I saw the first infection of this in August 2012. So you are completely right on this. The machine was from a new client. New server, cPanel installed, WHM, nothing tweaked in security settings. Send the customer updated and patched to do his own work. 7 days later it was sending spam. cPanel support could not find any rootkit or logins. Neither could I at first. Looking further there where "no users" logged in the servers, but there where allot of established connections port SSHD port, mostly from OVH. The behaviour was exactly like mentioned in this post, and this was last year. The only difference between this server client and my servers, is that SSHD port was not restricted by IP login. I suspect this is some hack to the openssh but some here mentioned they don´t have it enabled. But maybe they where targeted from other machines that had a key or access to this systems. I also agree with someone else here, that to rookit a machine a machine just to send spam does not make to much sense, so I suspect are not getting completely inside a machine, but possible exploiting what they can, in this case for spam.

Posted by jalapeno55, 02-19-2013, 09:07 PM
That is not likely the cause.

Posted by matbz, 02-19-2013, 09:07 PM
Thnx v much. I was more focused on libssh2. I'm really comparing packages on infected/uninfected boxes, looking for any common differences. As short as I can... only one of our boxes (none compromised) has libssh2 installed, the rest cannot even find it from the repos, the box with it installed used an rpm from rpmforge which is not enabled by default in Centos (libssh2-1.2.9-1.el5.rf.x86_64) and which was updated during a manual yum update shortly after the box was compiled with what libssh2 came with the OS, totally different from a compromised box I did a comparison with earlier. You also have the same libssh2-1.2.2-11.el6_3.x86_64 on all 3 of your boxes, and which is the same as that on a compromised box from which someone sent me the list of installed packages earlier. Without all 3 boxes infected, it seems unlikely for now. I'll run all 3 through a file compare and see if anything correlates between all the other lists I have so far.

Posted by Patrick, 02-19-2013, 09:09 PM
My experiencing finding security flaws, logic and common sense are all saying no. This is not an Apache exploit and sure as hell not a PHP exploit.

Posted by Patrick, 02-19-2013, 09:10 PM
Non issue. The ptrace exploit needs near perfect timing conditions for it to happen making it an extremely difficult exploit to achieve in a working non-controlled environment. Is it possible? Yes. Is it likely? No.

Posted by jalapeno55, 02-19-2013, 09:16 PM
It could be an apache exploit just as much as it could be an openssh exploit.

Posted by dicoemil, 02-19-2013, 09:17 PM
Well then just check the http logs for same IP accessing same script multiple times and grep for libkeyutils wget and other patterns in http space files or where http had permissions to write (maybe you are lucky). And check searches from google too for whatever interesting keywords in the last days on that server, maybe you are lucky there. But you know that usual only the tcpdump will help you, keep stringing the dump for wgets and other patterns... if will ever happen again. The russian spammers scan in bruteforce, they don't care about logs and high-tech stealth attacks. They know that after they send the first spam mail, the sysadmin will come and narrow the server until will find them, so has no sense to put advanced backdoors and spam if they have 0days.

Posted by tarsiran, 02-19-2013, 09:18 PM
how can i stop lfd on server2.pcserver.in: System Exploit checking detected a possible compromise Possible root compromise: File /lib64/libkeyutils.so.1.9 exists. For more information see: http://www.webhostingtalk.com/showthread.php?t=1235797 CSF email alert?

Posted by Steven, 02-19-2013, 09:23 PM
except litespeedtech servers were compromised

Posted by nofz, 02-19-2013, 09:25 PM
What kernel versions are the compromised systems running? And did we check for kernel injections?

Posted by jalapeno55, 02-19-2013, 09:26 PM
Nevermind then. ...although I'm not sure we've found any similarities so far...other than the server is connected to the internet

Posted by plexy, 02-19-2013, 09:27 PM
In process. The repos on all 3 this end are; -rw-r--r--. 1 root root 1926 Jun 26 2012 CentOS-Base.repo -rw-r--r--. 1 root root 637 Jun 26 2012 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 626 Jun 26 2012 CentOS-Media.repo -rw-r--r--. 1 root root 2593 Jun 26 2012 CentOS-Vault.repo I also think ive been skuppered with the true date this file first appeared. It looks like the time of 12:42 for LFD alraming co-insides with an update of CSF/LFD sadly cron:Feb 19 12:42:01 vs3 CROND[4365]: (root) CMD (/usr/sbin/csf -u) cron:Feb 19 12:43:01 vs3 crond[2659]: (*system*) RELOAD (/etc/cron.d/lfdcron.sh) cron:Feb 19 12:43:01 vs3 crond[2659]: (*system*) RELOAD (/etc/cron.d/csf_update) im guessing that was when the alarm for this file was put into place.

Posted by dicoemil, 02-19-2013, 09:29 PM
nibb If the spammer will execute commands via SSH will not appear on your last command, or messages logfile. However, will appear on secure logfile as authenticated. @jalapeno if would be an apache exploit, then usually somewhere in apache logs will pop something wrong and plexy could notice that, but since the logs are clean, this is not an usual http or openssh exploit, unless the russian cleaned up (which is unlikely in this puzzle)

Posted by Steven, 02-19-2013, 09:31 PM
version info is all in the thread. Latest centos -- hacked Centos + Ksplice -- hacked Latest cloudlinux -- hacked

Posted by suhailc, 02-19-2013, 09:32 PM
So just to clarify - RHEL, CentOS and CloudLinux servers aren't getting hacked simply due to SSH running on default port 22 and with password authentication enabled - else a few hundred thousand servers would have been compromised by now. It's obviously a local exploit on servers resulting in privilege escalations which is getting onto the servers somehow. Are infected servers running the same software such as Softaculous or Fantastico? Were the servers recently updated? Maybe specific repos mirrors are infected?

Posted by plexy, 02-19-2013, 09:32 PM
Well, at this point I have to say that the time I am anlysing logs for seems to be, per my last post, the time that LFD was updated to look for the exploit. This is as opposed to when the file actually first appeared. So the exploit, if HTTP or PHP based to begin with, could be in any arbitrary log line going back to when this exploit was first seen in the wild. I cannot state with any confidence at this time that the logs are clean as my initial analysis of the HTTP logs, modsec logs was based on T minus 2 hours of the first alert.

Posted by tecnobrat, 02-19-2013, 09:35 PM
I was stupid with the apache module question, I forgot someone mentioned lightspeed.

Posted by iseletsk, 02-19-2013, 09:38 PM
else a few hundred thousand servers would have been compromised by now 1. There are might be... 2. They might be targeting shared hosting servers.

Posted by tecnobrat, 02-19-2013, 09:38 PM
Add debian as well. It was posted a few pages back.

Posted by GOT, 02-19-2013, 09:42 PM
So much noise in this thread, I'm wondering if anyone assocaited with the projects like Redhat, cpanel, etc are actively looking at this. FWIW, we manege hundreds of servers and have found four infected. All ran cpanel, but I don't think that necessarily has anything to do with it. 5 and 6 were impacted. Problem is I cannot even read all the posts there are not enough hours in the day...

Posted by Olly-ellogroup, 02-19-2013, 09:44 PM
This could still be PHP, no? We've ruled out mail servers, name servers, web servers, but not PHP? We still don't know how this gets in. It's unlikely SSHd. Why? Because after band-aids have been ran on infected machines we see IPs trying to log in using SSH keys. If SSHd itself was compromised then surely all communications would come in via this compromise rather than public key?! Also, might be dumb this one but for people running CSF, if SSH was the entry point with stolen keys/passwords then why wouldn't a "root access from xxx" e-mail be sent? From what we see so far, the only real common denominator here is control panels and PHP.

Posted by Disconnected, 02-19-2013, 09:44 PM
This is not privilege escalation as a VPS main node was also compromised and this attacker did not break out of a VE. This is almost certainly a password compromise issue.

Posted by jalapeno55, 02-19-2013, 09:44 PM
Did any of the 4 get reinfected?

Posted by Hellsheep, 02-19-2013, 09:44 PM
I was speaking to a guy last night from cPanel who confirmed they have resources investigating it.

Posted by matbz, 02-19-2013, 09:47 PM
Wasn't there some trace of logs being accessed during the rogue SSH session in some people's (Steven for instance?) snoopy entries? it's possible there was a clean up.

Posted by Ceetoe, 02-19-2013, 09:49 PM
Too much noise and this demonstrates this forum mechanism is not designed for something such as this. CloudLinux's CEO and Grsecurity posted here. Orien and Prohacker...I know you're loving eyeballs on the site and the revenue probably isn't bad either but can you implement thread summary so it displays at the top of EACH page? Also, can you enable a responsible adult(like Steven) the permission to edit that? 50 pages and we are still not having large samples to identify what to test and narrow down? Last edited by Ceetoe; 02-19-2013 at 09:53 PM.

Posted by NetworkPanda, 02-19-2013, 09:53 PM
It will be interesting running the "last" command on compromised systems to check if there were any suspicious root connections before the infection occurred. Also, a good idea is adding some code to your root .bashrc file which will be notifying you via email for new root SSH connections. Edit your /root/.bashrc file (there is a dot . in front of bashrc) and add this code at the end of it replace servername with your actual hostname and you@yourmail.com with your email address Now, every time somebody connects to your server SSH as root, an email message with his host name will be instantly sent to you. If you are unable to save the file /root/.bashrc (it happens on some systems even as root) Then run first this command: And after making the changes and saving it, run:

Posted by Scott.Mc, 02-19-2013, 09:55 PM
Steven said he seen this on a server with nothing but a static html website. Also it still has to have escalation even if it was from PHP but given the vast nature of the different systems, software, versions being posted it's unlikely to be anything other than authentication detail leaks somehow so people need to start thinking where their auth info is and how it could be obtained. Be it a local machine infection or a shared resource (such as a vendor). Regarding root access emails most of these only work when a shell is spawned so they wouldn't be received and CSF/LFD checks the logs so it can easily be removed by the time the cron runs. Running a simple strings per one of my earlier posts might reveal if any accesses are there that have since been removed.

Posted by egillette, 02-19-2013, 09:55 PM
Well. . .that is certainly a good point. . .I *never* for one minute considered that the file might already have the immutable or undelete bits set. So that leads me to a new question. . .can we assume that whomever is responsible is reading this thread?? I mean, that's totally new, and something none of us have encountered yet as far as I know. The attacker would have had to know that folks were executing my script in order to modify the approach (setting the immutable bit in advance). I've modified the script to account for this for anyone interested: https://www.ericgillette.com/clients/exploit-cleanup Then: Should match this: 78590d9e29f99b3788cae6a694717231 Then: Again, I think we can safely assume they are reading this thread, since that is the first time we've encountered a file that had the immutable bit set prior as far as I know. What do you guys think?? Last edited by egillette; 02-19-2013 at 09:59 PM. Reason: Made a small edit to the script which changed the MD5

Posted by apocas, 02-19-2013, 09:57 PM
So like I said before, in a large cluster, the 8 affected machines were the only ones with ssh password auth enabled. In 1 of the 8 machines I realized, the attacker was still trying to login from the famous OVH ip. So I rerouted all SSH traffic from that server to other server. There I implemented a basic Python SSH server, that just printed the SSH auth passwords to stdin I found out that the hacker is using this password "XXXIGk2BwMq" To the people out there that are reverse engineering the malicious .so file, I hope this string helps you.

Posted by plexy, 02-19-2013, 09:58 PM
from the PHP side, the inefected machine had these modules that the non infected machines do not have. But as seen, this has happened on a machine with static files only exif mbstring mcrypt soap Xcache The compromised lib is part of auth and encryption keys in the kernel. Irrespective of the attack vector to change this file, this lib is doing somthing to SSH. You can see that from my earlier posting that when SSHD was restarted, the libkeyutil.so.1.9 file was re-written. Would I be surprised if there is something in there that has a backdoor SSH key built in? not at all. Would i then expect to see SSH key login attempts after the infection is cleaned up - you bet.

Posted by Hellsheep, 02-19-2013, 10:00 PM
Minor details always get forgotten, but when I used your script (the commands in it) as a starting point for removal of the file I can confirm my server had chattr +i on the file already, had to chattr -i before could do anything. Further to this, I forgot who asked us to do a strings /dev/sda etc etc but I am running that now on my box and will post results after. However my box no longer has the libkeyutils.so.1.9 on it but may still result in some interesting information.

Posted by plexy, 02-19-2013, 10:01 PM
Recon of ones adversary always helps to keep on step ahead. Id say your assumption is highly probable. Thanks for updating teh script!

Posted by dicoemil, 02-19-2013, 10:02 PM
ericgillette even if your script is something constructive, if the hole is unpatched, will be just a matter or time until the hacker will come back. And then you don't know if he will wipeout your system as payback... I think a saved lsof and a network stop are the best things to start with + a cron entry to check each min for that file. Will narrow down the things a lot.

Posted by matbz, 02-19-2013, 10:02 PM
Honestly. Your script is seriously flawed, please stop encouraging people to use it.

Posted by Olly-ellogroup, 02-19-2013, 10:03 PM
Scott.Mc: Ok, read the entire thread but must've missed static HTML comment. What about my comment re. public key auth after cleanup though? Surely if it was SSHd exploit there'd be no public keys ever used in favour of direct access via the exploit?

Posted by egillette, 02-19-2013, 10:04 PM
Gotcha. On the 5 total machines I had that were affected, none of them had that bit set. So I still suspect whoever is responsible may be 'watching' this thread honestly. Especially since this had been rolling on for a few days now, and (s)he/they would have had time to adjust their strategy. I could be TOTALLY off-base about that, but I don't think we should rule it out as a possibility.

Posted by Disconnected, 02-19-2013, 10:08 PM
Nice find. This string is certainly present in the strings output of the .so so it appears to be a hardcoded password.

Posted by dicoemil, 02-19-2013, 10:11 PM
Will be nice a honeypot there, maybe you are lucky and the hacker will start scanning or using some sites to download things from...

Posted by Steven, 02-19-2013, 10:12 PM
Pretty cool. Kippo didn't see that when I tried.

Posted by Hellsheep, 02-19-2013, 10:21 PM
This may be off-track but does anyone that's had compromised servers use services like lastpass to store passwords for their server(s)?

Posted by james2398, 02-19-2013, 10:21 PM
Hello, I ran the script that was provided by egillette and now I get a 500 internal server error on almost all of my websites that run MySQL. I checked the apache error logs and it is giving me a barrage of errors saying the following: Will someone explain to me how I can restore this library please? The only thing I have done is run the script, so I am sure it is the script that caused the errors. I tried a force update, yum update, yum reinstall keyutils-libs, and nothing seems to work.

Posted by dicoemil, 02-19-2013, 10:22 PM
I think we should keep this thread clean, for constructive input towards the fix. If the problem will be found, then obvious the fix will be something the hacker can't avoid. Any informations which are new and not reported are good. The personal fights, can be taken to the first pub with cold beer

Posted by spendergrsec, 02-19-2013, 10:24 PM
The password isn't always the same, it changes among version even: Version 1.0.4: XXXg3DMlnHh Version 1.1.0: XXXNBwgzYz2 Version 1.1.0: XXXYwZfC62L Version 1.2.1: XXXdYZulavB Version 1.2.1: XXXIGk2BwMq The exfiltration IP for all versions is I have about 8 samples here, 32bit and 64bit. -Brad

Posted by jalapeno55, 02-19-2013, 10:25 PM
Would you mind posting the Python SSH server script?

Posted by dicoemil, 02-19-2013, 10:25 PM
james search on google for your distro files and download the file libkeyutils.so.1.9 or archive/unpack and copy to /lib/ with proper permissions

Posted by Scott.Mc, 02-19-2013, 10:26 PM
How are you defining "cleanup" though? Removing some library? All the "cleanup" methods being referenced in this thread are nothing of the sort. They at no point address the original entry point which as I have said is sounding more and more like authentication details being leaked and the exact same source as before is likely being used. I agree with your statement though like I have been saying, the chances of an openssh vuln are very slim, they would just re-use and more importantly there are much better targets than a bunch of web hosts. This industry does get unique concentrated vulnerabilities that are used (mostly privilege escalation) targeted to it simply because of the large concentration of easy to penetrate users (specifically over HTTP). Remote vulnerabilities that allow code execution/access in major daemons are getting rarer and rarer now adays. Back onto the topic at hand, HTTP can be mostly ruled out given the different webserver types and also if people are observing this from static websites. It would be good to try focus on authentication details, vendors used and possible vectors that way to see if it turns anything up.

Posted by Scott.Mc, 02-19-2013, 10:27 PM
What OS and arch are you using given yum what's the output of cat /etc/*-release and uname -i

Posted by plexy, 02-19-2013, 10:27 PM
I saw this problem and believe this is a symptom of the scripts lack of error checking. You may have had the immutable bit set and now the library is not linked. While this may not work for you, this is what I did (from memory, ive got far to many commands ran tonight to track down what i really did). This may not work for you. its just what I had to do in my case, be warned! cd /lib64 rm libkeyutils.so.1 chattr -i libkeyutils.so.1.9 chmod -x libkeyutils.so.1.9 rm libkeyutils.so.1.9 ln -s libkeyutils.so.1.3 libkeyutils.so.1 /sbin/ldconfig /etc/init.d/httpd restart /etc/init.d/sshd restart *edit* best follow the advice of the chaps above and provide the output they ask for first.

Posted by Scott.Mc, 02-19-2013, 10:27 PM
Likely using kippo, check out http://code.google.com/p/kippo/

Posted by matbz, 02-19-2013, 10:29 PM
Noooo, don't do that! libkeyutils.so.1.9 is not a valid file!

Posted by james2398, 02-19-2013, 10:29 PM
ls -lah /lib64 | grep libkey output gives me the following: lrwxrwxrwx 1 root root 18 Feb 19 20:58 libkeyutils.so.1 -> libkeyutils.so.1.9 -rwxr-xr-x 1 root root 10K Jun 22 2012 libkeyutils.so.1.3* ---------- 1 root root 34K Feb 19 15:50 libkeyutils.so.1.9 ls -lah /lib | grep libkey gives me the following: lrwxrwxrwx 1 root root 18 Feb 19 21:00 libkeyutils.so.1 -> libkeyutils.so.1.3* -rwxr-xr-x 1 root root 9.4K Jun 22 2012 libkeyutils.so.1.3* Scott, I got the following output from the commands you specified CentOS release 6.3 (Final) CentOS release 6.3 (Final) CentOS release 6.3 (Final) x86_64

Posted by jalapeno55, 02-19-2013, 10:32 PM
Thanks Scott

Posted by dicoemil, 02-19-2013, 10:32 PM
Indeed matbz, i copy/pasted too much, wanted to put just the libkeyutils.so part. Well he wont find it on distro servers .1.9 anyway, and even if he finds the vulnerable one(which I doubt) at least his MySQL will work until will put the good one

Posted by apocas, 02-19-2013, 10:32 PM
Didn't knew kippo. Cool tool! It was something I hacked together in a few minutes, that's why I will not send code to anyone.

Posted by Scott.Mc, 02-19-2013, 10:34 PM
Get the packages from http://mirror.centos.org/centos-6/6.3/os/x86_64/Packages/ try something like lattr /lib{,64}/libkeyutils* Then remove any flags preventing it from being replaced such as, chattr -ai /lib{,64}/libkeyutils* And either unpackage the rpm and replace (rpm2cpio) or try a forced install such as rpm -Uhv --force http://mirror.centos.org/centos-6/6.3/os/x86_64/Packages/keyutils-libs-1.4-4.el6.x86_64.rpm http://mirror.centos.org/centos-6/6.3/os/x86_64/Packages/keyutils-libs-1.4-4.el6.i686.rpm Last edited by Scott.Mc; 02-19-2013 at 10:38 PM. Reason: removing url parsing

Posted by dicoemil, 02-19-2013, 10:35 PM
james is ok what plexy said then 1 page back

Posted by matbz, 02-19-2013, 10:39 PM
run this in ssh then please post the output of

Posted by plexy, 02-19-2013, 10:40 PM
JAMES - I recommend following scotts procedure or he help from matzb. Always best to replace the libs. Mine was just a band aid to get going.

Posted by james2398, 02-19-2013, 10:41 PM
Mattbz, I received the following when running the first part of your commands: ln -s /lib64/libkeyutils.so.1.3 /lib64/libkeyutils.so.1 ln: creating symbolic link `/lib64/libkeyutils.so.1': File exists Then, I received the following output on the second command: lrwxrwxrwx 1 root root 18 Feb 19 21:38 libkeyutils.so.1 -> libkeyutils.so.1.9 -rwxr-xr-x 1 root root 10K Jun 22 2012 libkeyutils.so.1.3*

Posted by matbz, 02-19-2013, 10:43 PM
actually great advice, but he still has the wrong symlink... this 'script' is wreaking havoc for some people!

Posted by The3bl, 02-19-2013, 10:44 PM
Lets stop the noise about the script, if people want to use it then fine if they don't then fine. Or Go start another thread and bicker about it all there. Lets concentrate on how this got installed on the servers to start with. I think we have dissected the exploit enough myself we know how it works. I think it is safe to also assume the hacker or hackers are well aware of this thread and reading it. So yes he is probably tweaking his script as we roll along here fighting about detecting it and how to clean it. We need to set up a repository of hacked scripts so as they are found they can be uploaded and checked for new code and changes. And only report if they see a new attack vector from them. In the mean time we need to get back to how they got in to start with. List it out and narrow it down. Right now this thread is so confusing I cannot tell if it is only on boxes with password enabled or if the ones with keys has keys that require a passcode etc.. until we start laying this out in some logical order it is like a bunch of cats going in circles.

Posted by matbz, 02-19-2013, 10:46 PM
Sorry, then run this... and post the output as before please. I have to hit the hay soon.

Posted by dicoemil, 02-19-2013, 10:46 PM
plexy looks clean his lib (i know can be wrong, but is not usual) However, that time on the symlinks is v. interesting. 2 mins between 2 symlinks and hours between download and symlink...

Posted by james2398, 02-19-2013, 10:49 PM
Thank you Matt, This time when running those commands, I received no errors with the exception of your third line: chattr -iu /lib64/libkeyutils.so.1.9 chattr: No such file or directory while trying to stat /lib64/libkeyutils.so.1.9 Then the fourth line worked as well. When running the command as requested on your previous post, I received the following output ls -lah /lib64 | grep libkey lrwxrwxrwx 1 root root 25 Feb 19 21:47 libkeyutils.so.1 -> /lib64/libkeyutils.so.1.3* -rwxr-xr-x 1 root root 10K Jun 22 2012 libkeyutils.so.1.3*

Posted by dicoemil, 02-19-2013, 10:52 PM
james, your mysql should be fixed now However keep in mind the machine is still vulnerable and most probably will be hacked again, so you might want to save the data asap.

Posted by matbz, 02-19-2013, 10:53 PM
That's good, means the bad file went away on the previous commands. You should be ok now. If not then please reboot and let us know Unfortunately, this 'script' has and possesses the potential to take a box out of action, whereas the actual exploit hasn't as yet.

Posted by bloodyman, 02-19-2013, 10:54 PM
Reasuming after reading todays posts: servers running ssh with password authenticaton and without password authentication will be hacked the same way, because libkeyutils.so.1.9 which is linked to ssh deamon includes password (build-in) so it changes the way sshd works and enables hacked to use password when he authenticate. This is my thought based on reading all today posts - because servers using password authentication and key authentication were hacked the same way also - existing of the file libkeyutils.so.1.9 does not indicate that you server is sending spam. so this file must be placed into the server in any other way than using ssh, I think that it might be placed using symlink hack correct me if I'm wrong about above thoughts. still I don't know how hacked restarts sshd deamon to get libkeyutils.so.1.9 loaded into memory?

Posted by dicoemil, 02-19-2013, 10:57 PM
Somebody with an infected machine can do a test and see if he ssh into it, the machine sends packets to some weird IPs on prot udp port 53?

Posted by Steven, 02-19-2013, 10:58 PM
Old news. Read the thread.

Posted by james2398, 02-19-2013, 11:00 PM
I rebooted my server and no longer am able to access my server. It has not come back online

Posted by tarsiran, 02-19-2013, 11:01 PM
i have a error: [root@xlhost /]# rm -f /lib64/libkeyutils.so.1.9 rm: cannot remove `/lib64/libkeyutils.so.1.9': Operation not permitted even for change premission recieve error: [root@xlhost /]# chmod 755 /lib64/libkeyutils.so.1.9 chmod: changing permissions of `/lib64/libkeyutils.so.1.9': Operation not permit ted please help me dudes

Posted by Scott.Mc, 02-19-2013, 11:02 PM
lsattr /lib64/libkeyutils.so.1.9 Likely has immutable flags (chattr -ia /lib64/libkeyutils.so.1.9)

Posted by Scott.Mc, 02-19-2013, 11:03 PM
Your daemons won't start if you have ran the above script and broken the link to the keyutils library. So you'll need to remote access/kvm and reinstall the library correctly.

Posted by tarsiran, 02-19-2013, 11:06 PM
oh it's so danger and high risk i wanna do that but i'm scare right now

Posted by dicoemil, 02-19-2013, 11:07 PM
Well from james copy/paste on that last ls the symlink was ok. (if he has x64). Why restarting the server anyway, you should check first if the MySQL works properly. The server will be hacked again anyway with/without restart. Linux is not a Windows machine to restart it all the time (or you can hope is making fsck)

Posted by The3bl, 02-19-2013, 11:10 PM
Now can someone that was saying the hack comes right back after removing it tell me what steps you took to remove it what you killed or didn't after? Did you see what processes were connecting to the hack in mem before you yum updated? Did you check your cron files to see what was in there?

Posted by Scott.Mc, 02-19-2013, 11:12 PM
For the record I checked ~900 (926 to be exact) systems for this and was unable to find this library which is across all sorts of control panels, os's, arches and a mixture of shared (not very dense however) systems and clustered systems. Also a good 100 or so have password auth and ssh open to the public. Systems are also located across many of the same providers people are using here. So it has to be something specific to those users. Last edited by Orien; 02-19-2013 at 11:24 PM. Reason: cleaning

Posted by chinook, 02-19-2013, 11:20 PM
Not sure if this helps and if it adds to the noise I apologize profusely. I have two machines that were both spun up on the same day last fall. For the most part they are identical. One was affected the other not (so far). I racked my brains on the difference. The one that was affected I had inadvertently left open client ssh (cpanel includes a link to it on customer pages). As far as I know 1 client did take advantage of my sloppiness. So if that is the source then this indeed may be very similar to gumblar. Once attacker has any credentials at all , log in then use the symlink race weakness (discussed at great length over at cpanel). Now having read all the posts all day long the fly in the ointment would be cagefs but frankly what if that is a red herring , a false positive. Ie how many cagefs systems have been exploited? Enough to draw a logical conclusion.

Posted by DaEmployer, 02-19-2013, 11:22 PM
Wow, 900+ servers - did you just run a single line in one terminal to check them all?

Posted by dicoemil, 02-19-2013, 11:23 PM
I checked a bit the snoopy reported log too. And the first line is interesting. The hacker is removing a file from /home/ from a non-existing-usually user. Which means, he needs to create that user and somehow downloads after something which makes him root. So, he either is weird by creating a random user to download something to setuid, when he has allready root acces (?!). Or is using a software which can create users, which can give him ftp access or whatever, probably even downloads a file for him with root privileges...

Posted by spendergrsec, 02-19-2013, 11:24 PM
You should be seeing some unusual DNS queries for randomly generated hostnames as well. It seems to be using these as some kind of fast flux, where the hostnames will look like ..(biz, info, or net). The IPs these hosts resolve to won't make sense because they're not used directly. Rather, the resolved IP is passed into an algorithm that computes an IP it will connect to. If someone's logging packets, they should potentially see two hostname lookups (modulo some local caching) followed by a UDP send on port 53 to somewhere other than your DNS server. If you can give me the random hostnames, their resolved IPs, and the subsequent host used, I can piece together the algorithm. -Brad

Posted by FastServ, 02-19-2013, 11:27 PM
Hi all, I hate to say it but a soft reboot of the server OR any services allows a chance for the malicious libs to be replaced since it lives in memory and can respond to a kill signal however it wishes (for example, replace the file). A hard reboot (power cycle) after cleaning up the libs (preferably into single user mode to further clean things up) would be the only 100% sure way to wipe it out, ASSUMING you've also cleared out any other vectors such as init scripts and cron jobs. Instead of trying to clean this up yourself only to be compromised worse, you (and the community at large) would be better served by handing these servers over to trusted community members like spendergrsec or Steven (ect) for further analysis. Last edited by FastServ; 02-19-2013 at 11:37 PM.

Posted by dicoemil, 02-19-2013, 11:31 PM
Yes FastServ, probably some cron or whatever is putting back the backdoor, tried to link the lib again to the hacked one, which was deleted. However, the server should respond to ping... (and in rest God knows what was changed on that server)

Posted by The3bl, 02-19-2013, 11:36 PM
Of the 5 boxes I saw hacked. They were cleaned of the file all processes that had connections to the bad lib was killed and restarted verified nothing was left connected in mem to it and then the passwords were changed and the boxes were then yum updated. So far no recurring hack. Though in all five boxes I found where that IP from OVH tried to log in as root using the password I assume the old one as he was unable too. Right now snoopy and a couple of other programs are running in case he does get in but so far knock on wood he has not come back.

Posted by kaniini, 02-19-2013, 11:48 PM
Hello, This is clearly a typical RBN rootkit. These things are designed to be memory resident -- simply rebooting the server will not guarantee it will not come back. In fact, it will probably come back when you reboot it if Apache is linked against libkeyutils. The behaviour of the rootkit reeks of RBN involvement, especially considering what IPs it calls back to. Which brings me to another point. I am very particular that the people who got rooted are all running Windows in their offices. What would this have to do with anything you ask? Well, it's simple, really! Here is how you all got rooted, at least based on previous RBN operations of this sort: 1. You received some flash advertisement with a malicious payload in it. 2. The malicious payload owned your office PC, and then likely caused Flash to crash. You probably got a frowning puzzle piece icon saying "this plugin has crashed" when this occured. 3. The malicious payload probably dropped a more sophisticated malware on your PC, including keylogging and screen monitoring capabilities. 4. As you logged into WHMCS or Ubersmith or physical servers, the keylogger activated and sent that information to the RBN. 5. The RBN logged into your server, through your own box most likely (this is called 'bouncing' usually), and installed the rootkit. You should probably: 1. Immediately reinstall your office PCs. 2. Immediately change ALL credentials. 3. Audit your WHMCS and Ubersmith logs to ensure the intruders did not gain customer PII (credit card info, etcetera). If they took a tromp through your servers, they probably took a tromp through your WHMCS and Ubersmith too! 4. Clean up the rootkit on your boxes AFTER changing credentials from a FRESH machine. There is NO POINT in changing credentials if there is a keylogger on your machine, now is there? 5. If PII is compromised, tell customers immediately. Don't be lame about this stuff, the RBN moves very quickly with PII. The reasons why it is clearly an RBN operation are obvious: 1. The IPs it backconnects to are very clearly related to RBN. 2. The RBN tends to use dedicated servers from high volume providers to take advantage on delays in the abuse queue (it gives them more time to do illegal activity). 3. RBN has targeted IT organizations in the past to gain server credentials for rooting. Why would they suddenly stop doing this? Oh, right, they wouldn't! TL;DR: Boxes are most likely owned through employee PCs which were most likely owned through other RBN malware operations. With targeted interest-based advertising, it is very easy to hit hosting companies with very little additional noise in the data. Oh, and they probably have your WHT passwords too, but RBN doesn't care about that I can assure you. Oh, and some RBN people are probably watching this thread and injecting misinformation into it. They used to do this with discussions about DNSBL providers on USENET and various mailing lists. EDIT: Here is how you remove the infection and ensure it does not come back. This involves using /proc/sysrq-trigger to trigger an immediate reboot, thus removing any possibility that it can reroot while memory resident. Last edited by kaniini; 02-19-2013 at 11:51 PM.

Posted by kaniini, 02-20-2013, 12:03 AM
By the way, if I am right about this (which by the way, I am right about this), that means WHT should stop allowing flash ads, because they are being used as exploitation vectors. But what would I know, I only operate a DNSBL that has dealt with RBN for years.

Posted by tarsiran, 02-20-2013, 12:04 AM
i have an infected on directadmin server: [root@xlhost /]# ls -lah /lib64 | grep libkey lrwxrwxrwx 1 root root 18 Feb 20 06:08 libkeyutils.so.1 -> libkeyutils.so.1 9 -rwxr-xr-x 1 root root 10K Jun 22 2012 libkeyutils.so.1.3 ---------- 1 root root 34K Feb 19 21:45 libkeyutils.so.1.9 so how can if prevent this maleware right now? (even temporary)

Posted by dicoemil, 02-20-2013, 12:16 AM
Ok, I found few more pieces of this puzzle, maybe will help. So, like I said few pages ago, I saw an unusual spamwave thru commands via ssh user account few days ago. I still see this ips, still trying to send spam (so try to find them on abused systems): The servers had centos 6.2, openssh-5.3p1-70 Even if the above ips you can find on the list of the ones which connect to rooted servers, this servers were not rooted, had lots of ssh accounts and users. However, just this/one client, having few hosting shared with ssh enabled, all accounts hacked, different servers. Whoever sent the spam, did not even bother to go into ssh command line to hack the server, probably he tried tho with ssh -c. However he somehow got the password, so in theory is something on user side which provides the passwords. User connected just via FTP, never via SSH. Had the passwords saved on Total Commander. No cpanel, no fastcgi, the servers actually runs a hosting control panel which you wont find anywhere, because is still in beta and not released, so the leak of passwords is not on the hosting side. However if people had rooted servers, probably is a combination between sniffed/logged passwords on client side + cpanel or local exploit on hosting side. Or those rooted attacks and the ones I saw are different types, even if they started exactly at same time.

Posted by kaniini, 02-20-2013, 12:20 AM
The instructions I posted will remove it permanently assuming that you have changed your credentials from a machine known not to be compromised (I suggest doing so from a machine you do not regularly use, or perhaps from a Ubuntu LiveCD). The important thing is to ensure you do not change your credentials on a machine that you use daily as it may (hint: probably) is infected with a keylogger. My suggestions are: 1. Reload your employee PCs or otherwise ensure they are clean of all malware infestations (the best way to do this is to reload immediately). 2. Change your credentials from a known secure machine. 3. Remove the malware given the instructions in the previous post I made, ensuring that you reset the symlink to the valid libkeyutils.so file, which is either 1.2 or 1.3. You will need to check before doing this or you'll hose your machine.

Posted by dicoemil, 02-20-2013, 12:23 AM
You can't prevent until somebody will find the cause. Looks like whoever is doing this hacks, has command line with root access on the servers. You can only try to filter the IP's I posted above. Maybe you are lucky and he wont connect to backdoor your server and you can find the remaining hack trails.

Posted by kaniini, 02-20-2013, 12:31 AM
This is just more proof it is RBN, as it fits their typical pattern of buying machines at hosting providers who are large (to take advantage of delays in abuse processing). As an example: OVH: HostEurope: LeaseWeb: RBN loves to use OVH, HostEurope and LeaseWeb verses smaller providers because abuse processing leading to full termination takes some time. So, they can usually get a couple of weeks to a month of criminal activity from these providers, simply because the provider has a lot going on and doesn't have time to micromanage each situation. Based on reverse DNS, it appears some of these IPs have been recycled which helps to make the appearance that it is compromised machines, but it probably is not.

Posted by Hellsheep, 02-20-2013, 12:38 AM
I do not remember who asked for this earlier, and I have obfuscated my IP address that I connect to the server with. It's a VPS, I also removed some hostnames for privacy reasons however if you absolutely need all the info I've taken out please PM me and I can provide you the full output that hasn't been obfuscated.

Posted by Steven, 02-20-2013, 12:41 AM
What about this: Feb 14 19:46:59 vmx11286 sshd[28278]: Accepted password for root from port 1747 ssh2 do you know this ip?

Posted by jalapeno55, 02-20-2013, 12:41 AM
QUOTE=Hellsheep;8566355]I do not remember who asked for this earlier, and I have obfuscated my IP address that I connect to the server with. It's a VPS, I also removed some hostnames for privacy reasons however if you absolutely need all the info I've taken out please PM me and I can provide you the full output that hasn't been obfuscated.[\QUOTE] Scott from AdminGeekz requested it.

Posted by Hellsheep, 02-20-2013, 12:43 AM
Yep I do! >.< Seems like I missed one, however that's OK. That's one of the many IP addresses my colleague gets (good ol' dynamic IP's)

Posted by dicoemil, 02-20-2013, 12:51 AM
Then I think we can agree on the fact that the libkey was backdoored just to capture user/passwords to send spam, and not to have fun on shell on the server. So probably RBN just infected with some exploit tons of servers, to capture more user/passwords before sending the spam wave... and some sysadmins noticed. The question remains, what was that exploit...

Posted by Zxctypo, 02-20-2013, 12:52 AM
I'm not saying that you're wrong, just playing devils advocate- But assuming it's from obtained root pw, why would the profile of compromised machines be so narrow? Why only RHEL machines?

Posted by jalapeno55, 02-20-2013, 12:56 AM
Debian has been reported compromised as well.

Posted by Zxctypo, 02-20-2013, 12:59 AM
Ah sorry, I missed that. Thanks for the clarification.

Posted by dicoemil, 02-20-2013, 01:00 AM
And probably is no exploit either, just keylogging and saved passwords from some infected sysadmins.

Posted by timta, 02-20-2013, 01:02 AM
Trying to find a copy of the latest 64bit and 64bit versions of the kit, getting ready to tear it apart and want to make sure I am working on the latest. If any one wants to help out let me know. I will be setting up a wiki shortly to centralize information on the disassemble.

Posted by james2398, 02-20-2013, 01:03 AM
Once I get my KVM / IP setup, and am able to login to my server again, after being locked out due to the running of the 'fix' script, I will be more than happy to send you the file.

Posted by Scott.Mc, 02-20-2013, 01:08 AM
Given you identified all the IP's does this resolve in your dns/host file and you know what it is? Feb 14 20:24:39 vmx11286 sshd[31033]: Accepted publickey for root from 9431b442-bf03-ea51-5288-a91c30b697af Other than that I suspect this is a node given the different hostnames but it should give you a good bit to go on. Match the login times with the initial compromise and also compare your current ssh log versus the output above (don't exclude your own IP's, could have been bounced through your own work stations [backconnect]) to note any that are missing which will be one of the quicker ways and after that compare logins with ones you know you've made and ones around the time of the infection.

Posted by Hellsheep, 02-20-2013, 01:11 AM
Does anyone know what this is, I did a ls randomly as the root user and found something called ipadd.sh, I've never seen it before and a quick google doesn't seem to show any results.

Posted by TravisT-[SSS], 02-20-2013, 01:14 AM
It looks like a script from your data center or someone who setup your server?

Posted by Hellsheep, 02-20-2013, 01:14 AM
Thanks, for a start 9431b442-bf03-ea51-5288-a91c30b697af definitely doesn't resolve and I have no idea what it is. Looks rather sus to me. Possibly, I'll contact them anyway and confirm just in case.

Posted by timta, 02-20-2013, 01:15 AM
It configures iptables(firewall/ipfilter)

Posted by zildjian42, 02-20-2013, 01:23 AM
IPv6 address? lol Also, google "ipadd.sh" WITH the quotes and it's pretty common.

Posted by Hellsheep, 02-20-2013, 01:24 AM
Definitely not, an IPv6 address looks like this: 2001:0:5ef5:73ba:204a:1a20:a83d:337c

Posted by zildjian42, 02-20-2013, 01:29 AM
Definitely not IPv6 because the exactly 32 characters of hexadecimal digits where an address should be don't have the : ? k

Posted by Hellsheep, 02-20-2013, 01:46 AM
Sorry, not an attack and I should have explained a bit better. From my understanding an IPv6 address cannot start with 9431 either. https://www.ripe.net/lir-services/ne...rence_card.pdf

Posted by vpswing, 02-20-2013, 01:53 AM
Someone mentioned getting all bleary eyed and needing to get up to pee after going through all the posts in this thread. I can so totally agree! A few suggestions if I may: --------------------------- 1. Those who need 'securing'/putting on the 'band-aid'; perhaps it would better if they start another thread. I'm sure some of the security specialist in this forum will be more than happy to help/reply to that new thread. Keep this thread focused on finding the cause. 2. Word of caution to newbies - never run any script or commands without knowing what they do. That's a surefire recipe for disaster. 3. Steve @ Rack911, since you started this thread, perhaps it might be a good idea to start a new one with a summary of the findings? And perhaps be allowed to moderate who can post to that thread? That will cut down all the un-necessary noise :-) 4. md5sum of the libkeyutils.so.1.9 ? Someone (Brad from grsec?) mentioned he has variants of the file. Could those with that rogue file post the md5sum here? If all the variants of that file share the same md5sum, would that be helpful detecting that file (even if the name has changed?) Would it be helpftul to rkhunter or the chkrootkit or csf/lfd? btw, those with rooted servers - could you check the md5sum of the sshd file? Someone (can't remember who now) mentioned the file could have modified sshd as well to allow re-entry. My servers (clean) are running: openssh-server-4.3p2-82.el5 (CentOS 5.9) 64-Bit: dedd962724841712a91674845aeba2da /usr/sbin/sshd 32bit 8d705e81fcb1b53c5d38fcd9cbf0690f /usr/sbin/sshd

Posted by Hellsheep, 02-20-2013, 02:17 AM
64-bit 2a939f5aa3666c468636ec567a0ba26a /usr/sbin/sshd openssh-server-5.3p1-81.el6_3.x86_64 CentOS 6.3

Posted by m6tt, 02-20-2013, 02:18 AM
I saw something containing this string hit my suricata IDS this morning... Google search yields a bunch of questionable results including wp-admin pages and weird failed build logs, and I see this string in one of the strings runs against an infected binary: AWAVAUI ýATUSH I'm pretty sure I saw it in an email, but it could only be email or http to hit my IDS.

Posted by kmike, 02-20-2013, 02:22 AM
I'd say it definitely points to the admin's machine infection as a culprit for this whole debacle. Could you friend contact the contractor and ask them to run the malware scan on their machines?

Posted by ra4h, 02-20-2013, 02:33 AM
So far none of our servers are doesn't infected with this vulnerability. However we have created cronjob to check the libkeyutils.so that doesn't owned by any of the package and mail me and so I can see how the things are changed with this vulnerability. Though I would suggest we can touch the file /lib64/libkeyutils.so.1.9 and set the permission to 000 with chattr ai which would possible to prevent uninfected servers.

Posted by LinqLOL, 02-20-2013, 03:59 AM
All our machines are clean. Just changed our puppetmaster to lock down all ssh connections for now. BUT people who got hacked did you had selinux enabled or disabled? I mean 90% of the exploits would be stopped by SELINUX normally.

Posted by mattmackman, 02-20-2013, 04:03 AM
Did you see my snoopy log? I have already mentioned that. The hacker is manually placeing that file. So its doesn't help if create the empty file.

Posted by webhostmaniac, 02-20-2013, 04:20 AM
Has anyone been compromised while using a non standard ssh port aka not 22 while also allowing access through that new port to all visitors? If so I'm going to set a trap as my 6.3 has not yet been comp'd and I notice a rpm difference among the guy who updated his server with yum update and 4 hours later got hacked. root@box [~]# cat packagesinstalled-1361348100 1 abrt-2.0.8-6.el6.centos.2.x86_64 1 abrt-addon-ccpp-2.0.8-6.el6.centos.2.x86_64 1 abrt-addon-kerneloops-2.0.8-6.el6.centos.2.x86_64 1 abrt-addon-python-2.0.8-6.el6.centos.2.x86_64 1 abrt-cli-2.0.8-6.el6.centos.2.x86_64 1 abrt-libs-2.0.8-6.el6.centos.2.x86_64 1 abrt-tui-2.0.8-6.el6.centos.2.x86_64 1 libreport-2.0.9-5.el6.centos.2.x86_64 1 libreport-cli-2.0.9-5.el6.centos.2.x86_64 1 libreport-plugin-kerneloops-2.0.9-5.el6.centos.2.x86_64 1 libreport-plugin-logger-2.0.9-5.el6.centos.2.x86_64 1 libreport-plugin-mailx-2.0.9-5.el6.centos.2.x86_64 1 libreport-plugin-reportuploader-2.0.9-5.el6.centos.2.x86_64 1 libreport-plugin-rhtsupport-2.0.9-5.el6.centos.2.x86_64 1 libreport-python-2.0.9-5.el6.centos.2.x86_64 2 freetype-2.3.11-14.el6_3.1.x86_64 2 freetype-devel-2.3.11-14.el6_3.1.x86_64 2 glibc-2.12-1.80.el6_3.7.i686 2 glibc-2.12-1.80.el6_3.7.x86_64 2 glibc-common-2.12-1.80.el6_3.7.x86_64 2 glibc-devel-2.12-1.80.el6_3.7.i686 2 glibc-devel-2.12-1.80.el6_3.7.x86_64 2 glibc-headers-2.12-1.80.el6_3.7.x86_64 2 glibc-static-2.12-1.80.el6_3.7.x86_64 2 nspr-4.9.2-0.el6_3.1.x86_64 2 nss-3.13.6-2.el6_3.x86_64 2 nss-sysinit-3.13.6-2.el6_3.x86_64 2 nss-tools-3.13.6-2.el6_3.x86_64 2 nss-util-3.13.6-1.el6_3.x86_64 Any package not marked 2 is not installed. I am not using port 22 and I have not yet been compromised on centos 6.3 64bit. If someone using a custom ssh port has been comp'd I'm going to update my packages and wait for the malicious file to appear on my server. edit * nvm I read the thread summary * going to update these packages now and wait test * Last edited by webhostmaniac; 02-20-2013 at 04:27 AM.

Posted by kaniini, 02-20-2013, 04:30 AM
Tonight we noticed our OVH and some other upstream login credentials were compromised, between now and 4PM GMT-6 yesterday. The credentials were changed from our office IP, at a time when nobody was in the office (after 4). An investigation of the infected Windows machine indicates that it was sending keystrokes to the IP as DNS packets, the same format as the DNS packets mentioned here. The machine is now unplugged from the network and isolated for later investigation.

Posted by ra4h, 02-20-2013, 04:31 AM
Ahh yeah I can remind this. Have you tried to set the chattr vaules a and i and as well sticky bit?

Posted by gigatux, 02-20-2013, 04:34 AM
It sounds like we're gradually coming to the conclusion that this initial attack vector must have been via an administrator's local computer by something like a keylogger. I would be interested in the following question (and I'm asking my team too): Has a server been compromised where all logins are performed by users not running Windows? There could of course be many more attack vectors on an admin's local machine, but perhaps this is a start. It might allow us to narrow down the bits of software used and also confirm that it's not an exploit on the server. It sounds like, from reading this thread, there are zero-to-few users that have had their servers compromised where key-based authentication is used, which leads a bit more credence to a password being typed or pasted on a local machine.

Posted by Steven, 02-20-2013, 04:37 AM
Windows has nothing to do with it. Just as an example, I will mention it again. If it is a local computer exploit.. it could very well be flash.. and it would affect mac and linux if they make a payload for it. http://www.cvedetails.com/vulnerabil...sh-Player.html Just look at the number of updates.

Posted by gigatux, 02-20-2013, 04:43 AM
I agree that it's very possible, nay likely to be an application running on the OS, but it would be good to try and narrow down things a bit. All accounts I've seen on this thread so to include some sort of access from a Windows machine so this information could let us know a) the current attack vector (e.g. a payload may only exist for Windows Flash exploits), and b) whether it's worth investigating Windows-specific (or vastly Windows-used) applications, e.g. PuTTY.

Posted by Cristi4n, 02-20-2013, 04:46 AM
Interesting statistic. I just scanned a pretty high number of servers (a few thousand) and only found 10 servers affected by this. This may very well point to keyloggers being used, as mentioned above.

Posted by HostSentry, 02-20-2013, 04:55 AM
I've setup monitoring on my servers, so that I'm notified if any changes are made to the lib64 directory. Is everyone consistently seeing that as a symptom of the exploit?

Posted by aeris, 02-20-2013, 05:18 AM
Manually? Considering the timestamps, that's some quick typing if it's manual...

Posted by nofz, 02-20-2013, 05:18 AM
It seems like we are heading more and more towards entry via workstation. But if so why almost all servers hacked are RPM based distro's? And if its true, then we should be expecting very different variants (different os/backdoor) very soon. They can basically do everything they want. And it will make things even more confusing when they add more tactics/variations to this whole thing. I still have my doubts about it. Last edited by nofz; 02-20-2013 at 05:24 AM.

Posted by Steven, 02-20-2013, 05:29 AM
Perhaps because rpm distros are common for hosting. Perhaps there was a malicious hosting related flash app that was targeted at us. Who knows.

Posted by tarsiran, 02-20-2013, 05:31 AM
deay steven did you find any absolute way to fix this maleware?

Posted by kaniini, 02-20-2013, 05:39 AM
Because RPM distros are used on 99% of all servers in hosting?

Posted by Alec, 02-20-2013, 05:40 AM
I'm not yet convinced of anything, either. However, one thing that could support this theory... As steven said... There are a *lot* of people using RedHat/CentOS/Cloud Linux/ etc for hosting, cPanel, etc. Also these OS'es, by default, allow you to ssh in as root, vs some of the other operating systems disable root login. I suspect not as many people change this as a security expert would think. How, I don't want to put on my tin foil hat yet... It could be because a keylogger sees 'root' and then a string of characters and the enter key and this becomes useful information vs someone who is logging in with user_name.

Posted by Martin-D, 02-20-2013, 05:43 AM
I don't believe it's keylogger based. The one server I've seen that was compromised was never logged in to directly from a desktop machine (Windows or Linux)

Posted by Zadmin, 02-20-2013, 05:59 AM
I can confirm call to on port 53 is triggered only by password logins not logins with public/private keys

Posted by Polina136, 02-20-2013, 06:01 AM
Hi everyone, Our servers were under that attack too. You all were trying to figure out which software caused the problem. Anfortunetly allmost all our server server have almost NOTHING on them and still were compromised. For example we saw "sleep 7200" process running on a machine which has just openvpn installed for managing corporate network and nginx. The other one had just OpenVZ containers. It's not just the CentOS problem - we have that both on CentOS and Debian Squeeze. Also grsec showed some errors like: grsec: From time set by /home/tmpp/openssh-5.9p1/touch2[touch2:7032] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:31078] uid/euid:0/0 gid/egid:0/0 In Selinux records we see suspicious records: [6278854.120131] type=1400 audit(1360293227.096:293): avc: denied { use } for pid=9110 comm="sshd" path="/dev/pts/0" dev=devpts ino=3 scontext=unconfined_uystem_rshd_t0-s0:c0.c1023 tcontext=system_uystem_rshd_t0-s0:c0.c1023 tclass=fd But we do not think they have to do something with that bad guy. Now we are tracking new process by looking at: egrep -a -o 'SSH_CONNECTION=[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' /proc/*/environ | sort -n | uniq It all strarted after we found strange sleep process running. And looked at their SSH_CONNECTION variable in environ. "sleeps" were running from ssh, user root, like it had just password/key and with no effort accessed the server. But logs showed nothing. But we need to say the servers which were not compromised had "Failed" records in auth.log with that IP. So the theory that libkeys used to steal passwords is not that unreal. Because libkeys were detected on many servers but spam sent only those of them which were accessed by plain password and not key. SSH daemons were or different ports. Also that's nor the first time we see something like that. In a year 2012 i guess there was something like that. Also those "sleeps" but besides that hacker ran perl script" which was named " " (space) and kept it in a /tmp folder, so in "ps aux" we saw just "perl ". Script sent casino spam like that one in this topic and bruteforced other machines. We suspect they are connected somehow. We doubt it is a side software issue. It might be a kernel bug CVE-2013-0871. Still not fixed in Debian and Centos. We doubt it is Windows related problem. Yes password of few our servers could be stolen from an Windows server where those password were copied from DC billing but the password on another one has been changed recently. Nobody knew it. And still - compromised.

Posted by www_webhost, 02-20-2013, 06:03 AM
I feel like that the emails/spam that is sent out from the compromised machines are playing some role in spreading this attack. Might be the emails that are sent out from compromised machines contain some sort of hidden exploit that when opened on the receiver's end steal some vital info from user system and send back to the hacker. Now suppose if you use filezilla etc etc as root then you have much data left in your system insecure. Also tmp files/cookies on systems have enough data for a hacker.

Posted by kmike, 02-20-2013, 06:15 AM
How could the password be unknown if somebody has changed it on the server?

Posted by dicoemil, 02-20-2013, 06:16 AM
Polina, actually the /home/tmpp/ pattern is the proof of the hacker. Thats what puzzles us before calling it a mallware-root-stealing-password. Because why in the hell somebody with the root password would actually create a /home/tmpp/ to download stuff into, and after go back to root to clean it up... And that "openssh-5.9p1" pattern is interesting... So after all he probably goes via ssh. The sleeps 7200 are just the delays added by spammer between spam waves to not overload the server while spam sending. They like to send spam slow and unnoticed.

Posted by eric92, 02-20-2013, 06:29 AM
I have accessed 2 cloudlinux / centos systems using putty / whm both accessed using root, and both are infected. I am also managing / accessing some windows server 2008 standard r2 based servers. None of the windows based IPs see any indications of spamming. All windows IP are clean in senderscore however linux IP lost a lot of reputation in last 2 days. The servers were also accessed on 3 different systems over the network. On one of these systems we detected a rootkit installed about 3 to 4 days ago which was removed. Just worried about the admin keylogger theory. I am using lastpass to manage all the passwords.

Posted by kmike, 02-20-2013, 06:34 AM
It's just a temporary directory. It could've been anything else. No actual user with that name is created. It's in /home because /tmp and /var/tmp and other usual temporary places could be restricted against execution. /usr could even be read-only, so /home is a safe bet for a writable and executable directory. Also, the commands run by the attacker are most likely automated - it's a script of some sort that creates that directory, downloads/builds stuff in it, installs the malware and removes the directory. I don't see that as an evidence for the penetration vector. It's a directory created after the attacker has already got into the server - maybe for building/installing a bugged ssh daemon version.

Posted by JensL, 02-20-2013, 06:42 AM
Is curl already ruled out? http://packetstormsecurity.com/files...-Overflow.html The binary might be checked but the lib is used in many programs. Is there a pattern in compromised VPS? E.g. Virtuozzo based systems not compromised? Differences in kernels? CVE-2013-0871 http://www.cvedetails.com/cve/CVE-2013-0871/ Would be interesting to see the kernel used in compromised systems.

Posted by patchwork, 02-20-2013, 06:46 AM
I've noticed SSH fails on a hacked system and requires a restart before it will allow you to login. I have 2 servers hacked, both with softlayer. Server one is pretty old (4/5 years old) "REDHAT Enterprise 5.2 x86_64 standard – WHM 11.34.0 (build 11)" Server 2 is very new (3 months old) "CENTOS 6.3 x86_64 standard – WHM 11.34.1 (build 7)" I use Firefox browser and when I checked the add-ons Java was disabled and Firefox said it wouldn't run it until it was updated, and flash had a "severe" warning. Maybe the plugins have just caught us logging in to our server control panels, grabbed the login details then had a party. (but I guess if that was true they could have restarted SSH for themselves)

Posted by Polina136, 02-20-2013, 06:47 AM
How could the password be unknown if somebody has changed it on the server? It means that password was a random series of numbers , chars and so on, generated on that server right after setup and never used again. The other ones were kept in panel, so it was a tiny small chance they could be stolen. I doubt that - i can't imagine hacker entered encrypted intranet that easy ... >> The sleeps 7200 are just the delays added by spammer between spam waves to not overload the server while spam sending. They like to send spam slow and unnoticed. Yes, i understand. I can also +1 your message because killing that "sleep" triggered spam sending activity on a server. Also accroding to environ attacker started/restarted differenct process via SSH. Generally it was sshd daemon. But on CentOS i detected udevd and acpid as well. environ contained something like this: USER=rootSSH_CLIENT= 22MAIL=/var/mail/rootSHLVL=1OLDPWD=/rootHOME=/rootSSH_TTY=/dev/pts/0DPKG_MAINTSCRIPT_ARCH=allDPK G_RUNNING_VERSION= TH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/mysql/bin:/bin:/sbin:/usr/bin:/usr/sb in:/usr/local/bin:/usr/local/mysql/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/ bin:/usr/sbin:/sbinDPKG_MAINTSCRIPT_PACKAGE=ssh-krb5LANG=en_USSHELL=/bin/bashPWD=/home/tmppSSH_CONNECTION= x.x.x.x.x Where "x.x.x.x.x " is just my ip hidden. As you can see "PWD" = "/home/tmpp" is pretty familiar. I posted that in a previous message in grsec output. "TERM=vt100" that thing also looks bad. Last edited by writespeak; 02-23-2013 at 05:11 PM. Reason: Added spaces so that one line wouldn't break the page

Posted by Polina136, 02-20-2013, 06:49 AM
I think hacker used some bug in a system to upload something. Like in my example ssh daemon, which replaced the original one. Maybe like that. That new daemon allows to enter server via hardcoded password and hides hackers ssh session. It runs in memory because as i remember md5sum of sshd binary is ok. I know it is bad to trust rooted system's md5 output and so on but that all i have. By the way "hardcoded pw" theory is supported by the evidence of honneyp log. I just installed it and tried to look what login data was used. IP was contiously trying to login with one password i never used. I changed hoonep's password to that one. One SSH connection ran "echo succeed" command after successfull login. The other IP still continued login attempts without entering pw. And i still think it is a linux kernel problem.

Posted by Moowalker, 02-20-2013, 07:04 AM
Hi, I am following the subject from start and I wanted to chip-in with any help I can provide. I had a compromised server too till I deleted the culprit file 2 days ago. So far, I think, my VPS Server has not been reinfected. If a not mistaken the day my server got compromised was 11/2/2012 and this is the CSF alert. I was searching till now for some evidence why this happened till 2 days ago. Since the start I was using SSH keys login to my VPS and not password. The day my server has this root access, it was the day I send the root password to cPanel helpdesk to help me out with a server problem. I am not sure if it has anything to do with it(changed afterwards and disabled password login). Two days ago, I removed libkeyutils.so.1.9 (I have a backup of it) and it seems (??) that the server is not re-compromised. At least no new file was created. I am running CentOS 5.9 with 2.6.18-308.8.2.el5.028stab101.1. My server is a OpenVZ VPS server. Please do ask me for any other information I can provide. I would gladly help. Regards.

Posted by SafeSrv, 02-20-2013, 07:29 AM
It is in cases where the server is fairly open (Not using CageFS) malware scanners find weak passwords all the time with brute force. Malware scanners will be using this exploit on alot of servers as we speak with most tasks automated for implanting and replacing binarys and checking if they are still "active", most are used for spam botnets, it will enter either via weak passwords via ssh or entry level may be via web applications. Its not all automated as some exploited servers will be from manual entry. I think people are looking too much into this, the answers are pretty much right in front of you guys. It;s only redhat systems that are affected, its not specifically cloudlinux, cPanel, or any other control panel for that matter. It will obviously try obfuscate its connection back to the controller server like most malware, the server based in spain will most likely be one. There can be a few. Regards

Posted by Server Adminz, 02-20-2013, 07:30 AM
We had a client with the same issue a week back. There were these mysterious ssh processes (with notty) that will call sleep, and in between send enormous amount of spam (strace should confirm this). Now, the thing to note here is that notty logins does not get logged in /var/log/secure. You can very easily ssh into a server and not show up in "w" or logs, by using -T option This is exactly how hacker is executing ssh and sending spam through it. You should never run sshd on port 22 to begin with. Also helpful would be to set debug logging in /etc/ssh/sshd_config so that you get all these notty login attempts as well.

Posted by nofz, 02-20-2013, 07:30 AM
Because if you have the root password, you can use a more generic backdoor that works on more distributions other than RPM-based ones. Anyways, I would advise people to use lastpass (password manager) and for the ones who know how to install pam modules use http://code.google.com/p/google-authenticator/ to defeat keyloggers.

Posted by Moowalker, 02-20-2013, 07:37 AM
I just got (or just found out) another of my VPS is compromised: So, I won't delete the file yet in case you need some more information(for a few days). The only thing I did was to change the root password (probably in vain) and disable SSH password login. This VPS is XEN with CPanel 11.34, CentOS 5.9 (2.6.18-308.24.1.el5xen) Regards

Posted by dicoemil, 02-20-2013, 07:42 AM
The attacker has root allready, he does not need to switch to whatever /home partition to execute things. Maybe there is no /home, why switching there when you are in /root with full power... So, thats why is a plus in the direction of whatever is giving him root access, is creating the /home/tmpp/ too and downloads his stuff there + access. On the other side, he goes in via ssh to clean up and backdoor libkey, which means he's not using the backdoor libkey. So the /home/tmpp is just the rootkit deployed automatically, but not installed. You can see him too on the snoopy that he moved the precompiled backdoored libzz8d70 lib to libkeyutils.so.1.9, so that means he is sure what he hacked and most probably was allready prebuilt and is hacking specific versions of something. Would be interesting on a hacked machine to see if are other /lib/whateverweirdfiles left, since he's moving just one file. Usually the hackers patch the hole or protect to be used with same exploit after is infected. Thats why the openssh or whatever is related to that open port may be the vector, probably his infected libkey not only gives him root access but is patching the hole too. (of course on that snoopy log could be commands without any sense, but if we try to put all pieces together and seek patterns sooner or later somebody will notice something/somewhere)

Posted by dicoemil, 02-20-2013, 07:45 AM
I would actually like to have access to some infected machine, just read-only, all the centos'es I have access to are not infected...

Posted by Flegmaa, 02-20-2013, 07:48 AM
I've checked my other machine runing CentOS 6.3 (only Webmin and CS:GO server installed) and the system is not infected. On the other hand, first machine running CentOS 5.9 and cPanel/WHM (with 100+ web sites) is constantly getting infected. Both machines are "updated" from same centos repo in Croatia (Plus hosting).

Posted by Polina136, 02-20-2013, 07:49 AM
One of comprmised servers has prints both of an already mentioned IP and I think they are related. http://cbl.abuseat.org/lookup.cgi?ip= Link contains some interesting info about the spam. was connecting to SSH and as I think sent spam. Maybe it is a combination of SSHD rootkit and the backdoor described in a link. Supports my theory and my obsevations. Indeed, kippo shows very close to "XXXdYZulavB" password: login attempt [root/XXXFwBInnA7] failed 2013-02-19 In case of success: [SSHChannel session (0) on SSHService ssh-connection on HoneyPotTransport,40,] executing command "echo succsdess" Also some more info from grsec: Interesting segfault. @Server Adminz SSH port is not a problem. Hacker tries to login via 22 port. If no success - maybe starts a port scan. Unfortunetly we had no software which detects portscan exactly. But we took precautions because as i said already had looking like that problem with SSH rooting so we brought an app on 22 port which bans anyone who tries to access the server. None of such servers were compromised (but attacker was dected accessing 22 port). The other part of server which had ssh daemon on non 22 port and had no blocking mechanism on 22 port were compromised. So hacker first of all hits 22 port and if that does not work - searches other ports. We detected the activity in DirectAdmin logs as well. Last edited by Polina136; 02-20-2013 at 07:58 AM.

Posted by kmike, 02-20-2013, 08:07 AM
How the password was changed? With "passwd"? If so, I guess it was entered by hand or copy-pasted - which is all the same to a keylogger if it's running on the workstation you are typing on. Hackers have had no problem entering Facebook's and Apple's intranets. It all starts with an innocuous looking Flash or Java applet...

Posted by Polina136, 02-20-2013, 08:18 AM
Sure with passwd and copy/paste. How else? That was really long ago. In 2012 that's why first thought was that our server was attacked the whole year. Even if our password were stolen from our self-made DB then how it happened with others? Very different cases here. It's definitively a mass-root thing. It is too hard to sniff passwords from very different sources and crack many servers within one week. What you are talking about can lead us to "that is all government watches us, they paid linus to make a special bug in kernel in order to take terrorists down some day which came now and woopsie affected normal people".

Posted by keppler, 02-20-2013, 08:24 AM
Did you really have libkeyutils.so.1.9 on your system? Debian often installs the package "libkeyutils1", which provides /lib64/libkeyutils.so.1.3 So, can anyone else here confirm that a system has been compromised where only public key authentication was allowed? If not, it's probably "just" sniffed SSH passwords (I always wonder how many guys allow root login with plaintext passwords...) -Klaus

Posted by Phil @ NodeDeploy, 02-20-2013, 08:50 AM
Yes 1.9 exists, see the post. Also verified that it contains IP's mentioned in this thread

Posted by Polina136, 02-20-2013, 08:59 AM
How did you verify? Want to check mine one.

Posted by Phil @ NodeDeploy, 02-20-2013, 09:06 AM
Couple posts back someone posted a perl script, if you run it and search the file you'll find some IP addresses in it

Posted by Steven, 02-20-2013, 09:29 AM
Why is everyone ignoring this?

Posted by plexy, 02-20-2013, 09:42 AM
anyone affected recently used teamviewer?

Posted by jestep, 02-20-2013, 09:45 AM
I use it all the time and none of my servers are affected. Not saying that means a whole lot though.

Posted by FastServ, 02-20-2013, 09:57 AM
Does anyone use themselves, or know of anyone who uses lastpass that may have had access to an affected server?

Posted by NetworkPanda, 02-20-2013, 10:17 AM
To everyone discussed the scenario about a Windows/keylogger infection, it sounds logical. We are using Mac and Linux boxes to manage our servers and so far, none of our 47 servers have been infected. Of course Mac and Linux systems run Flash and Java too, so a Flash/Java vulnerability could affect Mac and Linux, but we have permanently disabled this plugins and we enable them only when needed on sites we trust.

Posted by matbz, 02-20-2013, 10:17 AM
This could do more harm than good, at least until it is discovered how the file is being planted and the symlink being created, because 1. it will screw with future updates (there will without doubt be a legitimate version of that file which matches the current or any future exploit file) and 2. the hack still creates a symlink to that file so if it is empty it will completely mess up libkeyutils and render a box near useless.

Posted by Zxctypo, 02-20-2013, 10:21 AM
I use teamviewer all the time too (not for all machines) still no compromised machines yet

Posted by kmike, 02-20-2013, 10:22 AM
Because it's incorrect? I still see "Accepted publickey for {user} from {IP}" in /var/log/secure even when using "-T" ssh option. Or do you mean something else in that post? Last edited by kmike; 02-20-2013 at 10:28 AM.

Posted by Majester, 02-20-2013, 10:24 AM
Ubuntu, Mac OSX, and Windows have all been desktops used to access affected machines so it would need to be something infected on all of these OS's for the compromised workstation theory to be true. Host nodes, vanilla CentOS, Debian, and all of the major control panel systems have been affected so it's not sourced by any specific OS, rather it seems to mostly target web hosting servers. Firewalling SSH seems to have stopped the connections from happening on infected machines from what I've seen personally. Removing the library outright is not recommended as it will break any and all services that are relying on it such as SSHD, named, and more. It will lock you out of your system if not handled properly.

Posted by Steven, 02-20-2013, 10:39 AM
You only read half of it.

Posted by streaky, 02-20-2013, 10:46 AM
Scientific. I use windows and none of my servers have, does that now prove it isn't windows? Not for nothing but taking wild guesses isn't going to find the cause. Has anybody contacted ovh to get the box that's obviously at the center of all this at least removed from the internet? Long term it's not going to do much but it'll at least slow them down.

Posted by Polina136, 02-20-2013, 10:53 AM
I think many of us waiting for actions from hacker in order to get some more info on how he got into server.

Posted by Martin-D, 02-20-2013, 10:56 AM
well, the one I've been monitoring appears to be have been compromised again in the last 15 minutes.

Posted by Server Adminz, 02-20-2013, 10:57 AM
I just checked with the client whom I was referring to in my previous post. For him, the spamming has stopped once sshd was moved to a custom port. What he said was, "so far so good.. no spamming anymore but I still get csf alerts about invalid ssh attempts". The argument that port scan can be performed to get a list of open ports is valid, but c'mon are you guys running your servers without a proper firewall that can detect a port scan or a dictionary attack (I am directly referring to csf)? And touch & chattr stuff works for usual php exploits inside /tmp, but someone here has already given logs of the commands that the hacker execute. He manually moves the file from /home/ to /lib64/, so that is not a work-around. I would strongly suggest everyone reading this to please change your ssh port (whether you are hacked or not, doesn't matter). If you are hacked, change the port, get csf installed, run sshd in debug mode and see if any further incidents happen. Maybe all of us here can try this out and help others as well?!

Posted by jestep, 02-20-2013, 10:59 AM
Were you doing any packet capturing? I can't believe we don't have any definitive entry point identified yet.

Posted by finluxismytv, 02-20-2013, 11:00 AM
I think that google is hacked (could be the same like facebook) Look here: IP is part of the Google Network and contains russian porn sites (i think they are advertised in the spam... am i correct?) fx-in-f99#1e100#net a 74#125#39#99 AS sheetporevonow#ru a 74#125#39#99 (none) ergilmakina#com a 74#125#39#99 (none) dltorrents#org a 74#125#39#99 (none) www#pornvkontaktlike#ru a 74#125#39#99 (none) lussac#es a 74#125#39#99 (none) www#sheetporevonow#ru a 74#125#39#99 (none) www#ergilmakina#com#tr a 74#125#39#99 (none) mail#itvbg#net a 74#125#39#99 (none) www#pic-packs#com a 74#125#39#99 (none) www#sexonlinexxporn#ru a 74#125#39#99 (none) www#vietnamtravelx#com a 74#125#39#99 (none) ergilmakina#com#tr a 74#125#39#99 (none) rubop#com a 74#125#39#99 (none) itvbg#net a 74#125#39#99 (none) traxnaozereru#ru a 74#125#39#99 (none) In my opinion Google is hacked or do they now hosting sites like this... i think not. @google create a reachable Abuse Department## i searched 10 minutes for an abuse contact at google never found anything# There is from an other side an a record to this ip (from one of the servers who are used for the ssh keys transfer). The rubop domain is also on the server who is used for the ssh keys... -> Do you use anything from google? Last edited by finluxismytv; 02-20-2013 at 11:06 AM.

Posted by jalapeno55, 02-20-2013, 11:02 AM
Is SSH pass login disabled? And if so, did you change the key?

Posted by Server Adminz, 02-20-2013, 11:03 AM
If you still see "accepted public key for user.." dont u understand the hacker got his key installed in your /root/.ssh/authorized_keys file? That is expected, isnt? Won't you do it if it were you, to ensure you have password-less access to that server? Please review your files inside /root/.ssh/. Better change your keys as well if you regularly connect to other servers from this certain server.

Posted by Martin-D, 02-20-2013, 11:04 AM
No, none. It's currently enabled. I left it as is to act as something of a honeypot to see what came back if anything. Problem is, I forgot to set up the packet sniffers and what not. Been a hectic morning :| edit: worth noting the root pw was changed though.

Posted by gigatux, 02-20-2013, 11:06 AM
It doesn't prove anything, but it does provide some useful information. We are trying to narrow down the initial attack vector here, so if we can figure out that any potential malware only targets one OS, that's at least guiding us in the right direction. So far, I haven't heard of any non-Windows users that have had their servers attacked.

Posted by Martin-D, 02-20-2013, 11:07 AM
A little bit more information. from the logging I did enable, the new IP being used to connect is I can see the sshd callback is being sent to this same IP address, too.

Posted by afletch, 02-20-2013, 11:14 AM
Can you make the libkeyutils from this compromise available for analysis? Also, did you change the root password before or after removing/cleaning the libkeyutils and running ldconfig last time?

Posted by finluxismytv, 02-20-2013, 11:14 AM
What do you think about my google thing?

Posted by keppler, 02-20-2013, 11:15 AM
Maybe just drop a note to abuse@myLoc.de and ask them to inform the server owner about his server being used, while also providing them help in tracking down the issue for more details (myLoc is a major server provider in germany; I assume this is a normal server/VPS and his owner doesn't know about this problem...)

Posted by Steven, 02-20-2013, 11:20 AM
Just an update from yesterday. Still waiting for the servers with a newer openssh to get reinfected. Last edited by Steven; 02-20-2013 at 11:24 AM.

Posted by finluxismytv, 02-20-2013, 11:21 AM
Please send for further investigation also a report to cyber@bka.de, i think they can do more than myloc... (analysis..)

Posted by keppler, 02-20-2013, 11:29 AM
I don't know in which context you detected this Google IP, but just sending some packets to it doesn't mean they're involved too. A quick look at some of these domains showed that they're all registered via different registrars and served by different nameservers; their admins just set their A records to one of Google's IPs (maybe to just "disable" that site, though this is not very elegant). Maybe the malware makes an automated Google query to scan for potential next goals. Maybe one can track this traffic and - if it is not HTTPS - see what is does...

Posted by kmike, 02-20-2013, 11:31 AM
Nonsense. I only tried logging in to one of our servers with "ssh -T", and observed the login information in /var/log/secure to confirm that notty sessions are still logged into syslog. And you made a theory out of it. The level of FUD, misinformation and misunderstanding in this thread goes off the roof. Martin-D, have you seen anything in /var/log/secure at the time of re-infection?

Posted by FuNKyMIkE, 02-20-2013, 11:41 AM
Hello, I have had this certain IP login as root and as a user too : I immediatly changed root password and the server doesn't look compromised. I also gave the root password to cpanel support one week before. Regards Michael Last edited by FuNKyMIkE; 02-20-2013 at 11:49 AM.

Posted by Zimple, 02-20-2013, 11:41 AM
I have started reeving LFD alert But, is this server compromised or false positive LFD alert ?

Posted by NetworkPanda, 02-20-2013, 11:42 AM
The sites you have posted are not working. Some are expired domains, some other are giving DNS errors. The fact that their A DNS entry points to a Google IP does not mean that Google is hacked. Anybody can point his domains DNS entries to any IP address he wants. It is very easy. Since the sites are not opening while pointing to a Google IP means that Google isn't hacked.

Posted by TheVisitors, 02-20-2013, 11:50 AM

Posted by keppler, 02-20-2013, 11:53 AM
But you don't seriously assume that a spammer using this kind of attack requires a mail server software for sending mails?

Posted by Steven, 02-20-2013, 11:55 AM
Everyone should have a read since we are talking about passwords and what not: http://www.baekdal.com/insights/pass...rity-usability

Posted by TheVisitors, 02-20-2013, 12:01 PM
True.... But.... So far so good. As there is currently no "cure" or "solution" to resolve this. This is my limited resolve. And so far, so good. Last edited by TheVisitors; 02-20-2013 at 12:07 PM.

Posted by NetworkPanda, 02-20-2013, 12:05 PM
If somebody wants to send spam, they don't need exim etc to send it. They can write their own software, use telnet or use external SMTP servers. Also, if somebody gains root access to your system, it easy to enable/reinstall exim to send the spam. The important thing is to prevent unauthorized root access to your server and not disabling services which can be easily enabled/reinstalled by the intruders.

Posted by TheVisitors, 02-20-2013, 12:06 PM
http://www.webhostingtalk.com/showpo...&postcount=904 ^ rinse and repeat.

Posted by suhailc, 02-20-2013, 12:22 PM
Hi Steven, Any further info on exactly how the hack is being deployed on servers?

Posted by SoftDux, 02-20-2013, 12:37 PM
Surely if that's the case then any type of server will be hacked. I didn't see any reports of Windows, BSD, Slackware, etc being hacked? And yes the certain file that was hacked is probably easier to hack on an RPM based system, if that's what the hacker is accustomed with. But if he got a keylogger / screen logger to infect Windows PC's then he should be able to hack into any type of server with the credentials he's stolen. Let's also assume the hacking, once he has a backdoor is fairly "automated" which is probably why he targets Linux but that wouldn't rule out BSD, Sun and other Linux based servers....

Posted by Martin-D, 02-20-2013, 12:38 PM
I should probably point out that this isn't a live production server, it's just a 'spare' we have online.

Posted by weredigital, 02-20-2013, 12:40 PM
I'm not positive this is helpful but is new info i haven't seen posted. Since the infection we have started receving numerous log in failures from already blocked hack attempts on file. They are using new techniques using udp we have never seen before. As some mention the hackers were using udp to log in so i figured it may be relevant. below is an exert of message received ..hope provides some use full info PM if you want more. Greg Feb 20 10:26:01 webhost2 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:00:39:66:00:26:0a:27:0d:00:08:00 SRC= DST= LEN=95 TOS=0x00 PREC=0x00 TTL=107 ID=12492 PROTO=UDP SPT=45084 DPT=21430 LEN=75 Feb 20 10:26:21 webhost2 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:00:39:66:00:26:0a:27:0d:00:08:00 SRC= DST= LEN=58 TOS=0x00 PREC=0x00 TTL=108 ID=14252 PROTO=UDP SPT=45084 DPT=21430 LEN=38 Feb 20 10:26:21 webhost2 kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:16:3e:00:39:66:00:26:0a:27:0d:00:08:00 SRC= DST= LEN=52 TOS=0x00 PREC=0x00 TTL=107 ID=14253 DF PROTO=TCP SPT=49301 DPT=21430 WINDOW=8192 RES=0x00 SYN URGP=0 Last edited by weredigital; 02-20-2013 at 12:51 PM.

Posted by Cyclon, 02-20-2013, 12:52 PM
That is probably not related and doesn't really mean anything. Do you have anything listening on port 21430...? # netstat -nl | grep 21430

Posted by weredigital, 02-20-2013, 12:54 PM
No nothing Sorry just changed posting name never noticed I was using old account... Last edited by weredigital; 02-20-2013 at 12:57 PM.

Posted by coldbeer, 02-20-2013, 01:10 PM
For everybody here; I made an IP list, if you miss any please post. btw; I also blocked all French IPs since I read so many comments about OVH. If you use CSF look @ CC_DENY if you want to block a country. Note that I am NOT on a shared server with customers, just private use only. Last edited by coldbeer; 02-20-2013 at 01:15 PM.

Posted by Moowalker, 02-20-2013, 01:26 PM
Anyone else any comment on this?

Posted by huck, 02-20-2013, 01:32 PM
Just wanted to summarize some items as such long threads often run wild with little fact. Compromised System If your server has already been compromised, you will need to migrate/reload your OS. Any other measure are simply work-arounds to buy you time until you can move your operations to a known, secure environment. The presence of the libkeyutils.so.1.9 is evidence of a backdoor and if found you should investigate further. There is NO publicly confirmed exploit in SSHd. Based on information here and from other sources, there is no clear indication of a problem in SSHd itself. There is nothing from OpenSSH on this matter. As of now, I believe current versions of SSHd are safe. Some Observations If SSH was the attack vector, then I suspect more chatter in other Linux circles -- not just the hosting community. Facebook, Apple, Twitter and others have all been recently attacked via a Mac Malware. Macs are heavily used in the development community. I am still researching what the payload was from these attacks. See: http://www.f-secure.com/weblog/archives/00002507.html On 200+ systems with good SSH policies and root access management, we have seen no issues. Command captures have shown a manual replacement of the libkeyutils. This is more indicative of someone putting in a backdoor after they already have access - not them gaining access. This is further evidenced by someone manipulating SELinux settings. Theory Desktops or devices are being compromised via local exploits. The compromises are permitting root passwords to be stolen. Attackers are then using this access to backdoor the system for future use. This may be related to the recent attacks on Mac. I know Macs are used heavily in the development communities. Recommendations Disable direct root logins.Use SSH keysForce the use of HTTPS in WHM.Scan your system for malware.Change root password.

Posted by patchwork, 02-20-2013, 01:36 PM
After reading this thread it seems to me this is probably a server only based security hole that allows the hacker to upload only a small amount of data that gives them very very limited access (just a few commands as posted in this thread). The few commands are then used to install the new file. This causes SSH to partly fail (as I said in my previous post, on a compromised machine SSH seems to need a restart before you can use it) WE then restart SSH and this is when he probably gets your login details via the fake DNS request. If this is the case then this rules out local PC's having a key logger etc.., it also rules out Java or Flash hacks etc .... Maybe what we should all be doing it getting a FULL list of all the software/scripts (and versions) we use on our servers and see if we can find a common denominator.

Posted by weredigital, 02-20-2013, 01:38 PM
Huck Are you able to confirm new installs are not being affected? As of yesterdays some contributors posted that even with new installs they were still getting affected quite quickly.

Posted by unSpawn, 02-20-2013, 01:40 PM
I agree. Seeing this thread accumulates aprox. 10 pages a day w/o end at this stage we need to scale up and escalate. I sent SANS ISC a tar ball & notification earlier today asking them to look into it because the more exposure / experienced eyeballs the better. * Esp. Steven and Brad: please share evidence and findings with handlers @ isc.sans .edu, TIA.

Posted by weredigital, 02-20-2013, 01:41 PM
Has any one considered the tool used to SSH into the box? is it possibly what is giving access to hackers? Is it compromised in some what? ie putty? is there a common tool used possible

Posted by fishfingers, 02-20-2013, 01:41 PM
We also had a root login from this exact IP address, 4 days after handling cPanel support in January.

Posted by huck, 02-20-2013, 01:45 PM
If you are already compromised, the only way to properly clean this up is to reload or OS or move to a known good OS. To date, we've not had any issues on the servers we manage, so there's no cleanup to do. While the libkeys has been the subject of this thread, I would not be surprised for new backdoor payloads to be developed or used as cleanup procedures are developed.

Posted by weredigital, 02-20-2013, 01:49 PM
I agree that is the right way to go my point is users were claiming that even new installs since the attack were also compromised . Re-installing a server with hundreds of users is a ton of work to go through to be simply doing it again in a few hours once re-infected. What is your configuration that you using is it different to all others? ours was totally up to date and we though secure as well but

Posted by patchwork, 02-20-2013, 01:50 PM
New servers have been hacked within 2 hours, although this could be miss information posted to throw everybody.

Posted by TheVisitors, 02-20-2013, 01:51 PM
I understand that reading 60 pages of this thread is a lot...... So.... summery..... Format C:\ Re-install OS upgrade all Does not prevent you from being infected again.

Posted by TheVisitors, 02-20-2013, 01:54 PM
putty filezilla ^ Only 2 tools I use to connect to my VPS

Posted by Cyclon, 02-20-2013, 02:09 PM
This is exactly what I think. I don't think that login credential are stolen from desktops, because 99% of reported hacks had used standard SSH port. Few that didn't use standard port probably just didn't have any port scan detection. If there is malware on desktop that is stealing credentials it would also be very easy to steal SSH port, and we would have much more compromised servers with SSH on different port.

Posted by nicknn, 02-20-2013, 02:19 PM
Hello Has any server with custom firewalled ssh port been compromised?

Posted by jalapeno55, 02-20-2013, 02:30 PM
Good point but also keep in mind that there are a lot more servers out there with SSH on port 22, rather than another port, plus people who use a non-standard SSH port probably have better security practices in general, IE. it would be a lot harder to infect them with malware, and to keep the malware hidden from them, plus they may use Linux as their desktop, and their servers may have pass login disabled, or the SSH port protectec by and ACL.

Posted by finluxismytv, 02-20-2013, 02:33 PM
What is with this one?

Posted by Patrick, 02-20-2013, 02:35 PM
That's unrelated to the issue at hand. Please stop grasping for straws... NEXT.

Posted by cloudhopping, 02-20-2013, 02:44 PM
We have seen this on a number of our VPS I am not thinking it is desktop related unless the MAC we use for SSH is not secure. Steve has managed a number of servers for us in the past -and I trust him completely. that being said - on one system we allowed to stay online with this - (well we cleaned it up but it came back fairly quickly) they did something a bit odd and its a real pain. they installed the EL4 repo for RPMForge. (we have only seen this on the latest box that has the rootkit - so thinking it is fairly new uninstalled / reinstalled a few things - and so now that system is a complete nightmare. while websites run - they will do so only in DSO mode. FastCGI - good luck It took us a bit of time to find it - but now we have some inconsistencies with EL4 and EL6 repo's I am hoping we are closer to a lockdown method - but for now I know that 2 way authentication is the right way to go. The server should ask our cells or similar @ least now if someone can ssh in. I love that idea btw of getting a few of the folks here to have permissions to cludge out the rest of the noise - going over 60+ pages over the last week and not being any closer to a resolution is a real PITA. Steve If you want access to this or any other box - please pm me.

Posted by Cyclon, 02-20-2013, 02:47 PM
Yes, I completely agree with that also, and I can not completely rule out desktop infections, but again, only few compromised servers with different SSH port and none of those reports mentioned that server had a port scan detection, so that would be the same as SSH is on the standard port. Other strange thing is, why attacker is creating some temporary /home/* directory and than moving some files and renameing it. If it has a standard SSH access with stolen credentials it would just "wget" library directly to lib folder. Why this...? Why changing file owner to root? Who was the previous file owner?

Posted by Steven, 02-20-2013, 02:52 PM
Can't pm you, as you don't have enough posts. email steve@rack911.com

Posted by dicoemil, 02-20-2013, 02:57 PM
I'm interested to look at the abused server too (won't change anything). I'm on skype online if you want to share.

Posted by serve-you, 02-20-2013, 03:02 PM
I know this should go without saying, but please be smart people. Do not give random people from the forums access to your server! Especially not some random guy from romania who just joined the forums yesterday. No offense dicoemil.

Posted by dicoemil, 02-20-2013, 03:13 PM
I think some basic research can show you that I own a web hosting company Yes, I know Romania has bad reputation for a lot of things, but on the other side don't forget that romanian sysadmins are v. good. I just made the new WHT account, because my old one I don't know if was deleted or what user/pass/etc/had. I'm not spamming this forum usual, thats why I don't care about status/reputation. But this time I'm interested in this matter... The servers are allready hacked and if we don't find the solution, will be more, so why you should even bother if one dude in +- will have access to that server...

Posted by serve-you, 02-20-2013, 03:17 PM
That clears it up then. Feel free to hand over your credentials to anyone, since your server is already hacked

Posted by TheVisitors, 02-20-2013, 03:22 PM
Actually.... It depends on who's asking Steve was quick to send me PM requesting if he could take a look. I forward my details, but have no idea if or what he found. But of course Steve here is very trusted and well known.

Posted by Steven, 02-20-2013, 03:23 PM
I think I may have accidentally deleted your PM as I didn't see one. My inbox box was getting full so I wiped everything so perhaps you sent it as I was wiping it.

Posted by serve-you, 02-20-2013, 03:26 PM
This was my point. Of course we know Steven, huck, and a few others here run server management companies and are very well known. Random people who just appear in this thread out of nowhere should not be trusted.

Posted by Steven, 02-20-2013, 03:39 PM
How many people have had reinfection? Show of hands please? If so how fast did it occur? I have been waiting patiently for reinfection to occur on a number servers I 'cleaned' and have been watching. Not happening yet.

Posted by weredigital, 02-20-2013, 03:41 PM
We had two not sure if we cleaned completely though i also pm you did you get it?

Posted by deriamis, 02-20-2013, 03:44 PM
I've just read through this entire thread, and after a few coffees and pee breaks and sleeping for a few hours, I noticed four things: 1) There is still some debate over whether the compromise was local and then the hacker used stolen credentials, or whether this was some sort of daemon vulnerability. It's going to be harder to find the source of the local compromise, but one question I saw asked still has not been answered: were any of the comped servers running a grsecurity kernel? And if so, were the PAX flags being managed by RBAC? If we know boxen with grsec kernels are being comped, then it stands to reason that it's one heck of a multi-platform compromise effort. If no grsec boxen are being compromised, then it's much more reasonable to assume we have an exploit somewhere. 2) If this is some sort of local compromise, then it's going to be difficult to pinpoint an exact cause until somewhere farther down the road. It's more believable that there is something common to all of the machines affected that we're missing. My vote is OpenSSL - it has a history of serious security flaws, including one remote code execution exploit, and it's common to a lot of services on a running box, including SSH. Has anyone checked to make sure someone didn't drop a patch out of a package by accident? It's happened before. 3) Also, has anyone noticed what a "good" idea it was to use libkeyutils.so as a re-entry vector? Several others libs depend on it, including libkrb5, which is common to a lot of services. If you wanted to make sure there would be a copy of your lib running to give you access, even if the admin has "fixed" sshd, that's a sweet spot to pick. 4) In the interest of "thinking like your attacker", one thing about all this stands out to me: whoever this is, they have been doing it a while, they are doing it relatively quietly, and the motivation does not seem to be fame or external gratification. That person also appears to be building a farm of some sort, complete with hot spares and an active pool. This rather handily explains why tactics have changed and why not all compromised boxen are being used for something yet. The above may be obvious, or it may be nonsense. Either one is believable after reading sixty-three (!) pages of this. But it's still a morbidly fun read in any case. Thanks to everyone for investigating this problem - you're all doing a lot of good work out there.

Posted by 'David, 02-20-2013, 03:49 PM
I have an Amazon EC2 Instance and this is what I found: Am I exploited?

Posted by HrIT, 02-20-2013, 03:59 PM
I manage 30 + servers and I didn't see any compromise. Most of servers are cpanel based, some are centos 6.3 and some are 5.9 . I login to servers via ssh gateway that is firewalled from internet (in my office, available only from LAN) , ssh key is on that ssh gateway. Password auth is allowed and none of the servers have standard ssh port.

Posted by Flegmaa, 02-20-2013, 04:11 PM
Yes. Mine.

Posted by dicoemil, 02-20-2013, 04:23 PM
I don't like trolling... people should be constructive. Now if you want to wait for days until that hacker will make a mistake or something will go wrong, then is up to you. I just wanted to help. Now, considering that people ran scripts which crashed the servers, and they did not care about credentials... They should think about this credential problem when they let the server online with poor monitoring. Now, you can wait until somebody else will get the exploit too, and post your users on pastebin... or you will be lucky and some sysadmin with an abused server will find the problem. On the other side, if you preffer I can put up a sysadmin support package you can pay, if you dont like it for free and you feel more secure like that Or if an avatar and a signature on a forum and a paid vip status is more secure... But as you can see, I did not even advertise myself or my company, I just want to see this bug fixed and story ended. So, I will wait for a desperate sysadmin or an abused centos from friends, whichever comes first...

Posted by coldbeer, 02-20-2013, 04:24 PM
Please stop trolling. You're another troll here. Please ignore users that joined earlier than a month. You can block these IPs in your firewall to make sure they can't connect to your server(s). We should make a list together. Btw, I temporarily blocked France in CSF (set CC_DENY to FR) to make sure French OVH IPs can't connect anymore. Last edited by coldbeer; 02-20-2013 at 04:38 PM.

Posted by PLE, 02-20-2013, 04:38 PM
No, libkeyutils.so.1.4 is the default version from Ubuntu 12.04. The hacked version is libkeyutils.so.1.9

Posted by hb9aj4fn, 02-20-2013, 04:39 PM
CentOS just released a update that I first thought might be related, but something is wrong and I am confused. Look at this update from today: [CentOS-announce] CESA-2013:0271 Critical CentOS 6 libproxy Update http://www.mail-archive.com/centos-a.../msg07244.html But that page link to the firefox security update from RedHat https://rhn.redhat.com/errata/RHSA-2013-0271.html And when I look after the "libproxy" update at RedHat, it is nowhere to be found: https://rhn.redhat.com/errata/rhel-server-6-errata.html

Posted by GoTek-JP, 02-20-2013, 04:47 PM
We just got our first known case here, client has mod_sec setup.

Posted by Patrick, 02-20-2013, 04:48 PM
Completely unrelated. NEXT.

Posted by coldbeer, 02-20-2013, 04:49 PM
mod_sec? Can you please write a proper text so we all know what you mean by 'mod_sec setup'.

Posted by hb9aj4fn, 02-20-2013, 04:51 PM
It does NOT EXIST - so how can you tell anything about it. NEXT NEXT NEXT, STUPID NEXT - you said that way to many times in this thread. I am ignoring all your post in the future.

Posted by Darkimmortal, 02-20-2013, 04:57 PM
Has anyone running Arch Linux or a similar unpatched distro been affected yet? Looking at two boxes here, one updated daily, the other not updated for 6 months, neither has been affected. Also neither has any 'fancy' security beyond the likes of anti-bruteforce firewalling and proper configuration. If it's a 0day in some common windows application, I'd be surprised as I run no realtime anti-virus. Edit: SSH on standard port on both, keys only on one of the boxes Last edited by Darkimmortal; 02-20-2013 at 05:07 PM.

Posted by flopunctro, 02-20-2013, 05:04 PM
Please ignore users that joined earlier than a month. Sure, always a good idea! Newcomers are nothing but trouble. /s dicoemil: please don't offer to fix people's servers for free. There be dragons that way That being said, let's get constructive. I am a freelance sysadmin, I administer ~80 linuxes, mostly CentOSes installed by me and up-to-date. I have checked a handful of them and found no infection so far. They are very diverse: CPanel, DirectAdmin, mailservers, internet routers, fileserver. Some have ssh on 22, some on nonstandard ports. Some have password auth enabled, some don't. Most of them are CentOS, and some of them are Debian and Ubuntu. The common denominator is that only I have root access on them, and I login as root with key. I say these because a previous post ringed a bell for me: there is a guy here that sent his root password to CPanel support, and a few days later his server was compromised. Maybe CPanel had some kind of data leak ? Until 2 years ago, I worked at some online-solutions software shop, and during my last months of work there I have fought a targeted attacker with a similar Modus Operandi. While I haven't found the entry vector, the compromise was always like this: - (somehow get root) - download a "trojaned" openssh kit, compile and install it - the compromised ssh and sshd would send a UDP packet on all incoming and outgoing password logins -- even on failed ones. The attacker was stubborn enough to repeat this over and over again, and also not skilled enough to evade our booby traps. So I managed to intercept his commands and the source code of his trojaned openssh. However, my C skills are not good enough to attempt any analysis. Now, me being an old rkhunter user and its mailing-list lurker, I hope that the UnSpawn that commented here a few pages ago is the same UnSpawn that's a maintainer for rkhunter. If yes, UnSpawn: you might remember I sent you this tarball and the commands history, on 2011-03-01, with the hope that you might include some signature (strings in the binary files, or you-know-better-what) with the hope that the next version of rkhunter will detect this type of compromise. Now, if somebody is still interested in that tarball, just ask; i'm gonna be around for ~1h and refreshing this thread. Also, I wouldn't mind a more real-time discussion, maybe IRC or skype. Flo

Posted by Patrick, 02-20-2013, 05:13 PM
The fact that you think libproxy has anything to do in the SLIGHTEST with the exploit sums up your knowledge about what you are trying to help with... Good intentions are great and all, but at least know what you are referencing.

Posted by Zadmin, 02-20-2013, 05:16 PM
flopunctro would you please mail ur findings to kre80r [at] gmail

Posted by jalapeno55, 02-20-2013, 05:18 PM
That is the same UnSpawn, please share evidence and findings with handlers @ isc.sans .edu. UnSpawn contacted Sans and they are investigating this.

Posted by yourwebhostereu, 02-20-2013, 05:22 PM
You'll find the libproxy updates here: https://rhn.redhat.com/errata/RHSA-2013-0271.html The Advisory of that page is RHSA-2013:0271. Now search for that here: https://rhn.redhat.com/errata/rhel-server-6-errata.html Edit: in other words there is nothing to worry about. Just look better

Posted by patchwork, 02-20-2013, 05:22 PM
A link to the tarball would be good :-)

Posted by flopunctro, 02-20-2013, 05:39 PM
I'd rather not make suspicious code publicly available. I can email it to you.

Posted by demil, 02-20-2013, 05:40 PM
Well, this account is better if is older? Now my trust increased with +1? The ips you post above, I posted them few pages back, if I can't be trusted, then why are you using them? Filtering those IP's wont fix your server, sooner or later some more IP's will popup. Like today ip appeared, so you can add that too. And yes is "smart" if you block countries If you have no clue what RNB is, then here is some news for you. I'm pretty sure they have abused machines from where they can connect to your server on each country on this planet, probably even North Pole... Thats why I want to find this bug, because they are skilled compared with most sysadmins.

Posted by plexy, 02-20-2013, 05:43 PM
Dont tar us all with the same brush. Some of us have just forgotten our login details and our old rescue email accounts are connected to long expired domain names http://www.webhostingtalk.com/member.php?u=125335

Posted by demil, 02-20-2013, 05:47 PM
And actually plexy had the best input/debug in last 24 hours. So, any more news for tonight from that abused server? @flopunctro as you know the problem is not the backdooring right now, is how the attacker downloaded the backdoor on the server and more important how he got root on such a big variety of configurations.

Posted by patchwork, 02-20-2013, 05:48 PM
Thanks flopunctro patchwork14 hotmail dot com

Posted by lhg32, 02-20-2013, 05:50 PM
flopunctro, Could you send me a copy? jake.alexander at runbox dot com

Posted by plexy, 02-20-2013, 05:53 PM
not much from me. I had a bunch of invalid password logins in /var/log/secure from the usual suspect IP's after I removed the exploit. But, I had not changed the root password (on purpose as I was PCAP'ing). So if someone had compormised the root password for this machine, then why would they not try that and instead just try the password embedded in libkeyutils.so.1.9? Putting on a white hat for a second, if I had 2 passwords, a one from a rootkit and a one from a compromised user, I would not try just one and then stop when I get no where. I try them both. That didnt happen. Compromised root passwords is a good possibility, but for some reason its just not gelling with me right now. Call it a hunch.

Posted by moondog13, 02-20-2013, 05:57 PM
The InfosecNewsBot sent out this tweet which has links to this thread as well as some other good information. It looks like some malware scanners are starting to pick this up: Linux/CentOS SSHd Spam Exploit — libkeyutils.so.1.9: Someone shared a sample of the Linux root... bit.ly/12O9kKL #infosec #malware

Posted by Olly-ellogroup, 02-20-2013, 05:57 PM
I am 100% with Steve on his theory of local machine hacking. Reading this thread in Chrome, e-mail from CSF: "lfd on [server]: WHM/cPanel root access alert from [my home IP]" SSH inbound is firewalled on this server, this came in via WHM. No tab open to that server in Chrome, no passwords saved in Chrome. Lastpass never used. I have a local file with passwords in it (yeah, insecure!) and likely had the password to that server in my clipboard. Windows 8 x64, Panda Antivrus, MalwareBytes Pro (realtime shields active). TeamViewer, Last.fm, Spotify, FileZilla, Putty, Trillian, MS Word, Winamp, Dropbox running. Chrome Plugins: default + Java (latest). Chrome Extensions: Checker Plus for Gmail, Google Docs, Google Tasks (by Google), PageRank Status, Speed Dial, Thin Scroll Bar, TweetDeck, Yet another flags. Any other info you need to know, please ask.

Posted by coldbeer, 02-20-2013, 06:01 PM
Use Adblocker

Posted by egillette, 02-20-2013, 06:01 PM
Hey all I'm posting this based on the suggestion of someone who suggested I modify the script a bit to user auditctl, which I've done. Anyone interested can get it here: https://www.ericgillette.com/clients/exploit-cleanup Then: Should match: 35d43e7a7294c7d28255d0d4ca3f135e Then: For users who would like to know and read through what this script does: This will attempt to move the file into a directory where if requested, you can supply the file to requestors here. In addition, it will attempt to symlink to an existing library on your system that *may* or *may not* be accurate since system configurations, and versions vary somewhat -- that said you can execute the commands in the script individually if you prefer, otherwise you use it at your own risk. Ironically, I've received multiple messages from various users who have thanked me for creating the script, incorporating some of the suggestions of others, and for maintaining the script up to this point, despite the negativity and unhelpful attitudes of some. That said, to those that have expressed their opinions concerning the script prior, your opinions will have no effect on what I decide to do, so it's probably better to just keep them to yourself, rather than cluttering the thread with more of your opinions. As I said prior, agree to disagree, and move on -- find something new to have an opinion about, you'll be better off. To the users who privately thanked me, and asked me to continue maintaining the script -- thank you very much, and I'll continue to do the best I can, while the others investigate, because I do not have the time to investigate on an ongoing basis like some of the other guys do. I do have some quarantined files both 32-bit and 64-bit if anyone needs them, though I have posted them previously -- just PM me and I'll be happy to provide the files in both 32-bit and 64-bit models.

Posted by demil, 02-20-2013, 06:05 PM
As far as I can see right now, on the logs I monitor the bot does no care about logging clearly he does not verify if the password still works. So, the hacker probably had no root password, but he logged in just once to install and after he does not care anymore about comming back unless the libkey will ping him with a new user/pass or something. Or he has priority on new hacked and not blacklisted servers So please, who have abused servers make a cron monitoring script to check for /home/tmpp and if exists to stop the networking on that server so you can preserve whatever the botnet drops there. (and hopefully the servers will be hacked again) Last edited by demil; 02-20-2013 at 06:10 PM.

Posted by Zadmin, 02-20-2013, 06:37 PM
Old sample I got from flupcntro similar to what we are facing now a sample of history shows stuff like : perl -e 'print "abcdefghijklmnopqrstuvwxyz\nbiz\ninfo\nnet\n";' >> 1.tmp which matches parts of the de-obfuscatd code we have

Posted by Zimple, 02-20-2013, 06:51 PM
Anyone noticed this ?

Posted by jalapeno55, 02-20-2013, 06:52 PM
Thanks for making the script. I've had to use scripts like this before in a pinch and they come in handy. Its useful not just for the people infected today but anyone who gets infected by this a year or two down the road.

Posted by ramnet, 02-20-2013, 06:52 PM
Seriously, you need to stop distributing that script before you break anyone elses system. If you had any sense your script would do basic sanity checks (checking that /lib64/libkeyutils.so.1.3 or /lib/libkeyutils.so.1.3 exists) before moving a critical (if infected) system library and symlinking to another library that may or may not exist. You have already broken several people's systems due to your lack of sanity checks. Something like this: You need to fix that before you break anyone elses system. These simple changes at the very least would make your script safe to run on non-RHEL/CentOS 6 based systems.

Posted by ramnet, 02-20-2013, 06:58 PM
I am as well. nenolod and Steven actually have a copy of the rootkit keylogger that has caused this. It affects workstations and sends out keystrokes in dns packets out port 53. He used this infected workstation system to login to a honeypot and a few hours later that honeypot was hit. IP's all match the suspect IP's here. If you have a server affected by this, your workstation has been compromised.

Posted by jalapeno55, 02-20-2013, 06:58 PM
Layoff the script already, we're all fellow members of the opensource community, a community built on people writing scripts and code and freely distrubuting it for other people to use and add to. If you want to modify that script, or create your own script you're more than welcome to do so.

Posted by Hellsheep, 02-20-2013, 07:03 PM
Thanks for this, I concur as well as malware was detected on my local PC after running scans. (Appears the malware entered through some sort of java exploit at least on my machine) unsure if this is what actually caused the compromise in my case however it makes sense. Do you have any more info on this rootkit keylogger so i can have a look over it?

Posted by ramnet, 02-20-2013, 07:05 PM
His script will take down any system that is not a CentOS 6.x or RHEL 6.x system. His script as it stands right now is more dangerous to the stability of your server than the exploit he is trying to fix. All because he can't be bothered to do a basic one line sanity check.

Posted by ramnet, 02-20-2013, 07:06 PM
PM nenolod or Steven for a copy of it.

Posted by tecnobrat, 02-20-2013, 07:07 PM
What desktop OS's does it infect? Windows only?

Posted by tarsiran, 02-20-2013, 07:09 PM
so what's the best way to fix this infected? my server has infected

Posted by ramnet, 02-20-2013, 07:11 PM
It infected a Windows 7 workstation system, but other OSes may also be affected.

Posted by Patrick, 02-20-2013, 07:11 PM
1. Make sure your local PC is secure + updated since that is the likely point of entry. 2. Reload your server OS. Once root has been obtained the server integrity will never be 100% trustworthy. You have to reload and start over for true peace of mind.

Posted by jalapeno55, 02-20-2013, 07:13 PM
It would work on 5.x if libkeyutils.so.1.3 is changed to libkeyutils.so.1.2 Why don't you add a check for the OS version if its so dangerous?

Posted by tarsiran, 02-20-2013, 07:14 PM
thanks dude but i don't want to OS reload i wanna fix issue only

Posted by ramnet, 02-20-2013, 07:15 PM
I already provided him code to use so his script won't damage systems it wasn't designed for. Since people are linking to his script, and since his script (or any other script for that matter) doesn't actually fix the issue anyway, I won't be writing one myself. You have been rooted. An OS Reload is the only fix.

Posted by jalapeno55, 02-20-2013, 07:17 PM
What OS and version do you have?

Posted by weredigital, 02-20-2013, 07:17 PM
What malware is infesting the desktop I have scanner many times find none. can someone elaborate on what it is how to remove or suggest another tool to look for it?

Posted by matbz, 02-20-2013, 07:20 PM
Your box has been rooted eric, you don't have any other sensible choice.

Posted by weredigital, 02-20-2013, 07:20 PM
no choice if you were rooted install only fix

Posted by GoTek-JP, 02-20-2013, 07:21 PM
Mod Security

Posted by ramnet, 02-20-2013, 07:22 PM
This is from about 8 hours ago:

Posted by pjf5000, 02-20-2013, 07:23 PM
That is what my gut told me and I hope this is indeed the case... The only thing that doesn't make sense to me is I have a client with 3 CentOS/RHEL 5 cPanel servers, one of them is infected with this, the others are not. They all have the same configuration, the client logs into all three servers pretty consistently from the same workstation I would expect all three of them to be compromised. Of course there are other variables at play here... Just trying to wrap my head around if this is indeed 100% a local workstation issue. Thanks all for the work you guys have done on this. Hopefully some sort of AV signature will be made to detect this shortly.

Posted by The3bl, 02-20-2013, 07:24 PM
What scanner are you using? First if it is something like Norton or God forbid Mcafee then forget it. First thing a good java infection does is plant itself in and hides from what ever virus scanner you have installed on the system so it becomes useless at finding it. Download a new scanner I like Avast myself since it was the only one to reliable detect the Gumbler virus when it was first out. Disable your current scanner and let the new one scan the system. Don't trust what you already have installed.

Posted by matbz, 02-20-2013, 07:28 PM
May we ask, how or with what software was the root-kit discovered? Time for everyone to scan for it perhaps? Last edited by matbz; 02-20-2013 at 07:28 PM. Reason: typo

Posted by weredigital, 02-20-2013, 07:31 PM
will try your suggestion i suspect it maybe a login from home

Posted by ramnet, 02-20-2013, 07:51 PM
ramnet> nenolod, how did you find that windows rootkit earlier? where was it's payload located? ramnet, i just ran malware bytes anti malware, and it was a file in C:\Windows\System32 running as LOCAL_SERVICE permissions

Posted by matbz, 02-20-2013, 07:58 PM
Thanks very much ramnet!! Potentially the most useful post in this entire thread Anyone affected by this should do the same asap, and please post back to the community with their results.

Posted by GoTek-JP, 02-20-2013, 08:02 PM
Can anyone confirm if servers without CSF have been infected ?

Posted by brianemwd, 02-20-2013, 08:03 PM
It doesn't have to be a local computer that could be infected. Somebody earlier mentioned that it was possible that techs at cPanel (for example) could be compromised: The ONLY server that got hit on my network was the one server that I gave cPanel my root password to investigate an issue I was having. All my other servers have been fine and my computer at my office is the only computer I use to access them. So as I said, it is very possible that computers at places like cPanel can also be compromised.

Posted by coldbeer, 02-20-2013, 08:05 PM
DirectAdmin servers are also infected.

Posted by ramnet, 02-20-2013, 08:06 PM
That's correct. Any workstation used to login to any given server is an attack vector. That includes people with multiple computers at their home or office, people you hire for server management duties, datacenter techs logging in, and anyone else that had your server's login info on their computer and used it. Last edited by ramnet; 02-20-2013 at 08:11 PM.

Posted by GoTek-JP, 02-20-2013, 08:07 PM
snipped>> So far we have had a few infections the only constant(most likely a coincidence) is that they're running CSF. Just throwing it out there because CSF does have an auto update feature. Last edited by bear; 02-21-2013 at 09:14 AM.

Posted by brianemwd, 02-20-2013, 08:07 PM
Yep and my post said "for example" to reflect that. No where did in my post did I imply that ONLY cpanel servers were being infected. All I am saying is it is very possible that if a work station exploit is behind these attacks then it would be possible that companies such as cPanel (and there would be others) could have some of their techs compromised.

Posted by Hellsheep, 02-20-2013, 08:10 PM
ramnet, your posts have been most useful. Has there been any indication that the malware on windows machines has the ability to spread over local networks, infecting other vulnerable machines on your network. Has there also been any indication that there may be some sort of packet sniffing occurring? I am wondering if there are other methods than just keylogging the machine it's on. If this is the case then may need to check all machines in your local network or networks that have been used to access your linux servers.

Posted by ramnet, 02-20-2013, 08:14 PM
Based on what I've heard I don't believe the malware propagates over a local area network, as a single workstation in an office environment was compromised while other similar systems on the LAN weren't. That doesn't mean it isn't possible though. nenolod and Steven are still busy analyzing it.

Posted by TheVisitors, 02-20-2013, 08:23 PM
I would hope that goes without saying I have noticed a bunch of "noobs" though suddenly asking people.... Can't imagine anyone stupid enough to go along with though.....

Posted by GoTek-JP, 02-20-2013, 08:30 PM
Ok nevermind, we have servers with CSF that haven't been infected.

Posted by weredigital, 02-20-2013, 08:31 PM
i appreciated csf software very much but when this is fixed i will be so glad to get rid of the warning every few minutes. i know i can turn them off leave them there to annoy me and be more careful in future.

Posted by matbz, 02-20-2013, 08:34 PM
Does putting the file in csf.ignore make the warnings go away? If you personally already know it is there, and you don't want reminding...

Posted by Steven, 02-20-2013, 08:37 PM
lib64/libkeyutils-1.2.so.2 is popping up on centos 5 machines.

Posted by jrositas, 02-20-2013, 08:38 PM
I'm running a VPS hosting Minecraft and OpenVPN managed through SSH on Ubuntu Server 12.04.2 LTS. I configured 2 factor authentication for login and use a password for access. No public/private keys or anything. I haven't been hit yet. Just reporting in.

Posted by weredigital, 02-20-2013, 08:40 PM
I'm leaving it as punishment and reminder to be more vigilant. It will make me a better admin...

Posted by matbz, 02-20-2013, 08:41 PM
It's the right file version for Centos 5, that's why that 'script' was screwing up some boxes, but you quote the filename with '.2' on the end? Last edited by matbz; 02-20-2013 at 08:44 PM.

Posted by The3bl, 02-20-2013, 08:42 PM
Have you looked at what is inside it? Same as the other or has the code changed?

Posted by matbz, 02-20-2013, 08:43 PM
lol, don't be so hard on yourself. Maybe just create a filter in your mail program to put the lovely emails in a 'special' place?

Posted by Steven, 02-20-2013, 08:45 PM

Posted by serve-you, 02-20-2013, 08:52 PM
You'd be surprised. Some people are just so frazzled by being hacked that they'll do just about anything without thinking about the ramifications. People will execute random code, and follow advise of random strangers in effort to resolve whatever issues they may have. Obviously some people have only the best intentions, but there are plenty of "bad guys" (or people with good intentions, but lacking knowledge) out there as well. With all the people coming into this thread out of nowhere crying for help, I think it sadly needs to be said that not everyone here can be trusted.

Posted by matbz, 02-20-2013, 09:02 PM
Did someone here get really mad and cut OVH's fibre? http://status.ovh.co.uk/?do=details&id=4165

Posted by Hellsheep, 02-20-2013, 09:02 PM
Funny, That host is listening on port 25 and 22, 22 is open to the world and you can ssh to it as root. Out of interest I tried what we believe to be the hard coded password but it didn't work.

Posted by matbz, 02-20-2013, 09:04 PM
1 Good common sense should prevail at all times. Some of us are not new here but have not been in for so long that we forgot our WHT login, linked email and pass so created new accounts.

Posted by TheVisitors, 02-20-2013, 09:13 PM
So can anyone confirm that this is also happening to people using Debian or Ubuntu? We normally use Debian, but decided against our better judgement to give CentOS 6.3 a try so we could load up cPanel. Mostly because we've been short on time and simply wanted a more "point and click" setup. So has this indeed been hurting people on Debian .... Can this be confirmed?

Posted by Steven, 02-20-2013, 09:15 PM
There was someone who mentioned it.

Posted by TheVisitors, 02-20-2013, 09:18 PM
I noticed it by a few 1 post wonders.... .... Was wondering if anyone else could confirm it though. I setup Debian VPS yesterday with no security (none) and its still ticking away. I would not be all to surprised if Debian was in fact not compromised... And maybe someone wants people to think no place is safe.

Posted by Steven, 02-20-2013, 09:25 PM
The same could be said for many centos servers, there are lots of people with zero infections.

Posted by streaky, 02-20-2013, 09:25 PM
Yeah because these people don't know that their boxes are still rooted. Yup, much better. Well, more hilarious to have people running around wonder why they keep getting rooted when they're running a sooper-dooper cleanup script anyways.

Posted by TheVisitors, 02-20-2013, 09:29 PM
True, but people who were infected got re-infected. I installed Debian on the same VPS, on the same IP address, port 22, with no security, and the password for root was password. I made it so easy that I'm surprised some random bot had not just taken hold... And yet no re-infection so far 24 hours later. So can anyone with maybe a little more credibility (not just 1 post wonders), come forward and tell us if Debian is also a problem or not?

Posted by FastServ, 02-20-2013, 09:33 PM
Based on the recent findings as well as evidence in this thread... If it's a linux server and connected to the internet, it's vulnerable. Stop asking if such and such distro is safe or not. Scan your PCs and monitor your network for malicious UDP payload, or hire someone qualified to do it for you.

Posted by TheVisitors, 02-20-2013, 09:38 PM
That argument can be applied to anything connect to the Internet. Because nothing is 100% guaranteed hack / crack proof. If there is a will there is always away. My question is not toward "can" something be hacked / cracked.... That would an illogical argument because the answer is yes. Everything can be. My question was at this time does this single issue currently affect Debian? My findings so far would suggest at least for the moment, no. But I would like to know if anyone else (one of WHT more experienced and well know users) could confirm or deny if this single issue at this moment affect Debian. I believe it is a valid (even if you do not).

Posted by matbz, 02-20-2013, 09:49 PM
If I may... I don't IMHO think that's entirely true, you are drawing a distinction that is not yet qualified. There is evidence within this thread that a root-kit on an office PC is the root of the issue (a key-logger). The targeting of Linux boxes with the data captured from such a key-logger is not proof that Linux is vulnerable, but a choice of the hacker, surely?

Posted by FastServ, 02-20-2013, 10:33 PM
I should have omitted the Linux part -- it was said in context of the post I was replying to -- e.g. this distro is immune, this is not (in fact, windows logins, ISP login details, or even bank details could also be at risk).

Posted by vpswing, 02-20-2013, 10:42 PM
Ramnet, thanks for posting this! Steven & Nenelod - thanks for all the hard work you've put into investigating this! May I ask whether either of you have tested which antivirus/malware scanner that is able to detect this keylogger? That will be very helpful for all those managing servers to advise their clients to do a thorough scan of their PCs/laptops with scanners that can actually detect this rogue keylogger.

Posted by matbz, 02-20-2013, 10:45 PM
Indeed, the only really safe place from anything is in your room with everything switched off, but even that is relative

Posted by Steven, 02-20-2013, 10:52 PM
I don't actually have access to it yet. However nenolod has shared that malwarebytes picked it up.

Posted by bdx33, 02-20-2013, 10:54 PM
I have several antivirus installed (on different machines) all are up to date. If someone sends me the malware I can scan it for you.

Posted by weredigital, 02-20-2013, 10:55 PM
Im currently doing third scan on the same box . This is the only box I could have been infected on in my option. I found nothing with spybot nothing with MS Malware scanner now running malware bytes will advice soon as it finishes. Only thing I have confirmed thus far is I do have over 3 million files on my box

Posted by matbz, 02-20-2013, 10:59 PM
It's been previously stated that 'Malwarebytes Anti-Malware' picked it up. If you have an infected box that you have accessed from the scanned PC, could you please post the scan output from that PC here so the community can see it? If you have any malware found results of course. Last edited by matbz; 02-20-2013 at 11:02 PM. Reason: appended "If you have any malware found results of course."

Posted by vpswing, 02-20-2013, 10:59 PM
Thanks Steven! Thanks ThreadHo!

Posted by ramnet, 02-20-2013, 11:05 PM
Well, you must remember that the earliest reports of this infection go all the way back to August 2012. That's a huge 7 month window of possibilities. The infection on your server may have been laying dormant for months. You also have to keep in mind that it is possible one of your workstation computers was infected and cleaned up ages ago, and you have since forgotten about it.

Posted by matbz, 02-20-2013, 11:11 PM
Or not even known about it.

Posted by weredigital, 02-20-2013, 11:18 PM
Malwarebytes found nothing MS malware scan nothing Spybot scan nothing on to forth

Posted by weredigital, 02-20-2013, 11:29 PM
I did have java update that installed a tool bar and more when i forgot to un-check that little box. I had to restore after that. Possible I wiped it out then .

Posted by matbz, 02-20-2013, 11:30 PM
But you have an infected box? Was there anyone else who had root ssh access to the infected box?

Posted by james2398, 02-20-2013, 11:34 PM
Steven, Scott, mattbz, and everyone else involved in this. Thank you for your selfless efforts and dedication towards this cause. Mattbz, thankfully, I was able to restore my missing libraries and get my server back online thanks to a very good friend of mine. Just giving you an update on that as well. Thank you all

Posted by weredigital, 02-20-2013, 11:36 PM
Besides me on this desktop only cpanel and the data center which is "suppose" to be terminal. With a lot of the cpanel staff working remote using IP proxy and all i wonder

Posted by cloudhopping, 02-20-2013, 11:38 PM
Thats me - rather than fight trying to figure out the old login - since I have not been in the community for heck at least a year if not more - I just created a new account.

Posted by cloudhopping, 02-20-2013, 11:41 PM
Steve are we 100% sure this is from malware? reason I ask (trying to catch up on literally 70 pages now of this ) We use MAC's Not a PC in our office. So trying to figure out if it is malware how we got hit. One pain on a MAC is that it does not have the best way to do maldetect but is not as easy to infect (yet) {hoping i dont start a war on pc vs. mac - not my intention} Anyhow - Steve sent you an email to follow up as well.

Posted by matbz, 02-20-2013, 11:46 PM
Sorry I should have been more certain in my question... has anyone else accessed the box using user 'root'? Just asking to make it clear the possibility of a rootkit infected local machine (PC) or not.

Posted by weredigital, 02-20-2013, 11:51 PM
Those are the only possibilities me cpanel and data center all three use root at some point.

Posted by cloudhopping, 02-21-2013, 12:02 AM
that is an interesting thought. Only users who have root access are for us 1. Me (on a MAC) 2. two staffers (also on MAC) 3. cPanel 4. IPMI and KVM but we connect to those on a MAC. I am concerned that if this was a MalWare issue HOW DO WE FIND IT ON A MAC. Thus far Eset has not picked it up. Neither has MacScan from Secure Mac Our MACs do not have JAVA and they do not have Flash. (makes browsing some sites a bit more fun that way ) We have one workstation that does have JAVA which we use to connect to a kvm on Proxmox - but that is all it does. does not browse the web just to be safe otherwise.

Posted by Steven, 02-21-2013, 12:12 AM
I want to put emphasis on this. We do not know if its 100% malware, but it is one of the likely suspects because on what we know and the wide variety of servers it affects.. Also, for mac lovers. You are not infallible to malware. And this is exactly what I was talking about when I mentioned back connect a few days ago. New Mac malware opens secure reverse shell http://reviews.cnet.com/8301-13727_7...reverse-shell/ With something like this, it does not matter if you firewall off your server, they can login through your own mac and it looks like YOU logged in. Last edited by Steven; 02-21-2013 at 12:16 AM.

Posted by FastServ, 02-21-2013, 12:15 AM
You've got Java on your Mac if you use an IPMI console from it. There's another post somewhere in this thread mentioning ssh locked down to an office full of macs and server compromised (Debian in fact!).

Posted by FastServ, 02-21-2013, 12:45 AM
FYI, if anyone is running DD-WRT at your home or office you can block the malware's payload by enabling dnsmasq for local DNS, then redirecting remote DNS traffic to the local gateway. http://www.dd-wrt.com/wiki/index.php/OpenDNS You can take it a step further and redirect + log redirects with these extra firewall rules in Administration -> Commands -> Firewall. Use remote syslog for extra credit. Last edited by FastServ; 02-21-2013 at 12:58 AM.

Posted by GoTek-JP, 02-21-2013, 01:51 AM
Noticed the same thing, I re-installed a client's server last Thursday and the library was there this morning. He gave CPanel access to his box a few days back, they were at least 3-4 to work on that issue. The client used a different root pass for this new install so the keylogging must have happened recently.

Posted by SoftDux, 02-21-2013, 02:02 AM
This has been suggested to the script writer a few times but he insists on distributing the script and although I have suggested he look for the correct file on the server, 2 days ago, he still doesn't do it.

Posted by Steven, 02-21-2013, 02:04 AM
I need a server that is currently compromised can be booted into a rescue environment via IPMI. PM me if your interested in getting me something quickly. Need to check something. Last edited by Steven; 02-21-2013 at 02:12 AM.

Posted by whamcp, 02-21-2013, 02:10 AM
Just ignore the script guy. This thread would only be 40 pages now had he not posted his changelogs every now and then. Good to see that we are finally getting somewhere with the investigation. Has anyone plainly contacted cpanel support guys and asked them to scan their workstations?

Posted by SoftDux, 02-21-2013, 02:18 AM
As matter of interest, one of our servers was used to send out spam roughly in August / September 2012 and the script did send the spam through UDP port 53. We cleaned the box, applied some extra security measures and didn't see the spam on that box again. But whatever happened on the box at that time sounds exactly the same as what is happening now, large amounts traffic going out on UDP port 53. The box in question didn't cause any problems again and was actually freshly installed recently so the hacked code could very well have been deleted. Although back then we figured it was due to an insecure web application which we removed and secured. The box runs CentOS 5.{latest at the time} + cPanel but didn't have CloudLinux + CageFS on at that stage. I installed CloudLinux + CageFS shortly after discovering the hack and there weren't any more traces of it afterward.

Posted by RWJKM, 02-21-2013, 02:22 AM
If anyone's interested, I've written a script that checks for the existence of the malicious libkeyutil, and can optionally quarantine and remove it, and scan for any other libraries not know to rpm. It won't make any changes if the lib doesn't exist, and for safety's sake won't run on non-CentOS 5.x / 6.x machines. I've hosted it on gist.github, so feel free to inspect, fork and modify it: https://gist.github.com/RWJMurphy/5001090 Invoked without options, it just reports if it can find a malicious libkeyutil. Invoked with "fix", it will move the malicious libkeyutil to a dated folder in /root/.quarantine, run ldconfig to fix the symlink, and restart sshd. Invoked with "scan", it will check all files in /lib (or /lib64 on x64_64 machines) with rpm, and report on any that are unknown. Any feedback / comments is appreciated.

Posted by Steven, 02-21-2013, 03:01 AM
Anyone have /root/.mysql_history emptied?

Posted by Hellsheep, 02-21-2013, 03:15 AM
Hey Steven, I checked my compromised machine, it has not been emptied on mine, worth noting that my compromised machine never started sending spam either even thought it appears to have been compromised mid December last year. So maybe whatever they wanted to do was never finished on the box. Last edited by Hellsheep; 02-21-2013 at 03:26 AM.

Posted by TheVisitors, 02-21-2013, 03:31 AM
check_cpanel_rpms] Altered RPMs found on server The following RPMs are unneeded on your system and should be uninstalled: dovecot.1.2.17-2.cp1136 ^ Got that in my e-mail. Thought it was "interesting"

Posted by Alexander_T, 02-21-2013, 03:38 AM
Interesting. I wonder, why would they link to functions that log a message to the audit system? Last edited by Alexander_T; 02-21-2013 at 03:42 AM.

Posted by adebenc, 02-21-2013, 03:57 AM
I'm a new member here , I want to share my story , I read the forum post every day in last 4 days. I have more dedicated servers , all servers are hosted at same company , only one got affected , the one that I purchased about 15 days ago got affected by this rootkit (the other ones that are older 2-4 years are not affected.) I purchase the new server with CentOS 6.3 , I didn't install any software , I just configure my nameservers , and I needed an answer from cpanel about custom nameservers that I already configured on the server , i contact cpanel and ask them to check the namservers if they are ok (I give them the root password), after one they they reply me that everything should be ok , they check everything and I can start to redirect traffic tot the server. After couple of days I received the e-mail from Cpanel about the hackcheck script, that was the moment that I realize something is not right. I check on google , at that moment I didn't find anything on google about my problem , couple of days later after I contact cpanel again they give me this forum link and explain me that I was hacked. What I think : I'm 100% sure that my windows desktop "did not stole" the root password of this server, (I control more centos servers from this desktop - only the new one was affected Centos 6.3). I want to mention that I didn't install any software on the server. I think that there is 90% chances that cpanel computers are compromised and they compromise my server without any of their knowledge. there is 10% chances that the compromise came from linux updates. there are 0% chances that my password was stolen by a keyloger from my desktop. (all other of my centos servers are clean) and my desktop is for sure secure. I always connect with Putty over ssh to my servers (I have a very secure password) this is my story , I did clean my server with your instructions, and now seams to be everything ok , however I will for sure not contact the cpanel for a while , at least not give them the new root password This is my share , I hope that can help you guys in some ways. sorry for my English Thank you for your great forum

Posted by nofz, 02-21-2013, 04:29 AM
To prevent logging

Posted by Moowalker, 02-21-2013, 04:54 AM
First I want to remind you that 2 of my VPS's were infected. One was "cleaned" (deleted llibkeyutils.so.1.9, closed SSH password but kept port 22 as SSHD portl) and not re-infected till now. The second one, I kept it infected in case someone wants to investigate further. Also at the infected server, running 'last' Depends on what you mean by empty. Mine is (at the cleaned one) So I guess it is not empty. Last edited by Moowalker; 02-21-2013 at 04:57 AM.

Posted by Alexander_T, 02-21-2013, 05:31 AM
Well, heh, that's what I was wondering. I mean, is it possible to "easily" hook these functions to prevent the kernel from doing FIPS audit logging? Remember, the audit system runs in kernel mode.

Posted by Alexander_T, 02-21-2013, 05:32 AM
Found this: http://www.webhostingtalk.com/showth...ge#post8564042 Last edited by Alexander_T; 02-21-2013 at 05:37 AM.

Posted by Polina136, 02-21-2013, 05:34 AM
Interesting. But for those who blame Cpanel - we have no Cpanel on our servers installed. Also we may say that some of the servers were compromised during the time we had a Windows box for ssh connections. But it was replaced with Linux. During that it was off the network, new passwords, etc. Remote login to it is impossible as it has no external IP. Most of the currently infected servers were setup after that Windows gone. So it can only mean that if it is keylogger it came back on a reinstalled with linux machine and somewhy sniffs only passwords which are copy-pasted. That's why i think keylogger is like a bad luck for some of us and problem is actually somewhere else. By the way - if that is a work of keylogger why does attacker try different logins as well? Like "CUCU" and the words which depend on DC server in (DC's name for example if it's short). Last edited by Polina136; 02-21-2013 at 05:42 AM.

Posted by Flegmaa, 02-21-2013, 05:54 AM
Ah, so im new here and that automatically means im trolling? Im just trying to help, like everyone here. If you search my posts youll see i tried to fill in with some information. Just ignore me if you think im trolling, as im going to do with you. Bye. To get back at the point.. On my first server where i have installed cPanel, etc (i also used their support one time 2 months ago) there is a compromise, but on other server where only port 22 (ssh) is opened there is no compromise so obviously the solution is not to move ssd to a non-standard port. Also, when running Erics script i got some errors -

Posted by Moowalker, 02-21-2013, 05:56 AM
Ok.. Something weird happened at the infected server. llibkeyutils.so.1.9 disappeared , it was there yesterday. And a new file appeared (libkeyutils-1.2.so.2)

Posted by Cristi4n, 02-21-2013, 06:12 AM
I just had to quote this. On this subject, this is most likely a keylogger (or similar) as people keep finding out. speedtest.net has been compromised for some time, we had the java and flash issues also and again, from a few thousand servers I only found 10 compromised. Many servers I checked have the same OS/software installed.

Posted by Alex[nl], 02-21-2013, 06:21 AM
From what I read in an earlier post from someone who had the same file and showed the contents and that shows version 1.1.0 (earlier versions were 1.0.X) .. Maybe it is "just" an upgrade?

Posted by Flegmaa, 02-21-2013, 06:24 AM
Also, i forgot to mention that my compromised machine is on non-standard ssh port (but with some more ports opened), and not compromised machine is on standard port (with only port 22 opened).

Posted by GoTek-JP, 02-21-2013, 06:27 AM
It's not related to the panel but possibly to their support workstations.

Posted by NetworkPanda, 02-21-2013, 06:32 AM
Run and post the output here, maybe we could compare it with the results from the libkeyutils-1.9.so posted in previous pages. Also, where is the libkeyutils.so.1 symlink pointing?

Posted by Moowalker, 02-21-2013, 06:46 AM
Not sure. But I can provide it for further investigation if someone needs it.

Posted by demil, 02-21-2013, 06:50 AM
Check where he send the UDP traffic now when you log in, probably RBN changed their IP.

Posted by Alex[nl], 02-21-2013, 06:52 AM
I believe someone already has : LINK

Posted by Moowalker, 02-21-2013, 06:56 AM
As for the symlinks:

Posted by vpswing, 02-21-2013, 07:31 AM
Would you mind posting the md5sum of that file here? Just wondering if it's the same as the original libkeyutils.so.1.9 file. the command: Thanks!

Posted by deriamis, 02-21-2013, 07:31 AM
Ah, all caught up again. Just for the people for whom the thread summary is not enough and who do not want to read the entire thread, some takeaways from what's been going on so far: A potential cause for the compromise has been discovered that appears to be due to malware installed on the workstation accessing the server from one or multiple sources. The malware appears to act as a keylogger, sending your server login information to a remote IP address over port 53 outbound (and not as a DNS packet).Due to the nature of the compromise, any and all Linux servers can potentially be affected by it. We have had reports of compromises on CentOS and Debian servers running all types of services, with or without control panels, with SSH on both standard and non-standard ports, even if there was a good firewall running.This compromise does not at this time appear to be due to the recent kernel 0day nor any service or library vulnerability on the target box. SANS has already been notified of this issue, and several very knowledgeable people are working on this.It is not enough to simply search for a certain file named libkeyutils-*.so on your server. You actually need to run strings against it and compare it with a known compromise. Some of the hallmarks of the compromise in the strings output include an IP address, a password, and links against both pam_* and audit_*. You may need to deobfuscate the file before running strings against it. A simple Perl one-liner someone (I forget who) suggested is: If a server has been infected, you need to scan your workstation(s) (MalwareBytes is known to find the malware) and clean whatever you find. You may be able to get away with a simple cleanup of your server of the infection, but the safest and recommended route is to reload it because you never know what was done with root on your server.Use security best practices to prevent an infection in case your workstation is compromised again. If you got compromised, you might have a hole to plug up.Make sure you include third parties such as vendors and consultants in your evaluation of potential security holes. They may have a compromised workstation and not even know it.Do not use simple scripts to fix the problem. A script is not a replacement for a functioning human brain. If you don't know what it does and that it will definitely work for your environment (and you usually don't if you didn't write it), then don't run it.There are a few people here who know what they are doing and can generally be trusted to investigate a problem. Even so, if you don't know someone, don't hand your server over to them. Seriously, it's like what your mother told you about taking candy from strangers.Even though we finally seem to have some answers (thanks to everyone who worked so hard on this!), we still do not have all the answers yet. Don't let your fancies take you and wait for the real answers when they come in. I hope that helps someone who is looking for information. Since I am such a n00b here, it's about all I can provide.

Posted by Moowalker, 02-21-2013, 07:34 AM
Here it is:

Posted by patchwork, 02-21-2013, 07:34 AM
I have malwarebytes pro on my local pc and so far its found nothing, I also have eset security and that also found nothing. The problem I have with the keylogger theory is. 1. Why has no file names been posted if its been found. 2. Why did the hacker use 3 limited commands to setup the file on our servers if they already had the root password, they could have used any commands they wanted.

Posted by deriamis, 02-21-2013, 07:44 AM
Actually, it has. Why does a hacker do anything? Some people had their servers start spamming, and others did not. If I were a hacker, I would want a pool of spare boxes I could use that people had not found yet.

Posted by patchwork, 02-21-2013, 07:52 AM
That post does not mention any windows file names

Posted by xanubi, 02-21-2013, 07:56 AM
I know i'm new here, and that put me the label "strange guy, beware". But that's not the case. Until now, we're not infected, but i'm seeing lots and lots of connections trying to enter ports 22 and 23 TCP, ports 161 and 800 on UDP. They come from many many places in the world, it's indeed an attack. Fortunately, that ports are not available with us. I'm sharing this, because no one talked about massive port scanning on UDP 161 (normally snmp) or UDP 800 (normally used by proxy squid). Examples (i omited our server ip's of course, with XXX.XXX.XXX): I hope this can contributed to something. Last edited by xanubi; 02-21-2013 at 07:59 AM. Reason: english errors, not my native language :)

Posted by deriamis, 02-21-2013, 07:57 AM
No, it does not. I didn't mean to imply that it did. I was just pointing out where we finally had confirmation of a keylogger. A bit farther down you can see that MWB has worked for some users already. It's going to be a while before we have filenames and reliable detection methods because there are likely multiple methods of infection given the wide range of known targets. SANS has been notified already, and we will know more when we know more.

Posted by deriamis, 02-21-2013, 08:01 AM
Not specifically, no, but it's been mentioned that port scans and brute forcing is not likely to be part of this certain attack. That's fairly standard for the Internets these days, unfortunately. I get scans and brute-force attempts all the time even on my little low-traffic server.

Posted by eva2000, 02-21-2013, 08:07 AM
ouch nasty stuff

Posted by xanubi, 02-21-2013, 08:15 AM
Yes, i agree, every day we've plenty of that. But i only shared this, because we saw an increase and that attempts, and port 161, was very very rare, and it's not now. And port 800, never saw brute-force attacks, until now. One possibility was brute-force attacks in the past to some of the infected systems, they got the pass, and waited some time to attack. I've seen this with email accounts (they infected desktop pc's, got plenty of email accounts, and after 2-3 months they use it to send spam on all accounts at the same time). Thanks for replying.

Posted by deriamis, 02-21-2013, 08:30 AM
It's likely someone who put hands on a copy of Nessus or metasploit and decided to play in your IP block. Or someone has been reading this thread and thought that one hack deserves another. I'm not going to tell you not to worry, but so far as I have seen here those ports don't match anything concerning the current attack. Simply beating on ports on a server doesn't help you much if you're a hacker, contrary to popular belief. They're probably looking for things with known vulnerabilities on your network. I'll put it this way: most hackers follow a method because they either target a certain weakness (in this case, the workstation you log in from) or don't know how to do it differently. While trying to compromise a service may result in the same end-game of installing a new libkeyutils-*.so somewhere, that's really just a backdoor used once the attacker already has access. The real puzzler here was how access was being gained in the first place, and we're pretty sure by now it's a keylogger. Then again, I am not the professional security admin here, and I stand ready to be corrected. I'm just here to watch, learn, and (if I may be so bold) keep a running tab for people coming in on the umpteenth page of a really long thread.

Posted by matbz, 02-21-2013, 08:38 AM
CSF has been updated version 5.79 to check for the newly identified exploit Last edited by matbz; 02-21-2013 at 08:39 AM. Reason: highlighted relevant text

Posted by The3bl, 02-21-2013, 08:39 AM
That is the infected file he just changed the name of it. The strings output shows it is the infected file.

Posted by xanubi, 02-21-2013, 08:54 AM
I agree with everything you say, but it seems very strange all this situation, at least to me. The keylogger for instance, wouldn't most admins, use clean very protected desktop computers? For example, for access, we use an internet connection just for that. It's completed firewalled (which i supose it's normal for must of us), and we only use two computers to access the servers. Isn't this the pratice of all sysadmins ? And if yes, how can a keylogger be installed? It's strange. I'm more on the opinion, that same kind of global service, was hacked, keylogger or other, and someone had access to many pass's. Remember the case of WHCMS, thousands of servers with root access leaked, and user passwords leaked. How many of that root accesses were changed? I don't believe everyone changed that (i also don't believe how can anyone put the root access on a script like that, the maximum, they should use a reseller). I'm trying to thing a little out of the box, since there are too many servers hacked, it must be something common too them all, a service where passwords are stored, a support, a keylogger, and in the end, a collection of passwords got by the hackers over time, and now mass used. Let's see what the big guys can investigate. I supose that besides SANS, others should investigate like Cpanel, RedHat, Cloudlinux. I would like to give my big thank you to all contributions to this thread, and the work of many are giving for free. Without this thread, i would never now about this.

Posted by NetworkPanda, 02-21-2013, 09:09 AM
I have some good news, AVG for Linux has been updated and detects the infected libkeyutils* files, no matter what their file name is. Although we had not been infected, I downloaded the infected files (posted here, some pages before) to a test Linux server just to perform various tests. So, till yesterday morning none of the Linux antivirus tools was detecting it. Today while performing a scan with AVG for Linux (with updated virus definitions) I got this: The infection is detected no matter of its file name. I tried with libtest.so libkeyutils.so.1.2.so.2 etc and it was always detected as infected. So, it is a good idea installing AVG For Linux (no need to get the paid version for just performing scans and detecting infections) and scan in /lib /lib64 and then on your entire system. It will not break things, as it does not heal or quarantine files unless you tell it to do so. It is good just for finding the infection even if the infected files have new file names. ps.: ClamAV, CXS, maldet have no idea yet, they find the files clean... Currently only AVG detects the infection. Last edited by NetworkPanda; 02-21-2013 at 09:21 AM.

Posted by serve-you, 02-21-2013, 09:17 AM
You make a lot of assumptions about "admins". Remember that with the advent of server control panels, all one needs to be an "admin" is a browser. There are a lot of people out there managing servers that quite frankly have no business doing so. I wouldn't suspect that those people manage their desktops any better. Further, you have people like software support and even DC NOC techs who have access to the servers. Don't fool yourself into believing that these people all have secured their desktops/networks. Just look at the recent Apple & Facebook hacks. Anyhow back on point. We know that cPanel & Cloudlinux are both looking into this, as well as grsec and many other people up the chain. Hopefully the collective group will come together with some good analysis.

Posted by Alexander_T, 02-21-2013, 09:19 AM
Not only. This test was done ~ 5 hours ago: https://www.virustotal.com/en/file/a...51f3/analysis/

Posted by serve-you, 02-21-2013, 09:19 AM
Interesting. I wonder why CXS hasn't been updated since ConfgServer has already added checking in CSF a few days ago.

Posted by matbz, 02-21-2013, 09:25 AM
CSX is concerned with scanning user directories (i.e. in /home)...

Posted by NetworkPanda, 02-21-2013, 09:27 AM
Interesting, lets see how long will it take for the other antivirus scanners to start detecting it.

Posted by serve-you, 02-21-2013, 09:28 AM
While I've never used it, their description always made me think it was much more.

Posted by NetworkPanda, 02-21-2013, 09:30 AM
So, if a malicious user attempts to infect a system by uploading the infected libkeyutils file via FTP, SFTP or PHP, it will not be detected by CXS, although its job is to do so.

Posted by xanubi, 02-21-2013, 09:34 AM
That was my point. It could be a collection of passwords got in time, it could be a service with a leak (like a support person), it could be a bad protected desktop, it could be a browser with a plugin that is leaked. It could be all this methods together, and in the final, they've a colletion of passwords. 1) On any of this cases, all point to the same thing, lack of security, and password collection over time, and no one will find a vector of attack. 2) But if it's an OS or service problem, the "big guys", like cpanel, redhat, cloudlinux (etc.), will find the leak. At this time, no one has sure, if it's option 1 or 2, and that's what take our sleep away. Since, this is not a contribution to the resolution of the problem (unfortunately), let's hope someone can achieve a final conclusion soon. serve-you, thank you for all your replies and considerations.

Posted by Zadmin, 02-21-2013, 09:34 AM
has anyone checked his /usr/sbin/sshd & /usr/bin/ssh if they were altered or not ?

Posted by Alex[nl], 02-21-2013, 09:35 AM
Don't normal users usually only have that kind of access somewhere within /home/* ?

Posted by Steven, 02-21-2013, 09:37 AM
Checked two servers last night via sysrescuecd. Nada

Posted by matbz, 02-21-2013, 09:39 AM
That's a question that only Chirpy can answer NP

Posted by jestep, 02-21-2013, 09:45 AM
Depends on where / what they're uploading to. CXS can be configured quite a bit. If they're FTP'ing as root and messing with files outside of /home, which is a pretty bad idea, it probably isn't going to catch it. If this were the case any OS upgrades or user action would cause CXS to blow up with messages. It's basically to alert/protect other accounts from a malicious user trying to hack his way out of his home directory, or install some spammy or malicious software. I can only assume that most admins have their user directories locked down so that a cpanel user, with ssh access or not, cannot traverse into other directories or escalate their privileges. In this case it would be very difficult to configure the malicious libkeyutils file where it is and what it's doing. There has to either be a vulnerability or some method that the root or other privileged account is getting compromised.

Posted by Speckz, 02-21-2013, 10:07 AM
I had 2 tickets with cPanel support. One on Jan. 29 and one on Feb. 8. That server seems fine as I can not find libkeyutils.so.1.9

Posted by iseletsk, 02-21-2013, 10:14 AM
In case someone needs check/clean up scripts -- we have updated ours to include new name: http://www.cloudlinux.com/blog/clnews/sshd-exploit.php

Posted by unSpawn, 02-21-2013, 10:17 AM
Heh, after Steven shared his nfo I updated RKH in CVS and posted a simple ClamAV sig way back on the 16th. By now the samples submitted to SANS ISC are being analyzed (should see a SANS Diary entry in a while I hope) and have been shared with other incident handlers and AV companies.

Posted by IH-Chris, 02-21-2013, 10:28 AM

Posted by Zadmin, 02-21-2013, 11:00 AM
After really some helpful data from flopunctro I was able to gather some info that added up to what I have already here is what I got so far : 1-we are facing an advanced version of this exploit/patch : Where the username/password is send hashes via UDP port 53 instantly (ONLY in case of password login not via pub/priv key) UDP Packets sent look like this : Where XX.XX.XX.XX.XX is the connecting IP * That was root login btw if it may help anyone decoding the hash 2- This were inserted in openssh package includes.h which matches the strings pattern that was captured on they files found though they are not added in plain text rather obfuscated using xor like this : here is the xor routine : 3- This is not a privilege escalation exploit rather a "Maintaining access" exploit 4- Ips found associated with this so far if you want block/report them are : 5- the best route if you got this files reported is OS reload and force password changes afterwards as most if not all services login credentials are affected by this 6- it is suspected *till now* that super_password variable is used to spawn a reverse shell to attacker IP if he telnet to ssh port and pastes the super password though this has not been confirmed yet , hope somone can confirm/deny this . * While in process of fixing you can prevent ongoing password leaks by prohibiting access to outgoing port 53 for only a predefined IP lets say google ns Hope this helps anyone working on this .

Posted by unSpawn, 02-21-2013, 11:13 AM
Exactly, it tell us only the perp already acquired root privileges and not the entry vector some are hunting for. Doesn't detract from the fact that's a nice analysis though.

Posted by Steven, 02-21-2013, 11:14 AM
From what I seen last night They download NEW source, patch it, and then compile it on server. They also compile this: http://packetstormsecurity.com/files.../touch2.c.html to atomically change the inode of sshd --- fwiw, I checked two servers last night out of the OS and can't find the modified sshd binaries. It may be a new tatic, or even an old one since people have said this has existed since at least august last year.

Posted by Zadmin, 02-21-2013, 11:20 AM
Allow me to add I got a report from March 2011

Posted by cloudhopping, 02-21-2013, 11:45 AM
Right - only one system as noted. We are running scans on all those again however

Posted by TheVisitors, 02-21-2013, 11:45 AM
I doubt that is the cause..... I'm running off a LIVE CD with no physical hard drive. I basically have a fresh system every time I reboot.

Posted by cloudhopping, 02-21-2013, 11:54 AM
Learned something from that post. Interesting that we never did that before - but starting today for sure.

Posted by serve-you, 02-21-2013, 12:02 PM
Correct me if I'm wrong here, but I don't think this would protect you from this kind of attack. If you are still browsing the web and get fed bad data, it can still "infect" you. So at the very least a keylogger could be present for the duration of a session. All it would take is for you to type a password once. On reboot, the keylogger may be gone, but they already have your password.

Posted by TheVisitors, 02-21-2013, 12:08 PM
This system has 1 job... Connect to my site. I use it for no other reason. I don't surf Google or go anywhere else. I boot, I go directly to my site, and shut down. Would be interested to learn how else you think a LIVE CD got compromised going no where?

Posted by serve-you, 02-21-2013, 12:09 PM
Thanks for clarifying that. From the prior post it sounded like you used it as a desktop.

Posted by Ceetoe, 02-21-2013, 12:10 PM
I like your idea of a Live CD. Early you mentioned using passwords that are incredibly long(http://www.webhostingtalk.com/showpo...&postcount=323). Are you manually typing this in each time you use the Live CD or you have persistent storage that is accessible? Steven, Can you update if you are now in possession of the purported local malware?

Posted by TheVisitors, 02-21-2013, 12:14 PM
1 single usb drive with only 1 file on it. Open file, right click, copy, paste. No need to type a single thing. The usb drive is only used on this system.

Posted by Steven, 02-21-2013, 12:20 PM
Are you browsing this site on it?

Posted by Ceetoe, 02-21-2013, 12:25 PM
Thanks. I have to say you have one of the more interesting setups I've heard of. I appreciate what you have shared. I am curious about how you are building your Windows Live CD. You mentioned you only use Filezilla and Putty(http://www.webhostingtalk.com/showpo...&postcount=925). How often are you building/updating/patching your LIVE CD and then the question is how clean was the build system? Ridiculous questions but even more ridiculous not to ask.

Posted by matbz, 02-21-2013, 12:27 PM
I believe he meant the looong complex password Steven.

Posted by TheVisitors, 02-21-2013, 12:28 PM
I am browsing this site on a different computer. I use my old computer to boot into LIVE CD and connect to my site. I use it only for accessing my site to void such issues. It's basically stripped down. Using no hard drive, no sound card, on board video (VGA), a dvd drive, an old network card, and 2 sticks of ram (4gb).

Posted by Steven, 02-21-2013, 12:29 PM
Im going to say this again: http://www.baekdal.com/insights/pass...rity-usability

Posted by tecnobrat, 02-21-2013, 12:30 PM
And you never sent your root password to any third party support provider? Your datacenter does not have your root password?

Posted by TheVisitors, 02-21-2013, 12:35 PM
Only after the fact.. ie... After being compromised cPanel Steve In that order.

Posted by tomfrog, 02-21-2013, 12:46 PM
If the tool to maintain root access is published online and has been around since March 2011, can we exclude more than 1 entry vector? Can we even limit this to 1 group of hackers?

Posted by TheVisitors, 02-21-2013, 12:48 PM
LIVE CD is not Microsoft Windows LIVE CD is Linux

Posted by tomfrog, 02-21-2013, 12:50 PM
I bookmarked that blog. I loved the writing. Very clear. The ideas are also great. Thank you!

Posted by Ceetoe, 02-21-2013, 12:53 PM
So why use Putty or Filezilla on Linux?

Posted by iseletsk, 02-21-2013, 12:54 PM
If you are running R1Soft, please review your past backups (for the past 3-4 months), and see if you can find /home/tmpp backed up. We are highly interested in content of that directory (but it is very short lived on infected machines). if you find it -- please, restore it/compress it, and post it (or send it to me or Steven)

Posted by TheVisitors, 02-21-2013, 12:55 PM
Personal preference.

Posted by WhoElse, 02-21-2013, 01:14 PM
Does this update have anything to do with this issue: http://secunia.com/advisories/52312/ ?

Posted by Steven, 02-21-2013, 01:18 PM
Maybe. Do not see advasary for version 5.

Posted by demil, 02-21-2013, 01:21 PM
Makes sense yes, and probably the hacker got into local user with some open-source wordpress/whatever bugs. However it does not make sense why he goes in as root to put libkey backdoored and few other things... (after few seconds of thinking, probably is making sense... probably some of the files he wants to change are in use by the exploit, so he wants to cleanup and backdoor the syscalls...) But then, why none of my servers were infected, when the attacker had even SSH access as normal user... Last edited by demil; 02-21-2013 at 01:29 PM.

Posted by tecnobrat, 02-21-2013, 01:22 PM
Thats quite an old CVE .. 20121024

Posted by sprintserve, 02-21-2013, 01:35 PM
None of our over a hundred servers appears to be infected. So it is unlikely to be an exploit with the SSH daemon. I will definitely vote for a likely local exploit (key loggers, etc) that has been used / targeted specifically for stealing passwords to SSH.

Posted by serve-you, 02-21-2013, 01:37 PM
I'm still really interested to know why there are zero reports of this (the libkeyutils file specifically) outside of the hosting community. Nothing so far has shown this to be specific to hosting. As of a few days ago, a search on "libkeyutils.so.1.9" basically gave you this thread only. Now it gives a bunch of others basically regurgitating this thread. You'd think that all the linux admin forums would be exploding with reports of this kind of activity.

Posted by tecnobrat, 02-21-2013, 01:41 PM
I would put money on the fact that the java exploit (current theory) was actually placed inside a malicious ad that was displayed on a hosting forum or website that hosting admins frequent. Maybe even this site itself. The fact that it affects CentOS/Cpanel/etc machines more than others would also lead me to believe it may be a forum / site directly related to that. I have no proof of this, but ... logic would dictate?

Posted by Patrick, 02-21-2013, 01:43 PM
http://stackoverflow.com/questions/1...-dlopen-errors Presumably the OP in that thread does not run a hosting business... you can see the suspect file listed there. Date: Jan 4

Posted by tomfrog, 02-21-2013, 01:47 PM
Steven started this thread at WHT and everyone started checking their servers for the file. Who are we? Hosting providers. Does that mean that only hosting server have been affected? No. Is just means that hosting providers are aware of this file... Maybe it's not restricted to the hosting community and business.

Posted by Ceetoe, 02-21-2013, 01:48 PM
No problem. Are you the admin on sociallyuncensored.eu? This post mentioned having to manually download and upload all of the files and presumed over a couple of days(http://www.sociallyuncensored.eu/for...finally.16208/). I presume the server was clean(or at least not acting suspiciously) prior to your change of host? Did you use the LiveCD during the transfer/reinstall during all of these operations? Or a different pc? If you used a different pc and then went back to the LiveCD once the the site was live then this might be something to research. If you have removed the server side culprit, changed your passwords, and only used the Live CD during the cleanup and since then...can you confirm your server still appears to be clean? Last edited by Ceetoe; 02-21-2013 at 02:02 PM. Reason: Here be typos...

Posted by serve-you, 02-21-2013, 01:53 PM
Interesting. I didn't see that the other day when I googled for it. I seriously only got like 4 results back. Now it's pages, though a quick glance showed most were here. My point still stands though. Most admins would see the spam (that was originally reported) coming from their servers within minutes. I would expect this to be all over the place if it's not something targeting a specific piece of software or user base. While there certainly a lot of oblivious admins out there, there are hundreds of thousands of publicly facing centos/redhat servers out there that aren't related to the hosting industry.

Posted by tomfrog, 02-21-2013, 01:56 PM
serve-you I would bet that only a very small amount of infected servers started sending out spam. This thread has more than 100 000 page views. Ancient members created new accounts to participate... Hosting providers are now aware of this hack. That doesn't mean that only hosting servers were targeted...

Posted by jalapeno55, 02-21-2013, 02:24 PM
Very likely, Apple was hacked after an Iphone Dev forum had malware ads. http://iphonedevsdk.com/forum/site-n...g-with-it.html

Posted by deriamis, 02-21-2013, 02:42 PM
Except for the people who log in directly as root on their servers?

Posted by oxygene, 02-21-2013, 03:14 PM
Hello, due to some logs entries I am able to say that 1 of our servers have been infected with the libkeyutils.so.1.9 since October, 31st 2012. Here a piece of the log.. Maybe it is useful for tracing.. Regards Eike Time: Wed Oct 31 09:20:39 2012 +0100 PID: 29711 Account: Uptime: 579 seconds Executable: /usr/bin/php Command Line (often faked in exploits): /usr/bin/php Network connections by the process (if any): tcp: x.x.x.x:35471 -> x.x.x.x:80 Files open by the process (if any): /usr/local/apache/logs/error_log /usr/local/apache/logs/error_log Memory maps by the process (if any): 00400000-00a66000 r-xp 00000000 fd:01 23438531 /usr/bin/php 00c66000-00ccd000 rw-p 00666000 fd:01 23438531 /usr/bin/php 00ccd000-00cdb000 rw-p 00ccd000 00:00 0 0fc8f000-11c69000 rw-p 0fc8f000 00:00 0 [heap] 3772e00000-3772fd1000 r-xp 00000000 fd:01 23431119 /usr/lib64/libmysqlclient.so.16.0.0 3772fd1000-37731d0000 ---p 001d1000 fd:01 23431119 /usr/lib64/libmysqlclient.so.16.0.0 37731d0000-3773222000 rw-p 001d0000 fd:01 23431119 /usr/lib64/libmysqlclient.so.16.0.0 3773222000-3773223000 rw-p 3773222000 00:00 0 3c24e00000-3c24e06000 r-xp 00000000 fd:01 26181648 /lib64/libkeyutils.so.1.9 3c24e06000-3c25006000 ---p 00006000 fd:01 26181648 /lib64llibkeyutils.so.1.9 3c25006000-3c25007000 rw-p 00006000 fd:01 26181648 /lib64/libkeyutils.so.1.9 3c25200000-3c25224000 r-xp 00000000 fd:01 23430740 /usr/lib64/libk5crypto.so.3.1 3c25224000-3c25423000 ---p 00024000 fd:01 23430740 /usr/lib64/libk5crypto.so.3.1 3c25423000-3c25425000 rw-p 00023000 fd:01 23430740 /usr/lib64/libk5crypto.so.3.1 3c25600000-3c25608000 r-xp 00000000 fd:01 23430641 /usr/lib64/libkrb5support.so.0.1 3c25608000-3c25807000 ---p 00008000 fd:01 23430641 /usr/lib64/libkrb5support.so.0.1 3c25807000-3c25808000 rw-p 00007000 fd:01 23430641 /usr/lib64/libkrb5support.so.0.1 3c25a00000-3c25a2c000 r-xp 00000000 fd:01 23430753 /usr/lib64/libgssapi_krb5.so.2.2 3c25a2c000-3c25c2c000 ---p 0002c000 fd:01 23430753 /usr/lib64/libgssapi_krb5.so.2.2 3c25c2c000-3c25c2e000 rw-p 0002c000 fd:01 23430753 /usr/lib64/libgssapi_krb5.so.2.2 3c25e00000-3c25e91000 r-xp 00000000 fd:01 23430742 /usr/lib64/libkrb5.so.3.3 3c25e91000-3c26091000 ---p 00091000 fd:01 23430742 /usr/lib64/libkrb5.so.3.3 3c26091000-3c26095000 rw-p 00091000 fd:01 23430742 /usr/lib64/libkrb5.so.3.3 3c26200000-3c26246000 r-xp 00000000 fd:01 26181662 /lib64/libssl.so.0.9.8e 3c26246000-3c26446000 ---p 00046000 fd:01 26181662 /lib64/libssl.so.0.9.8e 3c26446000-3c2644c000 rw-p 00046000 fd:01 26181662 /lib64/libssl.so.0.9.8e 3f97600000-3f9761c000 r-xp 00000000 fd:01 26181661 /lib64/ld-2.5.so 3f9781c000-3f9781d000 r--p 0001c000 fd:01 26181661 /lib64/ld-2.5.so 3f9781d000-3f9781e000 rw-p 0001d000 fd:01 26181661 /lib64/ld-2.5.so 3f97a00000-3f97b4e000 r-xp 00000000 fd:01 26181688 /lib64/libc-2.5.so 3f97b4e000-3f97d4d000 ---p 0014e000 fd:01 26181688 /lib64/libc-2.5.so 3f97d4d000-3f97d51000 r--p 0014d000 fd:01 26181688 /lib64/libc-2.5.so 3f97d51000-3f97d52000 rw-p 00151000 fd:01 26181688 /lib64/libc-2.5.so 3f97d52000-3f97d57000 rw-p 3f97d52000 00:00 0 3f97e00000-3f97e02000 r-xp 00000000 fd:01 26181690 /lib64/libdl-2.5.so 3f97e02000-3f98002000 ---p 00002000 fd:01 26181690 /lib64/libdl-2.5.so 3f98002000-3f98003000 r--p 00002000 fd:01 26181690 /lib64/libdl-2.5.so 3f98003000-3f98004000 rw-p 00003000 fd:01 26181690 /lib64/libdl-2.5.so 3f98200000-3f98216000 r-xp 00000000 fd:01 26181841 /lib64/libpthread-2.5.so 3f98216000-3f98415000 ---p 00016000 fd:01 26181841 /lib64/libpthread-2.5.so 3f98415000-3f98416000 r--p 00015000 fd:01 26181841 /lib64/libpthread-2.5.so 3f98416000-3f98417000 rw-p 00016000 fd:01 26181841 /lib64/libpthread-2.5.so 3f98417000-3f9841b000 rw-p 3f98417000 00:00 0 3f98600000-3f98682000 r-xp 00000000 fd:01 26181931 /lib64/libm-2.5.so 3f98682000-3f98881000 ---p 00082000 fd:01 26181931 /lib64/libm-2.5.so 3f98881000-3f98882000 r--p 00081000 fd:01 26181931 /lib64/libm-2.5.so 3f98882000-3f98883000 rw-p 00082000 fd:01 26181931 /lib64/libm-2.5.so 3f98a00000-3f98a14000 r-xp 00000000 fd:01 26181930 /lib64/libz.so.1.2.3 3f98a14000-3f98c13000 ---p 00014000 fd:01 26181930 /lib64/libz.so.1.2.3 3f98c13000-3f98c14000 rw-p 00013000 fd:01 26181930 /lib64/libz.so.1.2.3 3f98e00000-3f98e07000 r-xp 00000000 fd:01 26181850 /lib64/librt-2.5.so 3f98e07000-3f99007000 ---p 00007000 fd:01 26181850 /lib64/librt-2.5.so 3f99007000-3f99008000 r--p 00007000 fd:01 26181850 /lib64/librt-2.5.so 3f99008000-3f99009000 rw-p 00008000 fd:01 26181850 /lib64/librt-2.5.so 3f99200000-3f99215000 r-xp 00000000 fd:01 26182001 /lib64/libselinux.so.1 3f99215000-3f99415000 ---p 00015000 fd:01 26182001 /lib64/libselinux.so.1 3f99415000-3f99417000 rw-p 00015000 fd:01 26182001 /lib64/libselinux.so.1 3f99417000-3f99418000 rw-p 3f99417000 00:00 0 3f99600000-3f9963b000 r-xp 00000000 fd:01 26182000 /lib64/libsepol.so.1 3f9963b000-3f9983b000 ---p 0003b000 fd:01 26182000 /lib64/libsepol.so.1 3f9983b000-3f9983c000 rw-p 0003b000 fd:01 26182000 /lib64/libsepol.so.1 3f9983c000-3f99846000 rw-p 3f9983c000 00:00 0 3f99a00000-3f99a15000 r-xp 00000000 fd:01 26181996 /lib64/libnsl-2.5.so 3f99a15000-3f99c14000 ---p 00015000 fd:01 26181996 /lib64/libnsl-2.5.so 3f99c14000-3f99c15000 r--p 00014000 fd:01 26181996 /lib64/libnsl-2.5.so 3f99c15000-3f99c16000 rw-p 00015000 fd:01 26181996 /lib64/libnsl-2.5.so 3f99c16000-3f99c18000 rw-p 3f99c16000 00:00 0 3f99e00000-3f99e0d000 r-xp 00000000 fd:01 26182012 /lib64/libgcc_s-4.1.2-20080825.so.1 3f99e0d000-3f9a00d000 ---p 0000d000 fd:01 26182012 /lib64/libgcc_s-4.1.2-20080825.so.1 3f9a00d000-3f9a00e000 rw-p 0000d000 fd:01 26182012 /lib64/libgcc_s-4.1.2-20080825.so.1 3f9a200000-3f9a2e6000 r-xp 00000000 fd:01 23430838 /usr/lib64/libstdc++.so.6.0.8 3f9a2e6000-3f9a4e5000 ---p 000e6000 fd:01 23430838 /usr/lib64/libstdc++.so.6.0.8 3f9a4e5000-3f9a4eb000 r--p 000e5000 fd:01 23430838 /usr/lib64/libstdc++.so.6.0.8 3f9a4eb000-3f9a4ee000 rw-p 000eb000 fd:01 23430838 /usr/lib64/libstdc++.so.6.0.8 3f9a4ee000-3f9a500000 rw-p 3f9a4ee000 00:00 0 3f9a600000-3f9a609000 r-xp 00000000 fd:01 26181994 /lib64/libcrypt-2.5.so 3f9a609000-3f9a808000 ---p 00009000 fd:01 26181994 /lib64/libcrypt-2.5.so 3f9a808000-3f9a809000 r--p 00008000 fd:01 26181994 /lib64/libcrypt-2.5.so 3f9a809000-3f9a80a000 rw-p 00009000 fd:01 26181994 /lib64/libcrypt-2.5.so 3f9a80a000-3f9a838000 rw-p 3f9a80a000 00:00 0 3f9aa00000-3f9aa17000 r-xp 00000000 fd:01 26181691 /lib64/libaudit.so.0.0.0 3f9aa17000-3f9ac16000 ---p 00017000 fd:01 26181691 /lib64/libaudit.so.0.0.0 3f9ac16000-3f9ac18000 rw-p 00016000 fd:01 26181691 /lib64/libaudit.so.0.0.0 3f9ae00000-3f9ae7f000 r-xp 00000000 fd:01 23442294 /usr/lib64/libfreetype.so.6.3.10 3f9ae7f000-3f9b07f000 ---p 0007f000 fd:01 23442294 /usr/lib64/libfreetype.so.6.3.10 3f9b07f000-3f9b084000 rw-p 0007f000 fd:01 23442294 /usr/lib64/libfreetype.so.6.3.10 3f9b200000-3f9b221000 r-xp 00000000 fd:01 23442723 /usr/lib64/libjpeg.so.62.0.0 3f9b221000-3f9b420000 ---p 00021000 fd:01 23442723 /usr/lib64/libjpeg.so.62.0.0 3f9b420000-3f9b421000 rw-p 00020000 fd:01 23442723 /usr/lib64/libjpeg.so.62.0.0 3f9b600000-3f9b631000 r-xp 00000000 fd:01 23441269 /usr/lib64/libidn.so.11.5.19 3f9b631000-3f9b830000 ---p 00031000 fd:01 23441269 /usr/lib64/libidn.so.11.5.19 3f9b830000-3f9b831000 rw-p 00030000 fd:01 23441269 /usr/lib64/libidn.so.11.5.19 3f9ba00000-3f9bb05000 r-xp 00000000 fd:01 23449204 /usr/lib64/libX11.so.6.2.0 3f9bb05000-3f9bd05000 ---p 00105000 fd:01 23449204 /usr/lib64/libX11.so.6.2.0 3f9bd05000-3f9bd0c000 rw-p 00105000 fd:01 23449204 /usr/lib64/libX11.so.6.2.0 3f9be00000-3f9bf2d000 r-xp 00000000 fd:01 26182007 /lib64/libcrypto.so.0.9.8e 3f9bf2d000-3f9c12c000 ---p 0012d000 fd:01 26182007 /lib64/libcrypto.so.0.9.8e 3f9c12c000-3f9c14d000 rw-p 0012c000 fd:01 26182007 /lib64/libcrypto.so.0.9.8e 3f9c14d000-3f9c151000 rw-p 3f9c14d000 00:00 0 3f9c200000-3f9c210000 r-xp 00000000 fd:01 23441981 /usr/lib64/libXpm.so.4.11.0 3f9c210000-3f9c410000 ---p 00010000 fd:01 23441981 /usr/lib64/libXpm.so.4.11.0 3f9c410000-3f9c411000 rw-p 00010000 fd:01 23441981 /usr/lib64/libXpm.so.4.11.0 3f9c600000-3f9c611000 r-xp 00000000 fd:01 26181998 /lib64/libresolv-2.5.so 3f9c611000-3f9c811000 ---p 00011000 fd:01 26181998 /lib64/libresolv-2.5.so 3f9c811000-3f9c812000 r--p 00011000 fd:01 26181998 /lib64/libresolv-2.5.so 3f9c812000-3f9c813000 rw-p 00012000 fd:01 26181998 /lib64/libresolv-2.5.so 3f9c813000-3f9c815000 rw-p 3f9c813000 00:00 0 3f9ce00000-3f9ce0b000 r-xp 00000000 fd:01 26181696 /lib64/libpam.so.0.81.5 3f9ce0b000-3f9d00a000 ---p 0000b000 fd:01 26181696 /lib64/libpam.so.0.81.5 3f9d00a000-3f9d00b000 rw-p 0000a000 fd:01 26181696 /lib64/libpam.so.0.81.5 3f9d200000-3f9d202000 r-xp 00000000 fd:01 26182004 /lib64/libcom_err.so.2.1 3f9d202000-3f9d401000 ---p 00002000 fd:01 26182004 /lib64/libcom_err.so.2.1 3f9d401000-3f9d402000 rw-p 00001000 fd:01 26182004 /lib64/libcom_err.so.2.1 3f9d600000-3f9d623000 r-xp 00000000 fd:01 23442297 /usr/lib64/libpng12.so.0.10.0 3f9d623000-3f9d823000 ---p 00023000 fd:01 23442297 /usr/lib64/libpng12.so.0.10.0 3f9d823000-3f9d824000 rw-p 00023000 fd:01 23442297 /usr/lib64/libpng12.so.0.10.0 3f9da00000-3f9da05000 r-xp 00000000 fd:01 23449203 /usr/lib64/libXdmcp.so.6.0.0 3f9da05000-3f9dc04000 ---p 00005000 fd:01 23449203 /usr/lib64/libXdmcp.so.6.0.0 3f9dc04000-3f9dc05000 rw-p 00004000 fd:01 23449203 /usr/lib64/libXdmcp.so.6.0.0 3f9e600000-3f9e602000 r-xp 00000000 fd:01 23449202 /usr/lib64/libXau.so.6.0.0 3f9e602000-3f9e801000 ---p 00002000 fd:01 23449202 /usr/lib64/libXau.so.6.0.0 3f9e801000-3f9e802000 rw-p 00001000 fd:01 23449202 /usr/lib64/libXau.so.6.0.0 3fa1200000-3fa120f000 r-xp 00000000 fd:01 23449199 /usr/lib64/libbz2.so.1.0.3 3fa120f000-3fa140e000 ---p 0000f000 fd:01 23449199 /usr/lib64/libbz2.so.1.0.3 3fa140e000-3fa1410000 rw-p 0000e000 fd:01 23449199 /usr/lib64/libbz2.so.1.0.3 2b39ce99f000-2b39ce9a1000 rw-p 2b39ce99f000 00:00 0 2b39ce9b1000-2b39ce9b5000 rw-p 2b39ce9b1000 00:00 0 2b39ce9b5000-2b39ce9f0000 r-xp 00000000 fd:01 4620292 /opt/pcre/lib/libpcre.so.0.0.1 2b39ce9f0000-2b39cebf0000 ---p 0003b000 fd:01 4620292 /opt/pcre/lib/libpcre.so.0.0.1 2b39cebf0000-2b39cebf1000 rw-p 0003b000 fd:01 4620292 /opt/pcre/lib/libpcre.so.0.0.1 2b39cebf1000-2b39cebf4000 rw-p 2b39cebf1000 00:00 0 2b39cebf4000-2b39cec4a000 r-xp 00000000 fd:01 4620426 /opt/curlssl/lib/libcurl.so.4.2.0 2b39cec4a000-2b39cee49000 ---p 00056000 fd:01 4620426 /opt/curlssl/lib/libcurl.so.4.2.0 2b39cee49000-2b39cee4c000 rw-p 00055000 fd:01 4620426 /opt/curlssl/lib/libcurl.so.4.2.0 2b39cee4c000-2b39cee4d000 rw-p 2b39cee4c000 00:00 0 2b39cee4d000-2b39cef85000 r-xp 00000000 fd:01 4620996 /opt/xml2/lib/libxml2.so.2.7.8 2b39cef85000-2b39cf184000 ---p 00138000 fd:01 4620996 /opt/xml2/lib/libxml2.so.2.7.8 2b39cf184000-2b39cf18e000 rw-p 00137000 fd:01 4620996 /opt/xml2/lib/libxml2.so.2.7.8 2b39cf18e000-2b39cf196000 rw-p 2b39cf18e000 00:00 0 2b39cf1d7000-2b39cf2c9000 r-xp 00000000 fd:01 23725369 /usr/local/IonCube/ioncube_loader_lin_5.2.so 2b39cf2c9000-2b39cf3c8000 ---p 000f2000 fd:01 23725369 /usr/local/IonCube/ioncube_loader_lin_5.2.so 2b39cf3c8000-2b39cf3d7000 rw-p 000f1000 fd:01 23725369 /usr/local/IonCube/ioncube_loader_lin_5.2.so 2b39cf3d7000-2b39cf3da000 rw-p 2b39cf3d7000 00:00 0 2b39cf3da000-2b39cf55e000 r-xp 00000000 fd:01 23725367 /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so 2b39cf55e000-2b39cf65d000 ---p 00184000 fd:01 23725367 /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so 2b39cf65d000-2b39cf683000 rw-p 00183000 fd:01 23725367 /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so 2b39cf683000-2b39cf688000 rw-p 2b39cf683000 00:00 0 2b39cf688000-2b39cf6ce000 r-xp 00000000 fd:01 24349538 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/timezonedb.so 2b39cf6ce000-2b39cf8cd000 ---p 00046000 fd:01 24349538 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/timezonedb.so 2b39cf8cd000-2b39cf8d0000 rw-p 00045000 fd:01 24349538 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/timezonedb.so 2b39cf8d0000-2b39cf8f2000 r-xp 00000000 fd:01 24349270 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/suhosin.so 2b39cf8f2000-2b39cfaf2000 ---p 00022000 fd:01 24349270 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/suhosin.so 2b39cfaf2000-2b39cfaf8000 rw-p 00022000 fd:01 24349270 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/suhosin.so 2b39cfaf8000-2b39cfafa000 rw-p 2b39cfaf8000 00:00 0 2b39cfafa000-2b39cfb10000 r-xp 00000000 fd:01 24349472 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo.so 2b39cfb10000-2b39cfd10000 ---p 00016000 fd:01 24349472 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo.so 2b39cfd10000-2b39cfd13000 rw-p 00016000 fd:01 24349472 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo.so 2b39cfd13000-2b39cfd77000 r-xp 00000000 fd:01 24349536 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo_sqlite.so 2b39cfd77000-2b39cff77000 ---p 00064000 fd:01 24349536 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo_sqlite.so 2b39cff77000-2b39cff7a000 rw-p 00064000 fd:01 24349536 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo_sqlite.so 2b39cff7a000-2b39cffcd000 r-xp 00000000 fd:01 24349537 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/sqlite.so 2b39cffcd000-2b39d01cd000 ---p 00053000 fd:01 24349537 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/sqlite.so 2b39d01cd000-2b39d01d1000 rw-p 00053000 fd:01 24349537 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/sqlite.so 2b39d01d1000-2b39d01d2000 rw-p 2b39d01d1000 00:00 0 2b39d01d2000-2b39d01d8000 r-xp 00000000 fd:01 24349535 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo_mysql.so 2b39d01d8000-2b39d03d8000 ---p 00006000 fd:01 24349535 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo_mysql.so 2b39d03d8000-2b39d03d9000 rw-p 00006000 fd:01 24349535 /usr/local/lib/php/extensions/no-debug-non-zts-20060613/pdo_mysql.so 2b39d03e9000-2b39d03f3000 r-xp 00000000 fd:01 26181827 /lib64/libnss_files-2.5.so 2b39d03f3000-2b39d05f2000 ---p 0000a000 fd:01 26181827 /lib64/libnss_files-2.5.so 2b39d05f2000-2b39d05f3000 r--p 00009000 fd:01 26181827 /lib64/libnss_files-2.5.so 2b39d05f3000-2b39d05f4000 rw-p 0000a000 fd:01 26181827 /lib64/libnss_files-2.5.so 2b39d0604000-2b39d0608000 r-xp 00000000 fd:01 26181823 /lib64/libnss_dns-2.5.so 2b39d0608000-2b39d0807000 ---p 00004000 fd:01 26181823 /lib64/libnss_dns-2.5.so 2b39d0807000-2b39d0808000 r--p 00003000 fd:01 26181823 /lib64/libnss_dns-2.5.so 2b39d0808000-2b39d0809000 rw-p 00004000 fd:01 26181823 /lib64/libnss_dns-2.5.so 2b39d08f7000-2b39d0979000 rw-p 2b39d08f7000 00:00 0 2b39d0979000-2b39d3f4d000 r--p 00000000 fd:01 23449193 /usr/lib/locale/locale-archive 2b39d3f4d000-2b39d3f54000 r--s 00000000 fd:01 23495537 /usr/lib64/gconv/gconv-modules.cache 7fff88f74000-7fff88fe4000 rwxp 7ffffff8e000 00:00 0 [stack] 7fff88ffc000-7fff89000000 r-xp 7fff88ffc000 00:00 0 [vdso] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall]

Posted by cloudhopping, 02-21-2013, 03:28 PM
Cloudlinux - Thank you for the quick script to check our systems with. I have found quite a few that are infected - and many that are not. Just wanted to clarify - this has hit other Distro's not just RH/Cent ? correct ? thusfar have not found it on debian or ubuntu but figured I would ask

Posted by NetworkPanda, 02-21-2013, 03:34 PM
Yes, there have been reports here by Debian and Ubuntu users who had their servers infected. Maybe the reports for Debian/Ubuntu infections are not so many here, because most WHT members use Centos/Redhad/Fedora for web hosting, but Debian and Ubuntu are affected as well.

Posted by tonytz, 02-21-2013, 03:38 PM
based on some of the threads, it appears that the attacker didn't change the sshd and ssh binaries, so I am wondering how it is possible to steal the newly created username/password, which were sent via a dns query? can anyone shed some light on this?

Posted by Steven, 02-21-2013, 03:41 PM
There is a library libkeyutils.so.1.* that is placed on the server and sshd is linked to.

Posted by tonytz, 02-21-2013, 03:46 PM
interesting, I see how it works now, good to know

Posted by shackrock, 02-21-2013, 03:48 PM
Hey all, I see some different entries that are quite similar but I'm not sure if this is another variant or legit: lib/libkeyutils-1.2.so lib64/libkeyutils-1.2.so lib/libkeyutils.so.1 lib64/libkeyutils.so.1 Happy to post any other logs as necessary - just might need some locations/direction as to where they are located.

Posted by shackrock, 02-21-2013, 03:52 PM
Can you list how to install the AVG package? Sorry, help for the newbs like me.

Posted by serve-you, 02-21-2013, 03:56 PM
Those should be the legit libs. Just run 'rpm -qf /path/to/file' and it will show that they are part of the keyutils-libs package.

Posted by weredigital, 02-21-2013, 03:57 PM
Last night I scanned the workstations with every thing I could find and was suggested here. I finally found a Java Trojans with Avast not found by any thing else. I sent Steven the logs of what was found and have the file quarantined. I haven't heard back form Steven yet if this is the file in question or not but fact 4 different scanners never found it I am curious. Last edited by weredigital; 02-21-2013 at 04:00 PM.

Posted by shackrock, 02-21-2013, 04:03 PM
Great while looking in the folder I just did this: http://superuser.com/questions/55545...at-it-was-call Kill me now, any help is appreciated. Sorry if offtopic, please PM me. The good news is that those are legit files for me =) [root@server /]# rpm -qf lib/libkeyutils-1.2.so keyutils-libs-1.2-1.el5 [root@server /]# rpm -qf lib64/libkeyutils-1.2.so keyutils-libs-1.2-1.el5 [root@server /]# rpm -qf lib64/libkeyutils.so.1 keyutils-libs-1.2-1.el5 [root@server /]# rpm -qf lib/libkeyutils.so.1 keyutils-libs-1.2-1.el5

Posted by Steven, 02-21-2013, 04:12 PM
New file name: /lib64/libkeyutils.so.1.3.2

Posted by Majester, 02-21-2013, 04:16 PM
lib64/libkeyutils-1.2.so.2 as well.

Posted by coldbeer, 02-21-2013, 04:18 PM
Steven can you make a list of all infected filenames?

Posted by weredigital, 02-21-2013, 04:22 PM
Just announced on CNN Facebook hacked via Java Exploit. wonder if related to our issues http://www.cnn.com/2013/02/15/tech/s...iref=allsearch apparently old news just came in on my feeds need new feed i guess Last edited by weredigital; 02-21-2013 at 04:29 PM.

Posted by NetworkPanda, 02-21-2013, 04:34 PM
Also (now that the infected files are circulating with various random new file names) if you are concerned if a libkeyutils* file is legitimate or if it is an infection, just install AVG for Linux and scan your entire /lib or /lib64 folder. AVG works fine on Centos/RHEL with or without cPanel. It is safe to have it installed along with ClamAV or other antivirus software, as AVG free for Linux does not run in real-time mode, it performs only scans on demand to find infections. So, no conflicts with the other antivirus tools you may have. Visit http://free.avg.com/us-en/download.prd-alf wget the rpm file to your server then run avg-xxxxxx.rpm = the downloaded file (its name changes with each new version released by AVG) Then run: (the first command avgupdate needs to be ran only once per day, to download the new virus definitions) No matter what is the name of the malicious file, it will be detected, so you will now which files are clean or not. Last edited by NetworkPanda; 02-21-2013 at 04:39 PM.

Posted by jalapeno55, 02-21-2013, 04:38 PM
Which is better AVG or ClamAV?

Posted by NetworkPanda, 02-21-2013, 04:40 PM
According to my experience, AVG is far better and detects new infections faster than Clam. For example, this new infection is not yet detected by ClamAV and the same has also happened with several other malware in the past. But it is safe to have AVG installed along with ClamAV or other antivirus software, as AVG free for Linux does not run in real-time mode, it performs only scans on demand to find infections. So, no conflicts with the other antivirus tools you may have.

Posted by nicknn, 02-21-2013, 04:41 PM
It think you may have right Last Saturday, (Feb 16) i used my liitle doghters old laptop at home to google for mod rewrite. This laptop is used by my daughter to play flash games, so flash ang java is allowed, and NOScript missing (and off course no root passwords:-) I followed a link to a site with the word htaccess in the domain I am 99% sure i can recall the domain, but since i am not 100% i wont post the domain name, and i don't want to visit this site again to confirm.. When the site loaded, the screen has frozen and the disk has started to spin very very fast I am 100% it was an infection I have scanned the laptop with compofix avira and malware bytes, but found nothing I reloaded the OS to be sure. Sounds familiar to anyone?

Posted by Steven, 02-21-2013, 04:53 PM
Dr.Web live cd, but i guess you reloaded so it wont matter. That is kind of common behavior for a flash malware.

Posted by weredigital, 02-21-2013, 04:57 PM
Possible related ? just released Moderate: openssh security, bug fix and enhancement update https://rhn.redhat.com/errata/RHSA-2013-0519.html

Posted by Steven, 02-21-2013, 04:59 PM
Centos 6 only.

Posted by cloudhopping, 02-21-2013, 07:43 PM
Thanks for the clarification on the Cent6 only ... Also - for the DR Web. Dr Web found a few things that the other scans did not... nothing big I hope... but those are offline @ least now.

Posted by Steven, 02-21-2013, 07:49 PM
What did it find? Please send me what you find.

Posted by Zadmin, 02-21-2013, 08:02 PM

Posted by TravisT-[SSS], 02-21-2013, 08:09 PM
Finally some big guys are noticing this.

Posted by Steven, 02-21-2013, 08:13 PM
They have server details too.

Posted by Phil @ NodeDeploy, 02-21-2013, 08:29 PM
Edit: nevermind, my bad.

Posted by NE-Andy, 02-21-2013, 08:32 PM
I am not sure if I follow this correctly. He mentioned in the beginning of the article that it hooks into md5 init, update and final, but then in the end he suggests the MD5 checksums are important. How can md5 checksums calculated on the system (to verify against original installation) be trusted, if the MD5 functions are compromised?

Posted by Steven, 02-21-2013, 08:38 PM
Ok. We all have been so worried on the sshd aspect to this. But we forgot to take a look at 'ssh'.. this little lonely binary used to initiate ssh connections with other servers. Well... You go and login to another server using an infected machine (which you may not know is infected). Guess what happens, our well known friend here: shows up again, sending your other servers login details out. So really all you need is 1 compromised server, to have multiple.. if you use your server to login to other servers. They are in memory too. --- With that said.. change all passwords to your servers even if they are not 'infected' if you may have used an infected machine to login to another server.

Posted by Zadmin, 02-21-2013, 08:43 PM
That's why I was asking if ssh binary was found modified cuz i saw it being patched as part of hacker bash history

Posted by brianemwd, 02-21-2013, 08:47 PM
Just received this from cPanel:

Posted by BeZazz, 02-21-2013, 08:50 PM
I just received this from cPanel (the email looked legit) EDIT: Beaten.

Posted by weredigital, 02-21-2013, 08:51 PM
I recived it as well Is cpanel going to help us all fix now?? many of us have said may be them many times Last edited by weredigital; 02-21-2013 at 08:53 PM. Reason: error

Posted by brianemwd, 02-21-2013, 08:52 PM
Yeah I am pretty pissed off right now.

Posted by mindnetcombr, 02-21-2013, 08:56 PM
And about servers without cpanel being exploited? Maybe using same passwords than cpanel servers, with password saved on cpanel ticket system?

Posted by matbz, 02-21-2013, 08:58 PM
They neglected to tell the recipients that if they are using keys, they should delete the private key from the server straight after they have downloaded it! Ugh!!

Posted by brianemwd, 02-21-2013, 08:58 PM
This is clearly a workstation exploit which means cPanel are not the only ones affected. No one knows how deep this rabbit hole is yet.

Posted by Steven, 02-21-2013, 08:58 PM
Its being done via the library.

Posted by Steven, 02-21-2013, 08:59 PM
Likely still a workstation issue.

Posted by Steven, 02-21-2013, 09:02 PM
Want to say again -- if you logged into another server from your INFECTED machine.. You NEED to reset the password again.

Posted by Patrick, 02-21-2013, 09:02 PM
You mean it's not PHP, cURL, lixproxy or HTML5 + CSS that led to the exploit?! Well I'll be damned. For a while there, I was doubting myself based on all the random crap being thrown around in this thread by people who would probably be lost using Linux without a control panel.

Posted by NetworkPanda, 02-21-2013, 09:04 PM
ok, so the cPanel servers compromise appears to be one of the possible ways of infection, but not the only one. The other methods mentioned in this topic (workstations infection etc.) are still valid, since there are servers infected without running cPanel, but Plesk, DirectAdmin or no control panel at all.

Posted by matbz, 02-21-2013, 09:10 PM
It can't steal private keys that are not there, but perhaps they could be stolen upon creation. Best to create the keys locally using PuttyGen and upload the public key perhaps.

Posted by brianemwd, 02-21-2013, 09:13 PM
I mean cPanel the company, not cPanel servers.

Posted by brianoz, 02-21-2013, 09:22 PM
It's occurred to me to wonder whether anyone is saving root WHM passwords in their browsers? Has anyone checked into that? The way these things work is they grab the passwords from saved files/registry entries. Of course, keylogging might also yield them fruit, but it's a slower process. We've experienced hackers grabbing saved passwords from customers for years now - at least 3 or 4.

Posted by ttoh, 02-21-2013, 09:23 PM
As an FYI, I had also opened a cPanel ticket regarding an issue a few days prior to CSF letting me know the box may be exploited - though I've not received a similar notification from cPanel. cPanel staff also logged into the server in reference to that ticket. Last edited by ttoh; 02-21-2013 at 09:29 PM.

Posted by Steven, 02-21-2013, 09:31 PM

Posted by Steven, 02-21-2013, 09:34 PM
Eric had plesk and directadmin servers compromised.. but he likely logged into them from a compromised server. 'ssh' was sending out the password info.

Posted by DedicatedBox, 02-21-2013, 09:36 PM
It seems a bit premature and baseless to accuse cPanel of being the root of all problems.

Posted by ttoh, 02-21-2013, 09:40 PM
I tend to agree - but, all possibilities need to be looked at in trying to determine possible infection vectors.

Posted by JPerkster, 02-21-2013, 09:42 PM
Cheers, this needs to be at the top. Tons of servers have been compromised and it's not very surprising that a company with (probably) a lot of servers would also have one or two. We still don't know the root cause yet, and there are still a lot more commonalities other than cPanel. tl;dr Don't go on a witch hunt without all the facts.

Posted by lbeachmike, 02-21-2013, 09:45 PM
Do we know if the new credentials are also hijacked when changed via WHM absent a subsequent SSH login?

Posted by FastServ, 02-21-2013, 09:53 PM
I had a ticket open with Cpanel last week and that certain server hasn't been compromised. Changed passwords today, just in case.

Posted by lbeachmike, 02-21-2013, 10:08 PM
Agreed. However, I think cPanel needs to provide more information and become an active collaborative participant in the investigation process. I am particular they must have additional details worth sharing sooner rather than later.

Posted by brianoz, 02-21-2013, 10:11 PM
As the hijack code was in a shared system library used for password verification in many programs, it's safest to assume all credentials on a hacked system were compromised. You could check this by checking to see if the cPanel binaries were loading libkeyutils, but you might miss one, and it might vary amongst machines.

Posted by FLDataTeK, 02-21-2013, 10:23 PM
This also raises the question as to if there is a possibility that cPanel's repo's being compromised if they use this same server/servers to access the repo boxes. I also want to take this time to thank all the people who have been working night and day to figure out this exploit.

Posted by The3bl, 02-21-2013, 10:28 PM
They don't, they have repo servers all around the world in various data centers. I have installed four new cpanel servers in the last 2 days and none of them are hacked I have checked for that. So I would think it is just the help desk or a work station there.

Posted by TheVisitors, 02-21-2013, 10:30 PM
Yes I used only the LIVE CD to connect. I have not cleaned up yet as there is no cure yet or prevention yet.... I'm not going to put my site offline in order to format and re-install until we know what is causing this and how to prevent it. Nothing yet from this thread or else where has found sure solution. Last edited by TheVisitors; 02-21-2013 at 10:44 PM.

Posted by FLDataTeK, 02-21-2013, 10:30 PM
Yeah I would figure it to be a help desk box also but you never know.. And if someone did not make them think about checking then it might go overlooked..

Posted by tomfrog, 02-21-2013, 10:36 PM
Many members and non members have made an outstanding contribution to the hosting community regarding this issue. But I would like to thank Steven, because I think he deserves a special: Thank you! Cpanel failed. But they've also been there for us in the past. It's our turn to stand by them now.

Posted by ttoh, 02-21-2013, 10:44 PM
I agree with thanking Steven. He, and his cohorts, have put a lot of effort into trying to shed some light on this - and continue to do so.

Posted by Steven, 02-21-2013, 10:44 PM
Everyone.. please make sure you keep your guard up... just in case. Please remember that just removing the lib is not enough. You need a is reload. While you can clean it by removing lib, rebooting, password changes, removing keys... you are still compromised... and you need to treat it that way. Also want to mention.. it might be a good time to get ssh keys in place, a vpn, Pam_access module, and more. Remember.. the root kit was sending ssh logins to other servers to the c&c. It might be wise to physically power cycle the machines as well. Make sure you keep up on updates both to your server and desktops/laptops. Even over looked things like acrobat reader... and Microsoft office.. keep up with updates Last edited by Steven; 02-21-2013 at 10:47 PM.

Posted by LeadDogGraphicStudio, 02-21-2013, 10:50 PM
Thank you Steven for all the hard work, help and advice. Do you have a link for info on how to use the VPN for better security?

Posted by Steven, 02-21-2013, 10:52 PM
No.. but you can Google for openvpn tutorials and then lock your servers down to the VPN. Same goes for staff sections of support desks, etc.

Posted by Wintereise, 02-21-2013, 10:52 PM
Basically, you configure a VPN network and make all of your servers have ALL their management interfaces on that private network. So to say, nobody can even get to SSH without getting into the VPN network first. OpenVPN is excellent for this, and be sure to build in some sort of redundancy on this. Otherwise, if your single VPN server dies, you won't be able to manage anything. That aside, excellent contribution Steven. A heartfelt thanks from me too.

Posted by Orien, 02-21-2013, 10:54 PM
Updated the thread summary again. Thanks Steven and everyone for their hard work.

Posted by Dathorn-Andrew, 02-21-2013, 11:06 PM
I would think this goes without saying but... if you ever need to give a 3rd party access to your server (such as cPanel, the company) you need to change the login immediately after their work is completed or simply provide a temporary login for them to use that is removed afterwards. The fewer people that know your login info, the safer you will be. You can't trust a company simply because they are big. Just look at the infections coming from the NBC website today. I cringe at the thought of browsing the internet these days without noscript, disabled java[script]/flash, or some other form(s) of protection well beyond your basic anti-virus/malware.

Posted by LeadDogGraphicStudio, 02-21-2013, 11:07 PM
Awesome, thanks for the info, that is what I had thought you meant. Makes good sense, to lock down the entry points for management to one to several addresses just as another layer of protection. So in my understanding, setup 1 or more other servers as VPN hosts, then set the web server to accept SSH, FTPES & WHM connections from those ip addresses of the VPN servers only.

Posted by Wintereise, 02-21-2013, 11:11 PM
That's the idea, yeah. Though, you can take it one step further. Connect your servers to the VPN network, and make the SSHD listen on the private IP assigned. Just adds another layer

Posted by Steven, 02-21-2013, 11:12 PM
Yes. With Pam_access you can grant root login to your Ips only if you allow people to login to ssh as users. Just make sure.. you VPN servers are secure. Some implemented duo security. I'm not sure how much I like about that however.

Posted by LeadDogGraphicStudio, 02-21-2013, 11:12 PM
Wow, who else is going to be added to the list before this month is over. Damn near half the internet will be hacked / infected before the month is over. It is quite obvious that large companies need to review their security practices.

Posted by Steven, 02-21-2013, 11:15 PM
I think.. everyone big and small need to review their practices, myself included.

Posted by Wintereise, 02-21-2013, 11:15 PM
I implemented Google auth, just have to use a custom auth script for it. Depending on the response from Google, either exit 0, or exit anything else. Pretty simple hooking mechanism.

Posted by NE-Andy, 02-21-2013, 11:26 PM
Might be a good idea to implement a 2-step authentication system for SSH.

Posted by jalapeno55, 02-21-2013, 11:34 PM
Am I hacked? This is what AVG is reporting:

Posted by Scott.Mc, 02-21-2013, 11:37 PM
No it's just trying to follow the symlinks which don't exist as you likely don't have the devel package. Nothing to worry about.

Posted by TheVisitors, 02-21-2013, 11:37 PM
Would love it, but let me know when they can do this without a smart phone. I'm all for it if Google calls me just as they do when I log into my gmail.

Posted by LeadDogGraphicStudio, 02-21-2013, 11:37 PM
I like this idea, i was looking into the Duo Security mentioned earlier in the thread, but this is the same type of setup that is implemented through Google Auth. This looks like a nice step for extra protection. I only wish there was a way to hook it into the WHM/cPanel login. The only info I could find on that idea is here: http://forums.cpanel.net/f145/multi-...279991-p2.html cPanel as always has a bag of excuses as to why they don't do something.

Posted by weredigital, 02-21-2013, 11:51 PM
What steps should we be taking now? i understand a new install is still needed but should we cpanel users wait till cpanel gives the all clear as i assume they will. no reason to start installing the new cpanel if its possible not good. will there be some notice given eventually does any one know. I do wish cpanel would pipe up here more. Last edited by weredigital; 02-21-2013 at 11:57 PM.

Posted by nwilkens, 02-22-2013, 12:07 AM
Duo can do this without a smart phone already, it just calls you or texts you auth codes. Also Duo integrates well with Yubikeys -- another no smart phone solution. There are many great 2FA solutions, and I'd be happy to discuss and provide my time for phone, email, or chat to guide anyone in the right direction to help prevent these types of scenarios. Nick Wilkens nick.wilkens@mnxsolutions.com 888-877-7118

Posted by LeadDogGraphicStudio, 02-22-2013, 12:11 AM
Google auth has a Chrome plugin you can use.

Posted by RomanVelcom, 02-22-2013, 12:21 AM
No new libs in /lib64, also no wierd connections to SSH or from SSH. ecc16b9f0c1fd1a4d699188ec5b13a78 /lib64/libkeyutils-1.2.so Its cPanel Update something ? cPanel + CentOs 5.x (2.6.18-348.1.1.el5) Last edited by RomanVelcom; 02-22-2013 at 12:24 AM.

Posted by lbeachmike, 02-22-2013, 12:33 AM
Hey guys - Wanted to echo the thanks to all partipants in getting this figured out and hopefully under control. Impressive work, and I look forward to the future point in time where my experience level will be such that I'm able to contribute. For now, I'm wondering which of the temp-cleanup scenarios that have been mentioned here is the recommended temp-cleanup procedure? There have been a few variations posted. I realize that these are only temp measures ahead of a fresh OS install. Thanks again for everything.

Posted by NoSupportLinuxHostin, 02-22-2013, 12:38 AM
I love cPanel but I don't expect them to take the lead on this. They still have not officially handled the SymLink issue, which is a serious problem in their default configuration. Here is a link to the SymLink issue that cPanel has still not addressed: http://forums.cpanel.net/f185/how-pr...rs-202242.html On major security issues, cPanel seems to want to leave it up to the server admins to manually fix vulnerabilities in their default configuration. In the forum thread about SymLink issues, there were plenty of possible solutions discussed but the default cPanel configuration does not include any of the potential solutions.

Posted by TravisT-[SSS], 02-22-2013, 01:00 AM
But one could argue that it really is not their job to patch other peoples software. It is Apaches. Apache honestly needs to fix Apaches security issue.

Posted by Steven, 02-22-2013, 01:04 AM
Technically.. its not even apaches problem. Its following symlinks, over linux permissions that allow them to do so. User nobody can read public_html, so why wouldn't apache be able to read the files? Sure.. the race conditions in symlinksifownermatch are apaches problem, but followsymlinks + the way cpanel is setup out of the box is not apaches problem.

Posted by NoSupportLinuxHostin, 02-22-2013, 01:18 AM
I agree up to a point, but I still think cPanel has some responsibility to deal with it. Most people who install cPanel believe it is reasonably secure by default. Even if the admin selects suPHP, the SymLink vulnerability lets sites get around the security and then read each other's config files (and then hack each other's databases). cPanel should at the very least document the problem and give admins some hints about possible solutions (like how to disable FollowSymLinks). Or better yet, cPanel could simply choose more secure default settings for Apache (by disabling FollowSymLinks by default). Shipping a product that has a serious security vulnerability by default is never a great idea. There are probably thousands of servers in production that have not been manually secured against the SymLink vulnerability, and there are more being brought online each day that are vulnerable by default.

Posted by TravisT-[SSS], 02-22-2013, 01:21 AM
Well anyone who is going to admin a box should know the services and be in the field to know what needs to be secured and not. Same goes with installing an OS. Typically it is setup to work then you have to fine tune and secure it to your needs. But I'll agree they could at least post some help regarding the threats.

Posted by tarsiran, 02-22-2013, 02:23 AM
i have a lot of email via csf: Possible root compromise: File /lib64/libkeyutils.so.1.9 exists. For more information see: http://www.webhostingtalk.com/showthread.php?t=1235797 how can i stop this email on csf? i wanna stop to send this email

Posted by Wintereise, 02-22-2013, 02:25 AM
Ever think about taking care of the underlying issue first? Disabling it will only make you weaker.

Posted by PLE, 02-22-2013, 02:31 AM
This is also related to cPanel's suPHP setup. I noticed on my last cPanel server that it will not automatically create a php.ini file for each site on the server. The problem is that the open_basedir path is blank in this case because suPHP require that it need to be configurated over a custom php.ini. The open_basedir setting in the Apache configuration file works only for an Apache embedded PHP interpreter like mod_php or mod_ruid2. Last edited by PLE; 02-22-2013 at 02:35 AM.

Posted by NE-Andy, 02-22-2013, 03:08 AM
Ah, yeah... there's no way to tell Google to call/text you when you try to SSH in, which means a capable device is required. Perhaps an Android/iOS tablet or iPod Touch? Bit clunky to carry a tablet around, though.

Posted by WiredTree Joe, 02-22-2013, 03:22 AM
The Nexus 7 is an awesome daily driver for those long too and from work commutes. I love mine. Anyways, It looks like you can use a Yubikey too... which is pretty easy to lose along with your keys. ( mnxsolutions.com/security/secure-ssh-and-wordpress-with-two-factor-authentication.html ) Authy is another method for 2 factor auth that's not Google based. ( github.com/authy/authy-ssh ) Sorry for the noob links. One day I will stop lurking and earn enough posts to have links! One day! Last edited by WiredTree Joe; 02-22-2013 at 03:24 AM. Reason: Noob links apology :(

Posted by LeadDogGraphicStudio, 02-22-2013, 03:30 AM
There is a Chrome extension for Google Auth

Posted by tomfrog, 02-22-2013, 04:03 AM
Life is perfect because it's not perfect... Read Steven's advice... It's very good advice... How many of us give away password to Datacenters, cpanel, rvsitebuilder and others without considering the security implication? I learned from this experience! I'm still learning from this experience. It's not over yet. Clean up.

Posted by ciprianbalus, 02-22-2013, 04:28 AM
Salutations, You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis. As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue. --cPanel Security Team I always change root password before giving them to cPanel support and after they investigate i change back to my password! Last edited by ciprianbalus; 02-22-2013 at 04:31 AM. Reason: adding text

Posted by tomfrog, 02-22-2013, 04:38 AM
ciprianbalus Maybe you shouldn't give them root password... Ask them for a ssh public key. In WHM, you can import and authorize it. At the command line, you will see the public key in /root/.ssh/publickeyname.pub and in /root/.ssh/authorized_keys Or don't even use root.

Posted by Phil @ NodeDeploy, 02-22-2013, 05:45 AM
So.. the only recommended thing is to reinstall the entire operating system? Dang.

Posted by deriamis, 02-22-2013, 05:45 AM
I have done this on my personal server to great effect. I disabled root logins and disallowed password auth, then set up /etc/pam.d/ssh and /etc/pam.d/sudo (basically, anything that asks for a password) to require a second factor. It's not quite enough to do that, though. You also need to make sure you slow down brute force attempts on your server. If you don't want to use CSF or fail2ban or something like that, this iptables ruleset works for me: That allows up to 8 connections per hour from a single IP and then temporarily bans the connection for a seriously long time. I should point out that changing your SSH port is fairly useless to a determined hacker (as we have seen here), so I don't know that I would bother.

Posted by adebenc, 02-22-2013, 05:49 AM
What did I told you guys ? I give 90% chances that cPanel was the problem , look what I just receive from cPanel : Important Security Alert (Action Required) Show Details From no-reply@cpanel.net To my e-mail Salutations, You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis. As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue. --cPanel Security Team

Posted by deriamis, 02-22-2013, 05:55 AM
Yep. It sucks, but you never know what someone else did with root on your server. You could spend a very long time looking for what they did and still miss something.

Posted by demil, 02-22-2013, 08:23 AM
adebenc are hacked servers which have nothing to do with cPanel, using Pleks, Directadmins, some servers using no cpanels at all etc. But if I would be from Plesk or Directadmin, for sure I will start thinking that probably this was a large attack to all hosting-cpanels-providers... and probably RBN abused all of them, like they did with all major companies...

Posted by NetworkPanda, 02-22-2013, 08:31 AM
Yes, it is faster formatting the disk and reinstall everything clean, than finding new problems every day and trying to fix them for days and weeks... Or even worse, having problems you have not found, with unknown results.

Posted by matbz, 02-22-2013, 08:42 AM
very sound advice!

Posted by adebenc, 02-22-2013, 09:14 AM
Those cPanel computers were affected by a threat, surely this threat has infected other computers (desktops), however at least 80% of servers got compromised because of the cPanel technical team computers infection. Every time the cPanel team login to an server , the server got compromised imediatly (1-2hours) (this is proven). is already known that this thread was found to more desktops from people , and they probable login from their desktop to their server and got their server compromised (in the same way how cpanel technical team affected the servers). We should be happy that the CentOS platform is not the cause of the issues and CentOS operation system should not louse reputation in this situation , they have no fault. good luck

Posted by jzukerman, 02-22-2013, 09:28 AM
A good tip when contacting support of any type and they require login credentials: Change the password to something new before giving it to them, then after the issue is resolved, change it again to something new. Of if you can, create a temporary user account then disable or delete the account after the support issue is resolved.

Posted by adebenc, 02-22-2013, 09:30 AM
yes , you are right

Posted by cmarchena, 02-22-2013, 09:47 AM
The question is, affects only the 1.9? I have a file with version 1.3 or directly 1. If I run the rpm, the message does not tell me, show me the version of libkeyutils. For example: [root @ xxx ~] # rpm-qf / lib64/libkeyutils.so.1.3 Keyutils-libs-1.4-4.el6.x86_64 I understand that in this case, is free from problems. That if, in Plesk, all in the chroot folder, there is a / var/www/vhosts/chroot/lib64/libkeyutils.so.1 if displays. example: root @ xxx ~] # rpm-qf / var/www/vhosts/chroot/lib64/libkeyutils.so.1 the / var/www/vhosts/chroot/lib64/libkeyutils.so.1 not owned by any package But maybe it has nothing to do ... because as I say, with all plesk, different versions, one installed until yesterday.

Posted by FastServ, 02-22-2013, 10:01 AM
Am I the only one who didn't get an email from Cpanel?

Posted by realvaluehosting, 02-22-2013, 10:02 AM
If you did not raised any support ticket with them in last 6 months, you might not have recd. the mail

Posted by GoTek-JP, 02-22-2013, 10:06 AM
Have you opened a ticket in the past 6 months? I doubt they're sending it to everyone.

Posted by shackrock, 02-22-2013, 10:06 AM
This is a good thing. It probably means you may be safe, like me!

Posted by matbz, 02-22-2013, 10:07 AM
No, we've not seen one either, but after their support left a server inoperable after they investigated an issue a long time ago, we swore we would never allow them into a server again but only advise.

Posted by FastServ, 02-22-2013, 10:11 AM
5 or 6 tickets. One last week in fact.

Posted by servermanaged, 02-22-2013, 10:17 AM
Hi all. Off topic: This night I've detected a distribuited attack against port 21 of a server of mine. Almost all probing come from dedicated server belonging to some small ad medium hosting company. I'm on the way to forward abuse reports to all.

Posted by tomfrog, 02-22-2013, 10:22 AM
The file name kept changing. Did you login from any cpanel server to this one? Paste the result of strings /var/www/vhosts/chroot/lib64/libkeyutils.so.1

Posted by SoftDux, 02-22-2013, 10:35 AM
Can you recommend any such application which don't run through a 3rd party service like google, which could be compromised before the hacked try and hack into the victim's server?

Posted by TheVisitors, 02-22-2013, 10:43 AM
So we've still not figured out the cause and so no one has yet found a prevention. The infected workstation theory does not apply to us, as we use a LIVE CD with Linux to boot from, with no physical hard drive. Its a fresh & clean system upon every boot.

Posted by cmarchena, 02-22-2013, 10:44 AM
Hello, root@rs lib64]# strings /var/www/vhosts/chroot/lib64/libkeyutils.so.1 @@2@: I P __gmon_start__ _init _fini __cxa_finalize _Jv_RegisterClasses keyctl syscall keyctl_assume_authority keyctl_set_timeout keyctl_set_reqkey_keyring keyctl_negate keyctl_instantiate keyctl_read keyctl_read_alloc malloc realloc keyctl_search keyctl_unlink keyctl_link keyctl_clear keyctl_describe keyctl_describe_alloc keyctl_setperm keyctl_chown keyctl_revoke keyctl_update keyctl_join_session_keyring keyctl_get_keyring_ID request_key add_key libdl.so.2 libc.so.6 _edata __bss_start _end libkeyutils.so.1 KEYUTILS_0.3 KEYUTILS_1.0 GLIBC_2.2.5 t$(H T$0H D$ L t$ H t$ H Not in cpanel servers, but in Plesk. The Cpanel in theory I have them all without problems.

Posted by Patrick, 02-22-2013, 10:45 AM
Here's the thing, When you have 1000 people saying one thing and 1 person (you) saying something else... that's your problem. I say that in the politest way possible, but it's clear for a lot of people and a lot of companies that the point of compromise was infection through a workstation. You either have a super unique situation that none of us will ever figure out, or you're trolling to be different. Either way, good luck with that.

Posted by TheVisitors, 02-22-2013, 10:52 AM
Actually, if you read the thread.... And I understand 80+ pages is a lot..... .... You'll notice not everyone agrees with the workstation theory either. In fact there are dozens of post where people said they scanned their workstation and found no infection. Some of those individuals even wiped out (formated) their workstation and their servers, but still got infected. So I guess that is "all" our problem? No disrespect, but people like you should not give support to anyone else.

Posted by NetworkPanda, 02-22-2013, 11:09 AM
Your file appears to be clean, the original libkeyutils file. It does not contain the strings of the infected libkeyutils files.

Posted by DennisC, 02-22-2013, 11:10 AM
I went into WinSCP and did not see the file. I am using cpanel with centos5.

Posted by cmarchena, 02-22-2013, 11:12 AM
Thanks! I can not find any infected file, can I tell where to see it? To buy it with others. For example, I have a 1.3 that I say this completely different: I P {?Nq __gmon_start__ _init _fini __cxa_finalize _Jv_RegisterClasses keyctl syscall keyctl_session_to_parent keyctl_get_security keyctl_get_security_alloc malloc realloc keyctl_assume_authority keyctl_set_timeout keyctl_set_reqkey_keyring keyctl_negate keyctl_instantiate keyctl_read keyctl_read_alloc keyctl_search keyctl_unlink keyctl_link keyctl_clear keyctl_describe keyctl_describe_alloc keyctl_setperm keyctl_chown keyctl_revoke keyctl_update keyctl_join_session_keyring keyctl_get_keyring_ID request_key add_key libdl.so.2 libc.so.6 _edata __bss_start _end libkeyutils.so.1 KEYUTILS_0.3 KEYUTILS_1.0 KEYUTILS_1.3 GLIBC_2.2.5 ATSubH D$`H D$ H L$8L D$@H T$(H fff. t$ H fffff. fff. t$ H fff. t$ H fffff. fff. ffffff.

Posted by FastServ, 02-22-2013, 11:17 AM
Playing devil's advocate, I'll mention that if you're using a live CD I can assure you are using out-dated software. Your theory is a nice one but it doesn't rule anything out. At this point I don't see anyone else STILL arguing the workstation theory except you.

Posted by NE-Andy, 02-22-2013, 11:18 AM
Those are some good alternatives. No need to worry about the links, copy and paste is easy enough, even being on my iPad now I should check into this when I get on my computer. Do you have a link to that? Yep, I use fail2ban on my. You could enable rate limiting on the 2 factor, would break brute force. Though, with 2 factor like google authenticator in place, the likely hood of them getting through is already so low that they probably will never break through. So fail2ban and other iptables tweaks are just nice bonus

Posted by WhoElse, 02-22-2013, 11:19 AM
About handing out login credentials to "unknown" people. (Like support desks etc). What I do is: - I change the root pw. - I create 2 users, one without any privileges, one with sudo privileges. - I login as the user without privileges and start screen. - In the screen terminal I ssh to the local host logging in as the user with the sudo privileges. - Once logged in I sudo to root level. - Then I disconnect from the screen session. - I give the details of the non privileged user to the people requiring access, and tell them to login and then connect with "screen -x" to the privileged shell. - At the same time I will have a screen session open attached to the same terminal. That way I can see exactly what they are doing in the root shell... If you see something you don't like, you just kill the screen process... - Once done I just remove the 2 new users, and everything should be back to (relative) safety. I also change the root pw again, just in case... I know, it is maybe not the best way, but it gives a lot more control/overview what is happening... Last edited by WhoElse; 02-22-2013 at 11:20 AM. Reason: Minor editing

Posted by TOCICI, 02-22-2013, 11:43 AM
I'd say it's an OpenSSH exploit: https://rhn.redhat.com/errata/RHSA-2013-0519.html

Posted by FastServ, 02-22-2013, 11:45 AM
Unlikely... that only affects RHEL6. Doesn't explain RHEL5 or Debian.

Posted by weredigital, 02-22-2013, 11:45 AM
Thats for Centos 6 not applicable to most Steven Commented yesterday on that

Posted by kpmedia, 02-22-2013, 11:45 AM
Yeah, but stuff still happens while it's running.

Posted by Patrick, 02-22-2013, 11:47 AM
Except that I am a highly knowledgeable person who has found several root / admin level flaws in software you use, along with a lot of other web hosting software over the years making me 100x more qualified to speak than the average person in this thread - especially you. You can ask Steven about that, if you would like some confirmation... The only people who are not agreeing with the workstation theory are still caught up in other stupid and highly unlikely theories involving exploits in PHP or cURL or OpenSSH making their opinion completely irrelevant. There is absolutely no reason at this point to suspect a flaw in OpenSSH or anything else that led to these compromises, the MOUNTAIN of evidence that has been presented in this thread and privately (that you are not aware of, since you don't have those connections nor would you understand what is being discussed) says otherwise.

Posted by SoftDux, 02-22-2013, 11:55 AM
Patrick, Surely if the workstation theory was correct, we'd see these hacks on other servers like *BSD, OpenSuse, Windows, etc as well. The only infected systems so far is Redhat based or Debian based, which have no significance or could mean the hackers only *knows* those systems. But that's a wild guess. In my case I'm the only one accessing our servers via SSH and always give cPanel staff a changed root password when they need to access it. This makes me believe the hack could have originated from their end and the infection spread through servers directly, i.e. when a sys admin logs into a DirecAdmin / Plesk server from an infected cPanel server. This could probably only have happened from one single admin but once the infection was on a DA or Plesk server, another admin on those servers could have spread it on. Unknowingly off cause. This is probably why more CentOS / cPanel servers are affected than other servers. But what about OpenSuse, Slackware (I know some ppl who still use it), FreBSD, and Windows servers?

Posted by Scott.Mc, 02-22-2013, 12:08 PM
Your asking people on a forum to make educated guesses as each and every environment. All that can be done is conclusions made and the conclusion is the initial vector for most is leaked authentication details. This is what has effected the overwhelming majority, it doesn't mean it's the source of all. Given it's leaked auth details there are plenty of sources they can come from due to users re-using this information. The notion that it's still some unknown vulnerability in some unknown service is highly unlikely and has been proven as such. Those still shouting of an unpatched vulnerability in openssh need to stop, this is frankly ridiculous. Regarding the OS types this is simply because the overwhelming majority of users around these parts exclusively use those OS's. The most common source (not the only) of the authentication details was cPanel and the only OS's they support are CentOS, Red Hat and CloudLinux officially. Remember the bulk of people around here take security as a secondary precaution and as such once someone has access to one system they quite often can jump from A->B->C->D simply because of password/authentication data reuse. A typical example, lets assume it wasn't even your work station and was leaked from cPanel's helpdesk, you have a shared server called "server90" you submit a ticket to cPanel for them to investigate an issue and provide them your root password. The attacker first obtains this and is contained to this server, through the keylogger they obtain other passwords that you used and they happen to try "server10" with one of those and as you used the same details they are now no longer contained to that one system. Rinse-repeat and now they have access to your billing system which in turn leads to even more servers, then your email system which in turn leads to your email spools which contain even more data to mine and access and the process keeps repeating and repeating. It's up to the people reading to draw conclusions on weather to believe those edge cases that disagree with the majority. It's easy enough to auth access logs from the system itself even if they have been removed, I provided a very simple and dirty way to do this earlier in the thread. Last edited by Scott.Mc; 02-22-2013 at 12:13 PM.

Posted by afletch, 02-22-2013, 12:22 PM
Note that the pam_ssh_agent_auth module is not used in Red Hat Enterprise Linux 6 by default." You will probably find that, unless you specifically installed and configured it, you don't even have pam_ssh_agent_auth installed.

Posted by TheVisitors, 02-22-2013, 12:38 PM
My web hosting service provider allows me to pick a pre-made image (see screen shot) I suspect that maybe the cause for my issues and that the "images" need to be wiped (formatted) and re-made. It is also possible the host system (OpenVZ) may also be compromised. This is my theory. Attached Thumbnails  

Posted by Ramprage, 02-22-2013, 01:14 PM
Is anyone using a file integrity scanner like tripwire or AIDE? Surely these can give everyone more details as to specific file changes. Found this - lots of great info if you haven't seen it: https://isc.sans.edu/diary/SSHD+root...the+wild/15229 Last edited by Ramprage; 02-22-2013 at 01:21 PM.

Posted by PLE, 02-22-2013, 02:48 PM
This directory has been generated by Plesk. That's why this file is in no package. It is a chroot environment to provide a secured shell access to customers.

Posted by Ceetoe, 02-22-2013, 02:54 PM
Nice number of Chrome vulnerabilities for RCE... http://msisac.cisecurity.org/advisor...3/2013-023.cfm

Posted by deriamis, 02-22-2013, 03:03 PM
I know I am not Patrick, but there is a relatively simple answer to this: not necessarily. You have to remember that most hackers are lazy and concern themselves with the easiest route to success. They don't need access to every server on the Internet - just "enough" to do what they want to do - which is usually to annoy the pants off some big corporation or government, or just to have fun. A keylogger delivered through a Java exploit would still need to target the keylogger on the platform uponon which it will be run. While they could make a keylogger for every conceivable system out there, it makes more sense to make it for the most populous, vulnerable, and oft-misused systems. In this case, that would be Windows and (possibly) Mac. Understand that the real story here was "how the hackers got root on my server" and not "what the hackers did on my server". We were fairly sure shortly into this what the trojaned sshd was doing. The question was of how it got there in the first place. We now have evidence of a keylogger on Windows delievered through a Java exploit, which explains the majority of the cases, and that's good news. It doesn't explain all of the cases, but that could be due to misremembered details, curious and forgotten circumstances, or any number of other reasons. It could also be due to a cause we have not found yet - but that does not mean the cause we have found is invalid, either. We just need more information for them if they become an ongoing issue. I'm not really a security expert, but I hope all of that makes sense. I'm just glad we can finally explain something.

Posted by MrCheck, 02-22-2013, 03:38 PM
Hey Guys, last time I gave a root pw to cPanel support was in September last year. Now I'm thinking about being infected, too. Some questions/thoughts.. 1. Are there any known cases of servers that have been infected such a long time ago? 2. I did not find any bad libkeyutils-file on my server. Isn't it possible that only a part of infected servers have such a file, and other servers are still "inactive"..?

Posted by patchwork, 02-22-2013, 03:39 PM
The bank Santander recommends Trusteer Rapport (It's Free) and can protect any website not just online banking access, with everything that's been going on I thought it might be useful to share the link.

Posted by Steven, 02-22-2013, 03:57 PM
There has been reports as long as august.

Posted by MrCheck, 02-22-2013, 03:59 PM
Do you know about anything else than infected libkeyutils? Anything I can do more to be sure I am not infected?

Posted by Steven, 02-22-2013, 04:01 PM
2 weeks of research, multiple eys on this, no one has found any proof of anything other than that file. There is a older rootkit involving replacing openssh itself, but thats not the case here. If you are concerned, you have ONE option and that is a os reload. Other then that, if you have concerns there is no other way for you to be sure.

Posted by MrCheck, 02-22-2013, 04:06 PM
Of course I could reload, but if there is nothing else known than libkeyutils I don't see a reason for doing this. You are doing a great job here, thank you!

Posted by MrCheck, 02-22-2013, 04:15 PM
Hey Steven, unfortunately I can't send you a PM, but I just sent you a contact form via your web page.

Posted by emptymind, 02-22-2013, 04:35 PM
After cross-referencing all cpanel tickets that were opened since our last system wide root password changes, we found one server that had a compromised libkeyutils. Generally we change the root password after any such access by cpanel, this one apparently slipped through. Please forgive any logic errors below, It was late. I also typed most of this up then, with the plan to go over it again today. Which I have, so if anything looks out of order. Thats why. I'll begin with the usual, thanks to Steven and all the others for all the time spent researching this, and all of their efforts to share with and help the community at large. OK, enough with that, to the meat! -- [/lib64]# stat libkeyutils-1.2.so.2 File: `libkeyutils-1.2.so.2' Size: 34584 Blocks: 80 IO Block: 4096 regular file Device: fd00h/64768d Inode: 49938719 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-02-22 01:04:35.000000000 -0800 Modify: 2007-01-05 23:55:38.000000000 -0800 Change: 2013-02-20 04:47:56.000000000 -0800 -- Version 1.1.0 XXXYwZfC62L Xver Xcat Xbnd sshd:1 %s %s %s gshd:1 %s %s %s ssh:1 %s %s %s %d key:1 %d %s %s %s %d %s %s %s g:%s %s u:%s %s ssh:1 %s %s %d LOGNAME PEM_write_RSAPrivateKey PEM_write_DSAPrivateKey MD5_Init MD5_Update MD5_Final options hostaddr idtable audit_log_user_message audit_log_acct_message hosts_access pam_authenticate pam_start __strdup root crypt abcdefghijklmnopqrstuvwxyz biz. info. net. --- Linux XXX 2.6.18-348.1.1.el5 #1 SMP Tue Jan 22 16:19:19 EST 2013 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release CentOS release 5.9 (Final) 'Cleaned up' by renaming the hacked library, repointing the link to the 'good' library, ran ldconfig, checked lsof for services attached to the library, restarting them and then quarantining the hacked library. Checked the ctime on the ‘good’ library, and it was updated the same as the ‘hacked’ library. I will note that just trying to change the link without renaming the library was fruitless as it was just instantly changed back. Renaming the hacked file file first appears to interrupt that loop. re-installed the keyutils-lib rpm after verifying the rpm version that the db said it was attached to with rpm -qf, and making a copy for comparison purposes. After the rpm refresh the md5 and filesize was different. Pre RPM re-install: [/lib64]# stat libkeyutils-1.2.so File: `libkeyutils-1.2.so' Size: 9472 Blocks: 32 IO Block: 4096 regular file Device: fd00h/64768d Inode: 49938660 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-02-22 01:34:01.000000000 -0800 Modify: 2007-01-05 23:55:38.000000000 -0800 Change: 2013-02-20 04:47:56.000000000 -0800 [/lib64]# md5sum libkeyutils-1.2.so 23839a4a162f1d664e2ceccf5afb7643 libkeyutils-1.2.so Post RPM Re-install: [/lib64]# stat libkeyutils-1.2.so File: `libkeyutils-1.2.so' Size: 7176 Blocks: 24 IO Block: 4096 regular file Device: fd00h/64768d Inode: 49938619 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-02-22 01:52:29.000000000 -0800 Modify: 2007-01-05 23:55:38.000000000 -0800 Change: 2013-02-22 01:52:23.000000000 -0800 [/lib64]# md5sum libkeyutils-1.2.so 2f77cdb95e0800b19e53ef789c36c37c libkeyutils-1.2.so Restarted all services again. Changed root pass by direct edit of the shadow file. As with all root level compromises. The server is now slated for wipe. Finally, apparently I'm a professional lurker, and do not have PM privs. If Steven or others want copies of these files for further investigation, let me know. James

Posted by Patrick, 02-22-2013, 04:49 PM
Doesn't matter. Whenever root is obtained, the system is no longer considered secure regardless of any further work one does - there's just no way to tell what other backdoors have been implemented or what was meddled with. The only way you can obtain 100% assurance that the system is clean is to start over from scratch. If it's just your random data and nothing too important, fine... but if you're hosting clients or sensitive data than I would reload as soon as possible.

Posted by MrCheck, 02-22-2013, 04:57 PM
I don't understand your point of view. There are no indications that my machines might be infected except the fact that I gave root access to cPanel some months ago. Even with a reload I can NEVER be 100% sure that there is nothing going wrong on my servers. So I would need to reload my system every day..!?

Posted by tumble, 02-22-2013, 05:00 PM
Just run a scan and see what it turns up UPDATE (Feb 21): Several Linux anti-malware scanners such as AVG now detect the malicious libkeyutils files based on signature instead of just name.

Posted by Patrick, 02-22-2013, 05:06 PM
Sorry, I think I misread your reply. If you have the SSH backdoor currently present, then you should reload for the reasons I explained above - because you know with 100% certainty that you have already been compromised. If you don't have the SSH backdoor then there's obviously no reason to reload.

Posted by emptymind, 02-22-2013, 05:08 PM
Quick update, Thought to look in our backups (R1Soft) Given that these files ctime was the 20th. the 1.9 version was in the lib64 directory on the 19th. And in the daily/weekly's all the way back to the backup set taken on Dec 15th. The next backup set in the list was Nov 15th and the file does not exist. Of note the Cpanel ticket relating to this server was issued Nov 15th.

Posted by MrCheck, 02-22-2013, 05:13 PM
Ok, now I am with you ;-) I did not find any suspicous libkeyutils file on any system. As this seems to be a proof for not being affected by this issue, I think my systems are clean.

Posted by tecnobrat, 02-22-2013, 05:27 PM
Yes, you need to wipe. And do not use "ssh" from that machine, the "ssh" binary was also replaced. (not sshd ... ssh)

Posted by Steven, 02-22-2013, 05:49 PM
It was not replaced..that binary linked to the library (just like sshd) which has code in it to send the credentials to the attacker and also store in memory. There was an older variant that DID replace the binaries, this is not that variant.

Posted by emptymind, 02-22-2013, 05:55 PM
stat `which ssh`; File: `/usr/bin/ssh' Size: 306064 Blocks: 616 IO Block: 4096 regular file Device: fd00h/64768d Inode: 26385823 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-02-20 04:05:23.000000000 -0800 Modify: 2012-02-22 08:04:57.000000000 -0800 Change: 2012-03-08 03:08:12.000000000 -0800 Doesn't look replaced. And uhh.. I already said I was wiping it so...

Posted by Steven, 02-22-2013, 06:08 PM
The older rootkit, faked that with thats, http://packetstormsecurity.com/files.../touch2.c.html but regardless. It doesn't matter, your reloading so thats good.

Posted by emptymind, 02-22-2013, 06:55 PM
OK.. sarcasm revoked.. (been a long day).. thanks for the heads up! James

Posted by NetworkPanda, 02-22-2013, 07:04 PM
According to this link: https://isc.sans.edu/diary/SSHD+root...the+wild/15229 So, based on this, I have wrote a command which checks all processes in ipcs -mp (using shared memory) and finds to which application or daemon they belong If you find any SSHD processes in the result of this command, then it appears that your SSHD has been compromised by the exploit. On clean systems you should usually find only php processes and in some cases, processes by your virus scanner. Last edited by NetworkPanda; 02-22-2013 at 07:07 PM.

Posted by chinook, 02-22-2013, 07:32 PM
parallels just issued an advisory related to this. Makes me scratch my head

Posted by suhailc, 02-22-2013, 07:57 PM
Care to share please!

Posted by Hellsheep, 02-22-2013, 08:01 PM

Posted by Steven, 02-22-2013, 08:11 PM
Its basically what was posted in this thread. http://kb.parallels.com/115589

Posted by mangia, 02-22-2013, 08:36 PM
I maintain more than 30 servers and non of them was compromised. Maybe the reason for good health of my servers was the fact that I don't use Cpanel The mail copied from one forum and it was sent by Wiredtree to one of their customers

Posted by pjkenned, 02-22-2013, 09:11 PM
1 for Steven!

Posted by WiredTree James, 02-22-2013, 09:20 PM
We have followed this thread closely since it's inception last week. Immediately following word that servers across the net were being systematically logged into mostly through the standard SSH port, we took the painfully proactive approach of blocking outside access of port 22 across our entire network. During that time we also ran continuous scans across our network looking for reported file changes and files not owned by RPM's. It was no small feat, especially considering how little was known at the time and how many clients we were watching over. Of the small handful of servers we found affected by cPanel's security issue since then, we have voluntarily begun OS re-installs just to be safe. We apologize to any WiredTree customers who were inconvenienced when we blocked port 22 access across our entire network. We had and have your best interest at heart. We will continue to investigate this issue as I am sure others will.

Posted by The3bl, 02-22-2013, 11:22 PM
Looks like attacking help desk and work stations is a new normal. http://edition.cnn.com/2013/02/22/te...ked/index.html Makes sense really, it is the place you can get the greatest amount of info at once.

Posted by Steven, 02-23-2013, 12:13 AM
Not really a new thing. Been happening for years. There is just a wide range of possible exploit vectors now in desktops and the same time. Just a couple years ago there was the http://en.wikipedia.org/wiki/Gumblar worm stealing ftp details by the thousands. I see countless cases where ftp details are stolen.. even things like email account details and spam being sent. A customer of mine about 4 months ago had his ftp and google account information stolen and his adwords screwed with. Malware was found on his desktop that was stealing all keystrokes in real time.

Posted by Steven, 02-23-2013, 12:16 AM
Who remembers this? http://www.securityfocus.com/archive...8/2004-09-14/2 It was a exploit that you could get malware installed on your computer by looking at an infected jpg.. yes a image file.. Back in the day, when this was rampant, there was a piece of malware that installed a irc bot on your desktop that logged all keystrokes to a irc channel. People would hack websites, and embed a 1 pixel image.

Posted by TheVisitors, 02-23-2013, 12:19 AM
I remember this affecting a lot of people. And there was no anti-malware, anit-adware, or ant-virus program that could detect it for sometime (a while) .... No one had thought a simple jpg photo could cause an issue. And when you think about it, the Internet is full of them (even on this page). Yes, I remember that.

Posted by LeadDogGraphicStudio, 02-23-2013, 01:01 AM

Posted by The3bl, 02-23-2013, 02:03 AM
Not saying hacking and malware are new that has been going on forever. I am saying it seems more and more they are targeting help desk and developer channels to be able to steal more passwords and user data with one hack than a single sources can provide them. So I suspect things like large providers with help desk will be attacked more often than in the past.

Posted by Linulex, 02-23-2013, 05:39 AM
Plesk also came up with a mail. Strange enough, all they can do is repeat what is said on this forum and even link to it. So they also have clearly no idea what they are talking about and in the end only just scaring people.

Posted by realvaluehosting, 02-23-2013, 05:43 AM
Could you please let me know the URL of the KB Article?

Posted by Linulex, 02-23-2013, 06:10 AM
The link is a few posts up on this page. I can not give it, i have to few posts.

Posted by mangia, 02-23-2013, 06:46 AM
Don't get me wrong but I think that you done what ever you could. I would do the same like you did. Actually you're the only one who actually found something.

Posted by Cyclon, 02-23-2013, 07:31 AM
Is there any news and details about a local workstation infection? I see that there are much more AV's now recognising libkey rootkit, but what is happening with local workstation infections?

Posted by deriamis, 02-23-2013, 07:42 AM
That's the Parallels I know and... well, just know.

Posted by JoyceBabu, 02-23-2013, 07:55 AM
Are these files okay?

Posted by deriamis, 02-23-2013, 07:56 AM
I remember that very well. I have a friend who, back in the day, thought a virus scanner and firewall was "too much bloat" for his desktop. He changed his tune when I installed Back Orifice via an image file I cooked up and made his CD-ROM drive open and shut like the computer was giving him a raspberry. Scared him half to death, it did. Ah, fun times...

Posted by Cyclon, 02-23-2013, 08:34 AM
Yes, that looks fine.

Posted by siwebsupport, 02-23-2013, 10:03 AM
I wanted to emerge from my lurking to publicly thank WiredTree here. I have MANAGED VPS's and Dedicated servers from several very well known providers on WHT (very tempting to list them here but I won't) and WiredTree is the ONE AND ONLY provider who acted proactively about this both with communication and taking action. Many thanks to them and others here.

Posted by Steven, 02-23-2013, 10:26 AM
You should be fine Joyce

Posted by Hellsheep, 02-23-2013, 10:56 AM
I decided to analyze my logs and emails from CSF tonight. My original thought was I was compromised around the 10th of December due to the changed date on the file being that. I checked my cPanel tickets, I had a cPanel ticket open on the 7th to fix a MySQL issue. Root password was provided to cPanel to resolve. Issue was resolved on the 7th. On the 8th I received an alert advising that CSF had crashed and a restart would be attempted automagically..Oddly enough this crash lasted 5 minutes, after CSF came back alive I got an e mail telling me its all OK. However I then received an email advising that /etc/init.d/securetmp failed its md5sum comparison check from CSF. Not sure if its related to anything but a bit weird that its the only time CSF has crashed since the server was built. Either way, this is starting to look more and more like the original entry point being compromised credentials via malware on a local pc/cPanel etc. The times fit for my server at least and since I've configured SSH to be a lot more locked down I haven't been re-infected. Once this all blows over gonna rebuild the box. Edit: Also it's worth noting I never received an email about someone originally logging in as root to the server to place the infected library. The only emails I got were for cPanel logging in as root and myself. So maybe they somehow used a compromised box at cPanel to log into the server to make it seem legit?

Posted by Zimple, 02-23-2013, 11:01 AM
Reloading server OS is not a simple job for Shard Hosting provider. Need to backup all the accounts to somewhere, reinstall the server + cPanel and restore all the sites back with previous configurations. Plenty of works with several hours down time. Really hard job. Do you have any alternative plan ?

Posted by Cyclon, 02-23-2013, 11:19 AM
Best method would probably be to migrate all user accounts to new server. There is no realy alternative plan. Only alternative would be to leave it like that and to pray a lot.

Posted by Zimple, 02-23-2013, 11:27 AM
Some clients already pointed sites using A records. Changing IP of some domains is not a simple task. That is why always I have to be on same IP (server) without moving to a new one. Already doing for last 2 weeks

Posted by NetworkPanda, 02-23-2013, 11:33 AM
If you don't have another server, to transfer temporarily the accounts, then the following one-line command backups all cPanel accounts The backups are stored in /home with filenames cpmove-username.tar.gz with all their databases, emails, email accounts and complete configuration. Copy them to a second hard disk on your server, reinstall the OS and cPanel (it takes 1,5-2 hours) and then restore the accounts. They will be restored exactly as they were before the OS reinstall. Of course notify your clients in advance about the 2 hours downtime.

Posted by Cyclon, 02-23-2013, 11:35 AM
cPanel migration will not alter custom records.

Posted by Hellsheep, 02-23-2013, 11:40 AM
Amen to that. That's my exact plan of action, except I will be moving all accounts to a second server, no downtime for them and then re-install and migrate back.

Posted by Steven, 02-23-2013, 11:45 AM
If you cannot tolerate some downtime for a proper fix, how are you going to tolerate it when some attacker walks into your server and deletes everything?

Posted by mconcierge, 02-23-2013, 02:24 PM
I became very alert to this issue and been reading all related threads since I've received an email from cPanel like many others did as well. Not being a real expert / pro (I just have a small vps hosting a couple of my websites and running WHM / cPanel ) it is still unclear to me wheter or not my machine has been compromised soit would be greatly appreciated if someone can shed some further light for me. So I've tried a couple of suggestions as pointed out in previous replies to this thread. I've tried the command: for i in `ipcs -mp | grep -v cpid | awk {'print $3'} | uniq`; do ps aux | grep $i | grep -v grep;done which results in: I've then checked the libkey file in the lib64 folder with the result: Can anybody tell me if my installation has been compromised based on these results? Some advise would be truly appreciated. Thank you very much.

Posted by Steven, 02-23-2013, 02:30 PM
Based on the output you are safe. You should run the following commmand to be sure: rpm -V keyutils-libs You should get back no output.

Posted by mconcierge, 02-23-2013, 02:40 PM
Thank you very much Mr. Ciaburri - very much appreciated and feeling quite relieved for the time being. I tried the command you've pointed out and I get no output. Are there still any chances though that I could be the next potential victim of this specific exploit?

Posted by Steven, 02-23-2013, 03:45 PM
Well, You are not showing any of the typical symptoms of it. I do have a suggestion -- ensure you reset your root password. Please read this: http://www.baekdal.com/insights/pass...rity-usability

Posted by israr, 02-23-2013, 03:52 PM
I am getting: [root@srv ~]# rpm -V keyutils-libs ....L.... /lib64/libkeyutils.so.1 [root@srv ~]# I am definitely infected. Two vps actually and for both I contacted the cpanel a month ago. Can anybody confirm if cpanel uses windows or linux at their support network? Because if it is windows and they have something malicious installed on their servers and their antivirus does not detect it, then how can we know our own network is not infected with same vulnerability... And an os reload will not again result in being infected again.

Posted by israr, 02-23-2013, 04:10 PM
Oh and thanks Steven, NetworkPanda and others for all your help!

Posted by RoyalJ, 02-23-2013, 04:31 PM
I have had 2 tickets opened with cPanel support in the last 6months. Though at the moment, it does seem that both my servers are unaffected (tried various tests mentioned on this thread, couldn't find any evidence so far). Was wondering whether anyone with direct root login disabled and using non-standard ssh port remained unaffected or was it just that I got lucky this time?

Posted by Losvre, 02-23-2013, 05:23 PM
That is the best solution for your clients

Posted by demil, 02-23-2013, 05:27 PM
And Microsoft too http://www.livescience.com/27373-mic...cs-hacked.html

Posted by Steven, 02-23-2013, 07:48 PM
There is going to be a lot more of this going on.

Posted by burloy, 02-23-2013, 09:58 PM
heh, i know about 0day in cpanel from like 6 month it will that solve all your problems ? ))

Posted by TravisT-[SSS], 02-23-2013, 10:02 PM
You mean the MU392 one?

Posted by Zimple, 02-23-2013, 10:53 PM
Yes, Managed to talk to data center and get same IP block allocated in to new server. Planning to go with minimal downtime.. Thanks for the tip.!!

Posted by FastServ, 02-24-2013, 01:11 AM
We've had several cases with Cpanel over the past 6 months and none of the servers were affected. Didn't get any notifications from Cpanel about this either. Not sure if it's a coincidence or not.

Posted by DedicatedBox, 02-24-2013, 09:09 AM
Which seems normal. There is a *possibility* of compromise, not a guarantee. I'd change passwords nontheless. The problem is that this hack comes at a time that other things are going on. People simply assume that it's the same issue OR that cPanel is the root of all evil while machines running no control panel at all or another control panel have been affected as well. It seems to be a coincidence so far, that's all. Nothing conclusive. It is rather funny to see some people take our their pitchfork at once but apparently don't have a single shred of evidence that it is related. There simply isn't 100% enough information to give anything other than theories... But theories shouldn't be sold as facts. Just one other comment: anyone who thinks cPanel, or any other control panel for that matter, makes a system secure by it's default installation should not be working in production environments nor make comments about it. :X (Plus... What is "secure" anyway... Secure is a term used to generate a false sense of security imho: you should always be vigilant. Systems are never 100% secure and even doing your outmost best to secure it can still lead to hacks thanks to 0days as an example... Stay on your guard, at all times. )

Posted by chukchuk, 02-24-2013, 12:28 PM
Am I infected Last edited by chukchuk; 02-24-2013 at 12:34 PM.

Posted by realvaluehosting, 02-24-2013, 12:32 PM
You can see if your server is infected by running: $ wget -qq -O - http://www.cloudlinux.com/sshd-hack/check.sh |/bin/bash To clean up libkeyutils library. USE IT AT YOUR OWN RISK, THE SCRIPT WASN'T FULLY TESTED $ wget -qq -O - http://www.cloudlinux.com/sshd-hack/clean.sh |/bin/bash and reboot the server. To protect against being re-infected again we recommend completely firewalling SSH from internet, allowing access only from your IP. Change your passwords for SSH, WHM and any other admin passwords you are using on that server. http://www.cloudlinux.com/blog/clnews/sshd-exploit.php

Posted by egillette, 02-24-2013, 12:34 PM
Honestly, I see where you are coming from because I support a guy who does shared hosting for his clients, and he had around 4,000 domains on his single server. I moved them all to one of my own servers that has not been compromised, reloaded his entire system, and then copied them all back. This was made somewhat easier in terms of DNS, since both servers had cPanel installed, so I setup DNS clustering, and moved his primary domain as well (where his customers where pointed to), so he and his customers experienced no downtime, and he only had to change the DNS for his primary domain to point at the server I moved him and his clients to temporarily. After I reloaded his system, he pointed it back at his server, and after we copied the accounts back, there was no downtime at all during the process (aside from perhaps the TTL on the DNS when it was pointed at the old server). You could technically do the same thing on a system without cPanel, but you'd have to copy the zone files and rebuild named.conf, so in essence, basically make yourself a DNS cluster, but you'd be doing it manually. Also, @Steven -- we initially got off on the wrong foot, but dude, I will tell you, that your undying effort to try and find a resolution is amazing. I'm with you on the infected workstation theory, but my concern is that of 122 systems, only 4 were infected, and they were systems where I was not the only one accessing them. I mean I would suspect my clients' workstations, as much as I would suspect my own workstation, but would you say it's safe to rule myself out, considering the other systems that I am the only one who has sole access were not compromised? The 4 systems that were compromised, have in common that myself and the clients who own the machines access them, while the others under my management, I am the only one who accesses them (root login is disabled, and I login with my own account, where I use sudo or su), were not infected. On my two workstations. . .one an Ubuntu machine, the other a Windows 7 machine. The Ubuntu machine doesn't have internet access (I use it internally on my network to manage an Iomega NAS and Uninterruptable Power Supply), and the Windows 7 machine which I use daily (and is the PC I'm writing this on) has Norton Internet Security (and Anti-Virus), though I suspect Norton wouldn't have caught this type of proposed keylogger anyway What do you think Steven??

Posted by chukchuk, 02-24-2013, 12:51 PM
Thanks for the link. Good thing I am not infected. I just panicked because I received 3 abuse mails today. But after checking the abuse mails it appears that the emails were sent before we ordered the server.

Posted by The3bl, 02-24-2013, 07:05 PM
Hey everyone we know Steven did some really good work and dug into this exploit and did not let go so give him a heads up and thanks. But also everyone wish him a Happy Birthday today. Happy Birthday buddy take a break and enjoy yourself.

Posted by kpmedia, 02-24-2013, 08:33 PM
Yeah, he's one of the few that I respect around here. Kudos for sure.

Posted by Steven, 02-24-2013, 08:54 PM
Thanks guys. For those interested: I have still been working on this. I have found that most boxes that were infected, have had their libraries replaced over time. So some boxes were infected for a longer period of time, than that the stat of the library has said. This one box in certain, the logs on the disk go to Nov 21st 2012, but this line is not in the logs: This is however on the disk... which hints to subtle log cleaning.. instead of them just running in and dropping the logs. Someone else I am working with on this, has found similar activity, except they also found that that IP had logged into a user on the server, and then a few days later logged into root with password. Not conclusive yet, but is promising in revealing more of the story.

Posted by Steven, 02-24-2013, 09:02 PM
Without having done an audit on the affected servers, and piecing together the puzzle I cannot really comment. There is so much that could be going on, you would need to track back to find the original infection and then work from there. If any infected server logged into any server that was not infected, that is likely how they got hit as well. The more I look at this, the more it looks like some servers have been infected for some time, and were getting updates of the lib as time went on. Last edited by Steven; 02-24-2013 at 09:14 PM.

Posted by Patrick, 02-24-2013, 09:08 PM
You guys should send him money! Steven: What's your PayPal address?

Posted by Steven, 02-24-2013, 09:22 PM

Posted by Hellsheep, 02-24-2013, 11:26 PM
I just want to thank you as well for all your hard work, also the other guys in this thread who have been giving useful information and assisting Steven in finding out what's going on. Happy birthday, and thank you.

Posted by james2398, 02-25-2013, 12:14 AM
Great work Steven, and you continue to relentlessly dig at it too. Amazing

Posted by Master Bo, 02-25-2013, 12:28 AM
Yes, happy birthday and thanks for helping quickly determine possible locations/exploited channels. Servers I am responsible for are not affected at this time, yet it cannot be enough security measures.

Posted by realvaluehosting, 02-25-2013, 12:50 AM
Thread hijacked Wish you a very happy birthday Steven

Posted by tomfrog, 02-25-2013, 03:44 AM
Happy birthday Steven! Just read the last posts.

Posted by xanubi, 02-25-2013, 09:10 AM
First, Steve did and is doing a superb job Happy Birsthday to this hero! Second, a couple of questions, if anyone could answer: 1.1) Anyone who had compromised servers, and use cloudlinux with cagefs, can tell me if they only had user root without being inside cagefs ? 1.2) If only root was not inside cagefs, did you've a support ticket in the latest 6 months with cpanel or cloudlinux? Why am i asking this? Because cloudlinux cagefs virtualize the filesystem and the libkeys, and even if a user had lose the password, if anyone logins (without being root), they only get the virtualized libkeys and limited access. So, i'm trying to figure out, if the servers with cagefs that some reported as compromised, had all users (except root), inside cagefs. Last edited by xanubi; 02-25-2013 at 09:15 AM.

Posted by iseletsk, 02-25-2013, 09:24 AM
Happy Birthday Steve! And yes, servers with CageFS with all the users inside CageFS had been compromised - which was one of the thing that forced us to believe that it wasn't privilege escalation from end user.

Posted by xanubi, 02-25-2013, 09:40 AM
Hello Igor, thank you for your clarification on this. That is really a very strong argument to say that malware infection at workstations, could really be the starting point. (and from that point, other methods can fork)

Posted by rfelsburg, 02-25-2013, 12:15 PM
Can anyone clarify some CageFS stuff for me. IS the root used getting virtualized libkeys as well? -Rob

Posted by Gavo, 02-25-2013, 01:17 PM
I have been looking at the thread since the 1st post, Seeming random servers are being rooted, different OS's and setups, there doesn't seem to be anything common server wise. I'm starting to think along the lines of an Android app or similar. Does anyone login to there servers using Android or use any SSH server apps eg. ConnectBot

Posted by colfer, 02-25-2013, 02:25 PM
Also, with a smartphone your whole email history is open if the phone is hacked. Think of the size of a Gmail archive. Same is true of any IMAP client, but smartphones often do not get OS updates, and the user cannot control that without jailbreaking. Add to that, WiFi is broken, with any security short of WPA2-Enterprise using enforced keys, due to evil twins, and you've got a bad situation. Root passwords should probably not be emailed without a second factor. I also wonder at my VPS provider asking for the root password for every support request, in the ticket form.

Posted by Zadmin, 02-25-2013, 05:51 PM
check this http://www.phoronix.com/scan.php?pag...tem&px=MTMxMTg

Posted by iseletsk, 02-25-2013, 05:56 PM
This is not related, as RHEL 5 & 6 kernels don't have this code.

Posted by Zadmin, 02-25-2013, 06:00 PM
yes , thought it might be one of the entry vectors used since what we are facing here is not the entry vector

Posted by charlie23, 02-25-2013, 09:00 PM
That is something I can not stress enough. The use of smartphones, how much personal data they store, and how users do not take security on them serious. Very good point about the smartphones

Posted by egillette, 02-25-2013, 09:44 PM
You know, it's funny you mention this. . .I have logged into servers using ConnectBot from my Samsung Galaxy SIII on T-Mobile. However. . .the servers I've logged into tested negative for the compromised library -- or any compromised libraries for that matter. So it's possible. . .but I think it might be a bit of a stretch. Not ruling it out though. . . Steven, thanks for the feedback man. . .best I can do is keep my eyes peeled. Happy belated Birthday buddy! Last edited by egillette; 02-25-2013 at 09:46 PM. Reason: Forgot to wish Steven Happy Birthday!

Posted by hafthorr, 02-26-2013, 01:18 AM
might want to ask the hoster of the CS:GO server to help out with the other one

Posted by LittleApps-Nick, 02-26-2013, 02:22 AM
I've checked all of my servers and (luckily) none of them are affected. They are all up to date and running CentOS (One is running cPanel and another is running cPanel DNS Only). I'm guessing this rootkit managed to slip in through one of the repositories.

Posted by Zimple, 02-26-2013, 06:14 AM
Could this be one of reason to get workstations effected ? http://www.adobe.com/support/securit...apsa13-02.html

Posted by afletch, 02-26-2013, 09:10 AM
One of many.

Posted by LibKeyUtils, 02-26-2013, 09:17 AM
In my server libkeyutils.so.1 removed!! my server and ssh and sites is down!! please solution for back libkeyutils.so.1 CentOS 5.8 Thanks

Posted by jzukerman, 02-26-2013, 10:16 AM
Without shell access, your options are limited. Do you have remote console/KVM access from your host? If so, you could manually download the keyutils-lib package and install it. yum install keyutils-libs Run that command via remote console and reboot. Should restore ssh access. You should then investigate why the file got deleted and if your server has been infected.

Posted by nkawit, 02-26-2013, 10:19 AM
Just look at the recent Java vulnerabilities! Mind boggling, and as you say still one of many

Posted by LibKeyUtils, 02-26-2013, 10:19 AM
yes, yum install keyutils-libs Not Running Please help me

Posted by martinger, 02-26-2013, 10:29 AM
Hello everybody, I'm new to this community. I followed this thread for a couple of days now. I found it really interesting and at a top level discussion. So I decided to join the community, tough, I can't really put something helpful into this problem. I'm running a small webdesign and hostung company in germany. We've got 2 servers running centos and plesk panel. At this moment I haven't noticed any break in attemps (rather than the normal abuse stuff). and my libkeyutils is still the original one. You talked about how to secure ssh login and I came across this aricle today, maybe you find it intressting: blog.duosecurity.com/2013/02/bypassing-googles-two-factor-authentication/ For the android and ios idea, I'm login into my servers at a daily use form a iphone and a android tablet. (I switched off root and password login and ssh server is running on a different port.) thanks guys, keep reading best regards Martin

Posted by Zimple, 02-26-2013, 10:55 AM
Java/Adobe/Flash/Chrome I have removed all those people from workstations...!

Posted by Dougy, 02-26-2013, 11:09 AM
Pay a professional to fix it or re-install your operating system.

Posted by jzukerman, 02-26-2013, 05:36 PM
I 2nd this.

Posted by patchwork, 02-26-2013, 05:41 PM
Or keep the money and figure out how to do it yourself, everyone has to start off knowing nothing, even the so called professionals. Pete

Posted by DedicatedBox, 02-26-2013, 06:14 PM
New email from cPanel.

Posted by chinook, 02-26-2013, 07:24 PM
I must admit when this thing first broke over a week ago, I too ended up deleting the libkey file and then finding out I couldn't put it back , couldn't YUM & couldn't SSH. What saved me? (I am telling the process in case anyone else can also benefit). I had another cpanel server that was the exact same version and of course now the problem was to move the file over to the damaged server. Luckily all my cpanel servers are on vmware and so I still had console access. But how to get the file over. Well, the cpanel was still working so I went into one of the test sites and the file upload was still working. So now it is on the disk but in the wrong place. A simple copy to the correct place using the console access did the trick. Implemented the soft link after that. Yum started working and then it was a simple case of yum reinstall keyutils-libs

Posted by RRWH, 02-26-2013, 08:35 PM
Great that CPanel have taken the time to respond and address the issue.... Not so great that on the published web page the commands that they show have errors in them that will potentially cause more problems for the more junior admins. Specifically, the bit about changelogs for RPM's - they show ... and this is wrong - you only need a -q and not a -qp flag.

Posted by papi, 02-26-2013, 10:35 PM
I believe someone earlier confirmed that even CloudLinux boxes with CageFS enabled have been rooted? How's this even possible when each user is jailed? Wouldn't this indicate that either: 1) it's a major services (apache or exim for example) which runs as root by default is exploited remotely and then they replace the libs etc or 2) possibly the Cloud Linux boses were compromised only because their login/passwords were stolen because someone logged onto them from an infected server?

Posted by Gogax | Simon, 02-27-2013, 01:36 AM
I knew this announcement was coming but did not want to intervene so i just waited. On manage2, they upgraded security and now request SSH keys for support which is a great thing!

Posted by TheVisitors, 02-27-2013, 02:01 AM
Formating C:\ Re-installing Wish me luck....... I know a lot of other people have done the same thing and have been re-infected. Let's hope I only need to do this 1x.

Posted by martinger, 02-27-2013, 03:17 AM
does that mean you have an running windows server which is infectet as well (because of the format c: )?

Posted by TheVisitors, 02-27-2013, 03:47 AM
I have nothing here running windows.... Not my workstation (it's a LIVE CD of Ubuntu) and not my server either (also linux). Was just generally using the term loosely

Posted by Master Bo, 02-27-2013, 03:49 AM
Who is General Failure and why he's reading my drive C:??" I wonder, whether it was necessary to reinstall OS on some infected systems? So far, as I heard and read, removing fake library and reinstalling true one was enough.

Posted by TheVisitors, 02-27-2013, 03:54 AM
Re-read this thread... Removing the library and / or reinstall does not seem to guarantee a cure. Although this will be my 1st time re-installing. I plan on IP banning ssh for everyone except for myself, as well as changing ports.

Posted by Master Bo, 02-27-2013, 04:00 AM
Sorry, no - 95 pages is way too many. OK, I re-formulate. After the malware library has been removed, are there any other changed/added files the attack creates? If they are all found and reverted/deleted, I see no reason to re-install OS.

Posted by mmoserv, 02-27-2013, 05:50 AM
Well,this thread is really interesting but we have still no idea what the weak point is. What we need is to log how they break in - we need a honeypot. When i read the posts right, a problem is that even after a fresh install the system got reinfected. What we need here is someone which got reinfected. When he got one time reinfected, there is a good chance it happens a second time to him. We need someone and his box got one time reinfected and then we need him to reinstall the box a second time but this time the box installed as honeypot with a pre installed own rootkit watching the ports. I know there a tools like that even i miss the experince with it. But i have no doubt there are people in this thread who can deal with it. We need to bring this people together and then watch here what happens. Perhaps we need more as one honeypot. That will bring us the need step up.

Posted by martinger, 02-27-2013, 06:03 AM
Well, you never know. Think about it, the attacker has root access to your server. so he can do basicly everything and install backdoors somewhere! that's why the common advice is to reinstall your server. I've got my problems with reinstall as well: What will happen if you reinstall? Are you really sure all the maleware has gone? as long as you don't copy anything back from your old installation, yes you can be sure. But the reallity looks a bit diffrent: you will at least copy back all the websites to bring than back online, so how do you know there is no maleware on one of the domains (cgi script or some sort of that).. so the only safe way is to delete everything and start from scratch but no one can do that...

Posted by TheVisitors, 02-27-2013, 11:33 AM
Removing the files is not a guaranteed cure. This is a root-kit... ie... If you're infected the hacker could have done 1,000's of things to your PC without you noticing it. This includes masking their IP as your IP or even localhost. Format is the only sure way to know nothing is there. However at this time, formating may not exactly prevent re-infection.

Posted by TheVisitors, 02-27-2013, 11:39 AM
Scan everything is about the only advise. I'm one of those who is not 100% convinced that it's malware. Having used a LIVE CD (Linux) with no physical hard drive and going no where else except my web site. However I've not 100% ruled it out either.... My host's pre-made images at the time could have had this tucked away and it could have been "them" who had such installed on their workstations when they originally made those "images". I informed my web hosting service provider and I believe they got everything cleaned on their end. I'm in the process now of setting up my VPS and so far, so good... No infections (hacking). Of course the first thing I did was restrict IP access to only trusted IP's for ssh. And I again changed the IP port (something I always do).

Posted by hydn, 02-28-2013, 08:20 AM
1 By far the most effective way to protect against infection is to firewall SSHD except for your ip. Also look out for official package updates that seem to be on there way. Source: http://stacklinux.com/discussion/63/...ct-your-server

Posted by mmoserv, 02-28-2013, 08:41 AM
This kind of "tricks" will like others (moving the ssh port for example) fail at some point with a 100% chance. If the break in szenario is like described, done by workstation or someone with previous root access, then the first attacking login will automatically come from a "good" IP. After that, your box is hacked. When the rootkit is installed, you are done with any trick. If the attack comes over a buffer overflow somewhere, you get root on the box without log in as root. For exim and such you can't firewall ips for logical reasons. And then again - after a rootkit is sneaked in, firewall your ssh is senseless. That it will work in this case perhaps is simple: The hacker did not cared for this attack.

Posted by grayloon, 02-28-2013, 05:10 PM
I have cPanel sitting on a CentOS 5.9 server. I found the following suspicious openssh packages. However, I haven't found any other evidence of this rootkit based on cPanels instructions. Iptables is configured to allow ssh for only specific IP addresses. Should I build a whole new server and move my accounts/websites?

Posted by colfer, 02-28-2013, 09:12 PM
AFAIK, users gave cPanel tech support their WHM passwords. One cPanel tech support proxy machine was broken, and in those cases, the hackers got the credentials. CPanel now has a plan for tech support to use one-time keys to access user's WHM accounts whenever possible. That seems to answer the question of how it happened completely. Right? See: http://forums.cpanel.net/f185/sshd-r...ml#post1334581

Posted by martinger, 03-01-2013, 04:02 AM
not really, because some rooted server doesn't use cpanel!

Posted by Steven, 03-01-2013, 04:05 AM
Many of the non cpanel rooted servers, had their servers logged into via compromised cpanel servers. In this case the non rooted server had their login details relayed to the attackers.

Posted by mmoserv, 03-01-2013, 05:34 AM
Yes, i see this - but thats not the point. We can guess as long we want - we need logs. We need a proofed infection by logs, and better more as one to see how they was able to break in. The cpanel thingy is pretty much possible and with a chance the only source but ATM its only a guess. Thats not enough.

Posted by Steven, 03-01-2013, 12:44 PM
I stated a few posts up, that there are logins that are not in the logs (likely due to cleaning), but on the disk itself pulled out with strings. Direct root password logins.

Posted by brianoz, 03-01-2013, 09:45 PM
Just a thought that might be worth making as it seems many are busily looking for a single point of entry - I know the cPanel support compromise is a possibility for the source of the initial exploit. However, in addition to this possibility (and cPanel seems to be saying it didn't happen) it's also possible that they harvested root passwords via a variety of methods and used those to install the ssh backdoor. So, not one single point, but a collection of various password harvesting inputs ...

Posted by alons, 03-02-2013, 02:50 AM
From my experience, its definitely not just cPanel. Some of our backup servers which had our control panel webuzo installed were also affected by this. Also our test servers which we use for various testing purposes were affected. Some were Plesk, cPanel, Webuzo. HOWEVER, all of our servers which had different SSH ports were unaffected. So we do recommend changing the SSH ports if your server is not affected or after you have cleaned it. Also one common thing between these servers was APACHE (I am not claiming to have found anything, its just a passing thought). None of our servers with regular SSH ports and NGINX (OR NO WEB SERVER) were affected. The libkeyutils is used by many and Apache does use it as well. On the basis of observing several servers, the ones with Apache and regular SSH ports of 22 were affected and the ones with either different SSH ports or with no webserver or with NGINX were not affected.

Posted by suhailc, 03-02-2013, 10:00 AM
So having read http://docs.cpanel.net/twiki/bin/vie...ion/CompSystem - it doesn't explain how/where the dodgey files came from. Does anyone know if the trojaned SSH binaries were sitting on OS download repos mirrors and unknowingly downloaded via OS updates?

Posted by Steven, 03-02-2013, 03:30 PM
Curious: Does virtualizer still run php as root via suphp?

Posted by Steven, 03-02-2013, 03:33 PM
As far as I could see no, they would have to be signed anyway for yum to install them if it came from an official repo.

Posted by Patrick, 03-02-2013, 04:32 PM
They most likely came from compromised work stations that allowed the attacker to log into the server and download the files, except that one guy from this thread who claims to boot into Live CD every single time he logs into his server. He is clearly the only victim of a zero day OpenSSH exploit... LOL.

Posted by epretorious, 03-02-2013, 08:29 PM
Except for the thread "Backdoor imitating ssh on RH/Centos boxes" on the ArchLinux forum, there don't seem to be any US-CERT bulletins or CERT advistories documenting this issue: There's been a lot of chatter about this issue but has there been any progress resolving this issue? e.g., Can anyone confirm that this issue is exclusive to hosts using the Exim MTA? Eric Pretorious Truckee, CA

Posted by Patrick, 03-02-2013, 08:34 PM
Compromises on servers not running Exim have been confirmed. Also, the Exim that was bundled with cPanel was apparently not vulnerable to that remote DKIM exploit so there is an extremely high probability that these attacks are unrelated to Exim in regards to the point of entry.

Posted by brianoz, 03-03-2013, 04:24 AM
Yes, but could those compromises have been via stolen credentials/root login? Back to the thesis that these guys are sophisticated enough to use a variety of methods to get to root.

Posted by mmoserv, 03-03-2013, 07:32 AM
Again: All this guesses are senseless at some point. Only a monitored honeypot will gives us at this point the information we want. Or we accept that the current linux versions are insecure.

Posted by epretorious, 03-03-2013, 07:49 PM
Brian: You've taken Patrick's remark out of context and, therefore, your reasoning is incorrect: This proves nothing. Those compromises could have come from any number of vectors. All of this begs the question, however: Why hasn't there been a CERT bulletin or a CVE advisory? Eric Pretorious Truckee, CA

Posted by epretorious, 03-03-2013, 08:03 PM
Agreed. This thread appears to be almost exclusively made up of idle speculation and hand-wringing. I beg to differ. Linux itself is not likely the weakness, but rather one of the software packages included in the installation (e.g., PHP) and the contol panel's security posture (e.g., compatability with SELinux and/or AppArmor). Perhaps it would be more correct to accept the trade-off between convenience and security. Eric Pretorious Truckee, CA

Posted by ZeroCool7, 03-03-2013, 10:45 PM
Thankfully not a single one of our boxes have been compromised. We block port 22 and only give SSH access to particular IP Address. Password authorization is disabled because we use keys only.

Posted by mmoserv, 03-04-2013, 06:46 AM
I know, i know My sentence was a clear teaser, of course. The big problem here is the root access by cpanel. At that point security is breached, unix security or not. Even more when you work with without one time logins and such. I for example run debian in minimal install, nginx only and dovecot/postfix minimal. SSH keys only access and even other ports. Accept my box with svn because it can't tunnel a port so far i know. The good point i such small system is, that you do not need AppArmor and othe higher level security. Because there is nothing high level to protect.

Posted by tomfrog, 03-04-2013, 09:41 AM
You're late. Gone with the wind, this thread's history. A memory of the past... But I guess you're still in time to add a few idle speculations of your own... Enjoy...

Posted by brianoz, 03-04-2013, 05:25 PM
Eric, My comment was not at all trying to prove anything, merely making the point that I suspected these guys are collating results from a variety of intrusion methods - Exim may of course be one of them, as may password theft, as may old kernel vulnerabilities, etc. All attempts to find a single common entry vetcor to this point have failed, so that seems a reasonable conclusion.

Posted by ducnanbbd, 03-09-2013, 03:07 AM
I came across this subject on your web site trying to find out more about this issue. I use Webmin and noticed that I had enabled "Allow RSA (SSH 1) authentication?" now I recall from many years ago there was some issue with SSH 1 am I right in that ? I have disabled SSH1 and allowed only SSH2 for SSH Authentication on my server. not sure if its an issue but wondering if anybody knows about it ? I do see that SSH 1 has a weaker encryption and possible attack mechansim. regards brian

Posted by TheVisitors, 03-09-2013, 08:09 AM
You're always better off using better encryption if you can do so. No harm in making things more secure (normally). SSH2 uses a bit more cpu and ram to code and decode, but the difference is so small that its almost not worthing talking about... ie... No worries. SSH2 will not prevent this root kit as there are those who have used it and still got infected.

Posted by ducnanbbd, 03-09-2013, 09:48 PM
ok, thanks. I maybe thought SSH1 might be a possible backdoor (not necessarily this) I have some issues with spam through my serve but it doesn't appear to be this libkey issue Brian

Posted by Webrything, 03-11-2013, 10:02 PM
Is there a definitive answer as yet on how this gets in? I am told a patch has been released which suggests it is now known, but could not find any mention of it here amongst the 97 pages of discussion. Can anyone authoritatively summarise?

Posted by luigidelgado, 03-12-2013, 01:50 AM
Well after reading some threads and this +97 pages blog I understand this: - Infection began mid february - 28 february cPanel announces compromised server. - Looks like a coincidence that lots of servers are cPanel driven. - I understand lots of non cPanel servers have also been compromised. - Libraries affected are most commonly on CloudLinux -Its very interesting that most of the servers had an opened ticket in cPanel Support - No ideas how to solve this or aovid this - No new contamination has been reported as far as I can see... My conclusion (if I understand this and sorry but I need to know if I got this right): - The most probable situation was that the compromised cPanel Proxy server was compromised and used to access the server to place rootkits. Thats it. From there the spread began. I got half rooted... my server runs behind a hw firewall with non standard SSH ports which only cPanel staff, DC and me can access. The server is infected but has never been SSH by any unauthorized person. I woulddeduct this is becouse I have the external firewall... and also this way there is no way that anyone could have used SSH nor WHM to access the server.... but an authorized server (cPanel). I see no other choices than to point them... Or why did they changed their support system so drastically? Its not a matter of proof, its a matter of probability. I think the true story will never go out.

Posted by lothos, 03-12-2013, 10:09 PM
Servers not running cpanel were reportedly hacked as well. Half of the story is out already, I think the rest will get out eventually.

Posted by Cryostasis, 03-13-2013, 12:16 AM
cPanel had access to all the servers of mine which got hit. Every single one had a ticket with cPanel support recently opened for it. Is the only way to sort the issue to reinstall the server still? Or has anyone been working on something to circumvent the need to do this?

Posted by TheVisitors, 03-13-2013, 04:37 AM
Let me try to answer this in philosophical way.... The problem about this hack is the hacker becomes the root user. As root, you're GOD. And just like GOD, you can remake the world (server) in your imagine. Despite how advance science, even as far as we are today we do not know everything in the universe. And it is ever so likely that no matter how long we live, no matter how far we grow, and how much we learn overtime... We'll never know everything. In other words.... There is no way for you to know everything "God" has done, is doing, or will do to your "world" (server). LMAO .... This from an atheist. Sorry.... I had to poke fun at this. Its been asked so many times..... The answer is, NO. You must format and restore your sites, because there is no way of know what else could be there. We could cure X tomorrow and discover W, Y, Z, 1, 2, 3 ...ect... Was also added.

Posted by Cryostasis, 03-13-2013, 06:33 AM
Brilliant answer! Made me laugh! Yeah I suspected as much -- just wanted to double check nothing had changed in the last couple of weeks before going into this. Thanks :-)

Posted by jalapeno55, 03-18-2013, 05:40 PM
Brad, how did you find they were using XOR 81h encoding?

Posted by martin33, 03-21-2013, 11:47 PM
Hi, cPanel, at first, was saying this was not related to their software... ...this was correct. ...this was related to the fact one of their support tech was having a trojan on he's computer. We got 2 servers infected by this trojan, right after we opened a ticket. We submitted another ticket to cPanel, wondering what was this new libkey file, and 2 days later, they sent an email to all their customers saying since the last 4 months, everyone who were requesting support and were helped by some specific agents were infected by this trojan. See this for more infos : http://cpanel.net/cpanel-inc-announc...-enhancements/ ...all you need (and can) do in regards to this is transfer all your files to a new server, and change all your passwords. We did this and no longer have any problem.

Posted by martin33, 03-21-2013, 11:49 PM
just think twice before you provide ssh access to someone else on your server, and you will avoid such problems ...we no longer outsource support since that time, and request email only support.

Posted by o-dog, 03-23-2013, 04:50 PM
cPanel were working on a ticket and they (and me) were supprised one of the servers was brute forcing the DNS Only server (and locking itself out). This was back in October/November!! Seems it has been rolling around for a very long time. In addition to not giving root passwords to vendors over the internet *doh*, and aside from SSH keys, different SSH ports, CSF+LFD, is there anything else that can reduce attack surface and reduce chances of being rooted again?

Posted by TQ Mark, 03-23-2013, 05:24 PM
Review your firewall rules closely. Only allow incoming traffic on your firewall to ports that you need to have open to the public. Restrict connections to your SSH port to only authorized source IPs.

Posted by Hamza, 03-31-2013, 08:34 AM
So long story short! there is no real solution whatsoever for this problem? We ignore how it got there, We ignore how to get *effeciently* rid of it, and worst! even if we opt for an Os reload, we may get reinfected! That's like the killed with a spoon video. The funny part is when you contacted cPanel, they say we can't do anything on your server as it's compromised, when we follow their checkyourserver thing, the server doesn't appear to be compromised whatsoever. Although, cPanel may be the ultimate cause for this injection in first place. It's like you got a food poisoning in a restaurant, and when you go back to the same restaurant, they won't serve food to you because you are already *infected*. CloudLinux was kind enough to have a closer look themselves at this, and they figured out that the server is not compromised. Today the hackers are sending out spam from the servers, what if they decide to do something else with it! Hamza

Posted by Patrick, 03-31-2013, 09:15 AM
Hamza, the general consensus among the experts of this forum is that it was most likely a localized PC infection that resulted in the compromises. That's how cPanel was infected. There is no reason to suspect a zero day exploit in any of the services right now. If someone has as server that was 100% for sure compromised, the best advise would be to reinstall all workstations that access the server, reinstall the server and make sure Java is disabled as that is the most likely culprit that was exploited. Some people in this thread initially said that was a stupid theory about localized PC infections and when they finally did a virus scan of their PC they found some stuff related to backdoors known for stealing credentials and setting up VNC like backdoors. A few other suggestions wouldn't be to disable password authentication and restrict SSH to particular IP ranges, if possible.

Posted by simmer14, 05-07-2013, 02:10 PM
right noq cent os 5 and cent os 6 both 32 and 64 bit are affected.. i'm under attack of this fujing rootkit.. debian 6 is secured by this rootkit.. once i install cent os my password gets hacked within 3 minutes aprox..

Posted by Steven, 05-07-2013, 06:22 PM
Are you using the same password for every install? What os is your workstation?

Posted by simmer14, 05-07-2013, 11:30 PM
i use very complex alpha numeric passwords.. diffrent on every install.. Cent OS 6 and 5 64bit Debian 6 is absolutely fine.. not even a single attempt as failed login.. as i mentioned in my other thread that these attacks are automated and are working on a server.. and attack comes from diffrent machines and hosts... i even saw failed attempts from a kimsufi.. i guess once ur box or slice is rooted then it is used to attack on others to root them.. right now the way to survive is debain 6 for me..

Posted by Steven, 05-07-2013, 11:34 PM
In theory its possible for remnants to remain resident in memory. Have you tried pulling the power completely to the machine prior to reload?

Posted by WKDedi, 05-07-2013, 11:59 PM
This video shows how long RAM keeps its contents after power is cut. It isn't about viruses but disk encryption key stealing. http://www.youtube.com/watch?v=JDaicPIgn9U it is relevant in that it shows you that RAM under normal conditions keeps it memory in tact for almost 2 minutes.

Posted by simmer14, 05-08-2013, 05:55 AM
yes i did shutdown the vps for hours bt it still got rooted when i reinstall cent os 6 on it.. i'm on debain 6 now btw..

Posted by Arnie21, 05-08-2013, 09:52 AM
Is it OpenVZ vps? or KVM/Xen? Maybe the images used by your provider isn't clean. Can you tell us your vps provider?

Posted by simmer14, 05-08-2013, 10:48 AM
yes arnie it is an OpenVZ vps.. and i notified my vps provider they did not said anything about the infected iso's i did told them to check the hash of the files and compare them.. WKDedi already told me to do this..

Posted by simmer14, 05-11-2013, 01:59 PM
i could not read all the 98 pages and i do not want to bump the thread but i would like to know does readhat security team knows about this rootkit are they aware about this exploit ? http://securityblog.redhat.com/

Posted by simmer14, 05-11-2013, 02:40 PM
i can not edit my post but what do you guys think about this ? https://rhn.redhat.com/errata/RHSA-2013-0519.html

Posted by simmer14, 05-12-2013, 05:41 AM
the attack on my vps is coming from this ip and password got changed to to chauthtok by pam_unix anyone intrested in talking a look at my vps ? this was a new vps which just got rooted again. [root@server1 ~]# lastb guestgue ssh:notty Sun May 12 05:11 - 05:11 (00:00) guestgue ssh:notty Sun May 12 05:11 - 05:11 (00:00) guestgue ssh:notty Sun May 12 05:11 - 05:11 (00:00) on 4th attempt he/it logged in.. http://us.hive.sshhoneypot.com/iplog...p= Last edited by simmer14; 05-12-2013 at 05:46 AM.

Posted by Steven, 05-12-2013, 12:33 PM
We discussed this earlier in the thread. It only affects Redhat 6 / Centos 6 and its not enabled by default:

Posted by brianoz, 05-14-2013, 02:18 AM
One important point worth making: No evidence has been found that there was actually an SSH vulnerability. Rather, the system itself was hacked through other root-level vulnerabilities, and as a result of that hack, a backdoor added to SSH. Again, there is no evidence that this hack was accomplished through a weakness in SSH.

Posted by simmer14, 05-27-2013, 12:58 PM
no update ?

Posted by simmer14, 06-17-2013, 09:10 PM
This is an email I just received from SolusLabs. PLEASE READ THIS INFORMATION CAREFULLY. THIS INFORMATION IS RELEVANT TO ALL VERSION OF SOLUSVM, INCLUDING BETA VERSIONS. In the last few hours a security exploit has been found. This email is to inform you of a temporary fix to eliminate this exploit whilst the issue is patched and transferred to our file servers for release. Instructions: You will need root SSH access to your master server. You are then required to delete the following file: /usr/local/solusvm/www/centralbackup.php Example: rm -f /usr/local/solusvm/www/centralbackup.php Once the file is deleted the exploit can no longer be used. This file only exists on the master server and the slaves will not be affected. You will receive a follow-up email once the patch versions are available. Regards, Soluslabs Security Team could this be the exploit used here ?

Posted by Steven, 06-18-2013, 12:03 AM
It could sure, but not likely the cause of this problem.

Posted by Magiobiwan, 06-18-2013, 12:20 AM
No. The most recent SolusVM exploit the email was about involved the centralbackup.php file in SolusVM. It has a vulnerability in it that caused massive security issues. RamNode was exploited by it. If you run SolusVM UPDATE IMMEDIATELY!!! That exploit is VERY DANGEROUS.

Posted by mmoserv, 06-18-2013, 02:54 PM
A description of the exploit is here: http://localhost.re/p/solusvm-11303-vulnerabilities Incredible that such a code quality is part of anything people put on their server. This is even not amateur standard. Its just plain wrong. Its also not a php issue, because they forwarded the unfiltered web server data from the calling connection to mysql.

Posted by prgs1971, 08-08-2013, 09:54 AM
Hello to all users I find accidentally this topic and read some pages of it and i get very worried about this rootkit exploit. I am a newbie in Server Admin and i want to see if have this rootkit in my 2 Linode VPS with Centos. How can i check if i have this rootkit?

Posted by BeZazz, 08-08-2013, 09:59 AM
You will have to read the thread to find more answers to your question. One way posted was http://www.cloudlinux.com/blog/clnews/sshd-exploit.php

Posted by prgs1971, 08-08-2013, 10:09 AM
BeZazz This topic have 98 pages and i have read about 10 of it with people discussing that i am better server admin than the other... take a break!!! This not seems the final solution and i don't have cloudlinux installed. If anyone can point me the final solution for this problem i will appreciate very much

Posted by BeZazz, 08-08-2013, 10:12 AM
I am sure that tester will work, especially if you are running CentOS.

Posted by prgs1971, 08-08-2013, 01:41 PM
I have tested it and return I hope that is really clean...

Posted by Steven, 08-08-2013, 02:13 PM
You need to check and make sure you do not have compromised openssh packages too. That was an older variant of this problem.

Posted by prgs1971, 08-08-2013, 02:29 PM
Steven How can check openssh packages? Remember that i am a newbie

Posted by gopkris2000, 09-05-2013, 03:01 AM
I have manually removed libkeyutils from lib directory and rebooted the server. rm libkeyutils.so.1.9 rm libkeyutils.so.1

Posted by martin33, 09-06-2013, 01:53 AM
Yup : i saw this on 2 servers. This was a gift from cPanel support. You should reinstall asap if you see this file : no other ways to make sure the problem is really gone if you just delete it. You should also verify every linux boxes that connected to the infected host : it's easy to login, and get the trojan in your own computer! That's the reason why their support area is now different : http://cpanel.net/cpanel-inc-announc...-enhancements/

Posted by martin33, 09-06-2013, 02:09 AM
Linux is not the same as Windows, and there are no way's to remove such trojan other than a complete reinstall of the system. You should definitely REINSTALL on EVERY servers where you see this problem. That's the ONLY real solution. Also : if you are using Linux on your own computer, don't forget to check it! There may be other (new or existing) files infected / corrupted . Don't take any chances. Follow theses instructions to check if you are affected or not : http://docs.cpanel.net/twiki/bin/vie...ion/CompSystem Here is the email we got from cPanel (i am keeping it in souvenir . All instructions regarding what to do in case if you get infected by this (and how to confirm are there). Last edited by martin33; 09-06-2013 at 02:12 AM.

Posted by Steven, 01-31-2014, 10:45 PM
Ebury has been kicking around for a while with a new technique of hiding itself now, this time installing modified libkeyutils rpms that are NOT signed. rpm -qi keyutils-libs (these examples are 64bit centos 6) will report: This is what it should say: Note the increased size of the package and the missing signature. Not infected: Infected: Last edited by Steven; 01-31-2014 at 10:55 PM.

Posted by Steven, 01-31-2014, 11:05 PM
If anyone finds yourself infected with the new variant, I would love copies of your libkeyutils.so libraries.

Posted by JPerkster, 01-31-2014, 11:52 PM
Thanks for the heads up

Posted by alons, 02-01-2014, 01:22 AM
Is there no report on how servers are getting infected ? I know for sure that SSH with port 22 are infected. If you change the port you dont get infected.

Posted by Vex76, 02-01-2014, 01:36 AM
I haven't seen the new version yet, will keep an eye on this topic if someone manages to detect it.

Posted by reto4, 02-01-2014, 06:34 AM
Hi alons, that is really really interesting to hear! we have multiple servers which are infected with the new variant, one of that is: -CentOS 6.4 (installed on Nov, 2013) -It has a long completely different root password -I've connected only from a secured host to this server -I've never logged in since december -On the server is only the following application installed: Subversion Edge (from CollabNet) -Open ports from outside: Port 22 and Port 4434 (Subversion) You can see, there is no signature on keyutils-libs: [root@*** ~]# rpm -q -i keyutils-libs Name : keyutils-libs Relocations: (not relocatable) Version : 1.4 Vendor: CentOS Release : 4.el6 Build Date: Fri Jun 22 08:20:38 2012 Install Date: Tue Dec 10 08:36:04 2013 Build Host: c6b10.bsys.dev.centos.org Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm Size : 59336 License: GPLv2+ and LGPLv2+ Signature : (none) I'm pretty confused about that, from my view, Ebury can only come from: -SSHD Exploit -Installation Repository was infected Because no other service is running on the machine than Subversion Edge, but its on a really high port and it is quite unlikely that they have exploited this service. Btw. interesting thing on CloudLinux which is infected with the new variant: The keyutils-libs has the vendor "CentOS" instead of "CloudLinux"! Best regards Reto

Posted by alons, 02-01-2014, 07:29 AM
That is what I said. Only servers with SSH ports set to 22 are getting affected. I had two VPS one with port 22 and the other with port 3910 and nothing got infected on the one with port 3910. Similar thing happened on another server as well. Change your ports to some random big number for safety !

Posted by Steven, 02-01-2014, 09:09 AM
It is not a sshd exploit and it never was. Remember people get infected with this with firewalled off ssh. It is not a compromised repo. There would be other signs, such as a newer version number most likely. You logged into the server from a compromised box.. The design of this rootkit is that it is self infecting. WHM transfers, ssh binary, scp binary all are infected with this so if you do a WHM transfer from an infected machine the rootkit obtains your login details. Any server software that has libkeyutils loaded (which is quite a few) all are vulnerable to having details logged in shared memory. So despite you saying that machine you logged into it form being secure, it is not secure if thats the only server you logged into it from. Not really all that interesting, they are installing a premade RPM that was made on CentOS. Even on a fresh Cloudlinux machine you would see CentOS if cloudlinux didn't upgrade it. Cloudlinux doesn't replace all your RPM's on install unless you use the ISO to install the system. Last edited by Steven; 02-01-2014 at 09:15 AM.

Posted by Steven, 02-01-2014, 09:10 AM
Changed ssh ports are not immune. The first servers I have seen infected with Ebury were non standard ports. The problem with this rootkit is, unless you have a 100% clean network, if you login to boxes from other boxes it will spread and reinfect. None of this is new, its still all the same just a different install method. Remember originally it was openssh binaries, then it was libkeyutils, then it was actual openssh rpms, and now it is libkeyutils rpms. It still operates the same as it did before. Last edited by Steven; 02-01-2014 at 09:13 AM.

Posted by pleiades, 02-02-2014, 12:48 AM
What is the safe way to migrate cPanel accounts from a infected server to a new clean server?

Posted by Steven, 02-02-2014, 12:41 PM
scripts/pkgacct username place somewhere web accessible and wget them You could do a script like: To package them all. You can then put them somewhere web accessible and wget them them all to the new server. Then run a similar bash script to restore them. This eliminates the logging in action over ssh.

Posted by pleiades, 02-02-2014, 07:03 PM
Hello Steven, thanks! And about OpenVZ?

Posted by reto4, 02-03-2014, 06:39 AM
Hi Steven Yes, I fully agree with you! This is definitely possible. Best regards Reto

Posted by Patrick, 03-18-2014, 11:02 AM
http://www.theregister.co.uk/2014/03...o_unix_botnet/ ... a bit late. Steven was pretty much the first guy to bring attention to this matter in early 2013.

Posted by ttgt, 03-18-2014, 08:15 PM
Hi, I do a test on my server,does the result mean it is safe ? root@server [~]# rpm -q -i keyutils-libs Name : keyutils-libs Relocations: (not relocatable) Version : 1.2 Vendor: CentOS Release : 1.el5 Build Date: Sat 06 Jan 2007 03:57:58 PM CST Install Date: Mon 15 Aug 2011 12:54:43 PM CST Build Host: builder1.centos.org Group : System Environment/Base Source RPM: keyutils-1.2-1.el5.src.rpm Size : 32836 License: GPL/LGPL Signature : DSA/SHA1, Wed 04 Apr 2007 08:24:17 AM CST, Key ID a8a447dce8562897 URL : http://people.redhat.com/~dhowells/keyutils/ Summary : Key utilities library Description : This package provides a wrapper library for the key management facility system calls. Name : keyutils-libs Relocations: (not relocatable) Version : 1.2 Vendor: CentOS Release : 1.el5 Build Date: Sat 06 Jan 2007 03:55:58 PM CST Install Date: Mon 15 Aug 2011 12:54:21 PM CST Build Host: builder5.centos.org Group : System Environment/Base Source RPM: keyutils-1.2-1.el5.src.rpm Size : 33608 License: GPL/LGPL Signature : DSA/SHA1, Wed 04 Apr 2007 08:24:45 AM CST, Key ID a8a447dce8562897 URL : http://people.redhat.com/~dhowells/keyutils/ Summary : Key utilities library Description : This package provides a wrapper library for the key management facility system calls.

Posted by NetworkPanda, 03-18-2014, 08:15 PM
One year later, better late than never... I suppose Symantec/Norton Antivirus will issue a notification about 5 or 6 years later (always the last one, to detect the malware when they already have caused damage). I remember that AVG for Linux (even the free version) was the first one that was able to detect this rootkit from the first 3 days it started circulating.

Posted by egillette, 03-18-2014, 09:01 PM
Try this my friend:

Posted by Patrick, 03-18-2014, 09:30 PM
Except they didn't give proper credit to Steven for all of his hard work and based on our communication with ESET regarding this matter... what a bunch of douche bags. Needless to say, someone has won a free audit in the future for all of their products.

Posted by FLDataTeK, 03-18-2014, 09:37 PM
It was posted over here also http://arstechnica.com/security/2014...-and-exploits/

Posted by NetworkPanda, 03-18-2014, 09:44 PM
Too bad practice, big companies stealing your work without any credits. Interested to see the findings of the audit of their products!

Posted by Steven, 03-18-2014, 09:51 PM
I think what upsets me the most was, I worked directly with one of the people involved with their 'joint' effort and gave them access to a compromised box and everything. We got a footnote to this thread, not even a mention of rack911. We basically were the first people to talk about the keysutil varient pubically. They instead marked cpanels compromise as the first event in 2013 in their whitepaper. The next time something like this comes around, I'll just leave the info I have to my self. Last edited by Steven; 03-18-2014 at 09:55 PM.

Posted by ttgt, 03-18-2014, 10:43 PM
if my server get infected and i need to move the sites to other server. as i know,i can not use whm's backup feature directly. but i can use /scripts/pkgacct to backup each account and use wget to transfer the accounts to other server and restore,it will be safe.correct ?

Posted by Steven, 03-18-2014, 10:56 PM
Yes that is safe. I would also scan your workstation, this malware is distributing password stealing malware to desktops.

Posted by ttgt, 03-18-2014, 11:51 PM
do you recommend any software to do the scan job ? thanx

Posted by cyberhouse, 03-18-2014, 11:59 PM
anyone see this infection on freebsd ? or is this a linux only problem ?

Posted by Steven, 03-19-2014, 12:19 AM
I have not come across a freebsd server with this personally HOWEVER, they can infect your openssh binaries without too much issue.. its cross platform.

Posted by unSpawn, 03-21-2014, 02:26 PM
I understand how you feel about it but this: would never do, right?..

Posted by Addora, 03-21-2014, 02:42 PM
No got it on centos though. The infection is related to any linux platform.

Posted by Steven, 03-21-2014, 02:50 PM
I definitely would. Why would I help provide information to people so they can take the benefit and kick the little guys like my self to the curb with no recognition? They can figure it out their self. I provided boxes for the people involved in the group that worked with ESET to release a report, and they didn't even have the decency to credit me, **** them. If you read their write up PDF, in their timeline they listed cPanel as the first event in 2013, which it wasn't... cPanel's compromise was announced weeks after I started talking about this publicly. They even included Steinar H. Gunderson which was the first discussion of the openssh variant.. To top it off, eset stopped taking to me about it, and is basically ignoring me. And then you have Leif Nixon that said I stopped working with them because I didn't respond to his emails (which I didn't get). He didn't try very hard to get a hold of me, I mean after all he was using my personal non-company email address. I have seen early variants of alot of highly publicized malware being in the industry I am in. I only even brought this one to light because it bothered me. I'll keep things to my self like I have done in the past from now on. Last edited by Steven; 03-21-2014 at 03:02 PM.

Posted by sam007, 05-02-2014, 04:29 AM
What is the best software one should use for scanning in such cases?

Posted by Steven, 05-13-2014, 11:40 AM
Eric, Please take down your script. Someone pm'd me about this. https://www.ericgillette.com/clients/exploit-cleanup Google cache mirror: http://webcache.googleusercontent.co...&ct=clnk&gl=us There is a malicious command that hopefully you did not intentionally place in there. http://puu.sh/8KKWV/63926ea538.png Last edited by Steven; 05-13-2014 at 11:44 AM.

Posted by HSN-Saman, 05-13-2014, 12:11 PM
echo "exploit found and whole system erased. go home"

Posted by JakeMS, 05-13-2014, 12:30 PM
Hi, Normally, I would read the thread to get an answer, but with 100 pages... maybe not so :-P. Just one quick question, has anyone found how this issue was being exploited, and if so, how to prevent it? Additionally, does this still effect fully up-to-date servers?

Posted by Steven, 05-13-2014, 12:38 PM
It typically was caused by leaked login details. Does not matter if port is changed to ssh.

Posted by NetworkPanda, 05-13-2014, 12:47 PM
Hopefully not many people will run this script. When we said that an infected server should be formatted and reinstalled we didn't actually mean that it should be done with an rm -rf / command inside a cleanup script, without the user knowing and having created any backups.

Posted by AcheronMedia-VK, 05-13-2014, 12:49 PM
Oh lol, that's hilarious. Thankfully, the rm command in most modern distros have --preserve-root as default so that command wouldn't work in them. Edit: by all means the malicious command should be removed from that script, I'm not implying it wasn't malicious. Last edited by AcheronMedia-VK; 05-13-2014 at 12:51 PM. Reason: Disclaimer

Posted by JakeMS, 05-13-2014, 01:12 PM
Thank you for your response. This probably won't effect us then :-). (We don't use passwords)

Posted by Steven, 05-13-2014, 01:22 PM
If you use WHM and have it publicly accessible, it could potentially affect you. There is several ways work around the password restriction including restarting openssh in a safe/default configuration with autofixer. If not probably safe.

Posted by JakeMS, 05-13-2014, 01:30 PM
Hi, The only method of control of the servers (server side configuration wise) is through SSH. There are no control panels installed . Bare in mind, these are company servers, so they are not used for selling hosting or otherwise so there is no reason to have any control panels on them.

Posted by Steven, 05-13-2014, 01:31 PM
You should be fine then, I was only stating WHM because I do not want people to get a false sense of security that if they have passwords disabled they are safe after they read your prior post.

Posted by JakeMS, 05-13-2014, 01:35 PM
Ok no problem . Thanks again.

Posted by egillette, 05-13-2014, 02:01 PM
Steven, Nope, I didn't stick that in there -- in fact that server was compromised. I'm in the process of rebuilding it as we speak. Not just for this thing, but also for the heartbleed issue that was found a bit ago as well (my SSL cert may have been compromised as well so I need to re-issue). Thanks for identifying that buddy. :-) I removed the script as well.

Posted by martin33, 11-06-2014, 04:07 AM
Hi, Found back this thread accidentally. Wow : it's still active We've been infected on 2 servers there is 2-3 years by this **it. cPanel support proxy infected us. ...we've been told by cPanel a couple of times there was no sure ways to remove this malware. I would perform a complete reinstall even if there is a "removal" tool. I'm surprised nobody has patched the security hole that allowed this file to get there yet, after all this time! ...or it's patched and i don't know? I remember we were one of the firsts customers who notified this problem to cPanel. The day after, cPanel confirmed the security issue by email, to all their customers. Last edited by martin33; 11-06-2014 at 04:13 AM.

Posted by martin33, 11-06-2014, 04:16 AM
Use CloudLinux ...and pray if you provide your ssh credentials to a third party I heard the Grsecurity Kernel is not vulnerable to this. 1h.com products are vulnerables, since they protect against barely nothing and only provide very old binaries. We got infected while using them. You need to protect the kernel first. Best option to go is CloudLinux on cPanel IMHO. I did not tried BetterLinux, but i'm not sure it would be benefical for this kind of thing. Seems like it's working pretty much like 1h products. Last edited by martin33; 11-06-2014 at 04:21 AM.

Was this answer helpful?

Add to Favourites Add to Favourites    Print this Article Print this Article

Also Read
managed.com down ? (Views: 288)