Finding outbound access

Portal Home > Knowledgebase > Articles Database > Finding outbound access

Posted by newbie_player, 04-14-2016, 09:30 AM
Helo everyone, I need a help here. I got report from bitninja.io that my vps is accessing (or they said attacking) a website's sitemap.xml. I have the domain and the ip of the victim (who reported that my server is attacking them). Lets say the victim's domain is victim.com and the ip is 22.22.22.22 for example. Im using the vps for hosting around 50 customers inside it with cpanel. How do i find which domain or which of my customer's websites or files that accessing the victim's domain or ip? Thank you
Posted by Tony -B, 04-14-2016, 10:36 AM
try checking for any unusual Consumption for the bandwidth per domain
Posted by jep_, 04-14-2016, 10:43 AM
Is it windows or Linux? You can use netstat to see which process id is connecting to that IP (-o in windows, -p in Linux), and go from there. IIS/apache logs may show an abnormal increase in size for one of your customers, if the culprit is the web service.
Posted by newbie_player, 04-14-2016, 11:11 AM
Hi there, Thank you for the reply. Im using cloudlinux OS.. Im currently using temporary quick fix, that is to block incoming and outgoing access to the victim's IP (from CSF). So what should i type on the terminal? netstat -p? Is it live? Or is there any log that i can check what domain that accessed the victim's IP in the previous day for example. Thank you
Posted by jep_, 04-14-2016, 11:54 AM
Yes, if you type netstat -np (n for numeric output), and look for that victim IP in the "foreign address" column, you'll see which PID/Program name is creating the traffic. If it's httpd, the next step would be to check web logs for unusually large files. You could also run a dump of traffic going to that IP, with tcpdump. Something like tcpdump -v -X dst host victimIP
Posted by newbie_player, 04-16-2016, 09:47 PM
Thanks for the reply.. For this time, i just block outgoing connection to that IP using firewall.. Seems like a temp fix for me.. But it's okay atm
Posted by HostingBig, 04-17-2016, 01:24 AM
chances are they all ready banned your IP so its not going to connect to it again anyway. but the exploit will still be there and your server will be attacking others you can use netstat to find the outbound connections and use tcpdump and lsof to find the corresponding inbound process that is controlling the outbound
Posted by newbie_player, 04-17-2016, 06:16 PM
Hi everyone, The netstat -np seems good to check the current connection to the victim's IP. But what if the attack happened yesterday? Could i also use netstat to see any outgoing connection to the victim's IP? If yes, what netstat command? Thank you
Posted by HelpOps, 04-17-2016, 07:21 PM
If you want to know what is going on you could log all outbound and inbound traffic using a firewall which is highly recommended for web hosts so investigating issues is easier along with finding out abuse internally and externally. I would recommend logging it and system events to an external syslog server so you do not have to use your regular production web hosting server when reviewing logs.
Posted by newbie_player, 04-17-2016, 07:23 PM
Hi HelpOps, That is a nice concept. How do i do that on CSF? Logging outbound and inbound traffic. For now, maybe i will use the production server for logging as it's not high loaded yet.. Thank you
Posted by HelpOps, 04-17-2016, 08:26 PM
You can try the following if they are not already in your iptables chain which should take care of most of the services, insure they are towards the top of your iptables chain. Change eth0 to the name of the interface for your server which should be located in /etc/sysconfig/network-scripts or by typing in ip addr which will include the ip addresses of your server. Don't forget to insure you have enough space for the additional logging, if not you should logrotate on the messages log so it doesn't get too big and fill up your /var partition if it is separate. You can then find the additional logs in /var/log/messages. iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 443 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 80 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 25 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 587 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 465 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 22 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 21 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 2087 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 2086 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 2083 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 2082 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 143 -j LOG iptables -I INPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 993 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 443 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 80 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 25 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 587 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 465 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 22 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 21 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 2087 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 2086 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 2083 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 2082 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 143 -j LOG iptables -I OUTPUT -i eth0 -s 0.0.0.0/0 -p tcp --dport 993 -j LOG Last edited by HelpOps; 04-17-2016 at 08:29 PM.
Posted by newbie_player, 05-02-2016, 06:33 AM
Hi @HelpOps, I just need the output logging, so i guess only the output codes above that i have to put in, right? When i tried to put the code, i got this error: iptables v1.4.7: Can't use -i with OUTPUT Should i remove the -i from the code? Thanks
Posted by HelpOps, 05-02-2016, 07:04 AM
-i is for the interface Try using iptables -N OUTPUT first to create a new iptables chain called OUTPUT. Then use iptables -L to view what you have.
Posted by newbie_player, 05-02-2016, 07:07 AM
Hi @HelpOps, I have this when inputing iptables -N OUTPUT: iptables: Chain already exists. should i use like iptables -I OUTPUT -s 0.0.0.0/0 -p tcp --dport 80 -j LOG? Thanks..
Posted by HelpOps, 05-02-2016, 06:07 PM
Yes, that will work, though I recommend you log outbound traffic for at least 443 too. If you log inbound 80,443 it may help you find out who is driving the attack too once but might be a bit much to search through when you match it with the IP address. Note after you get the proper IP addresses you may need to search through your apache logs to find the culprits.
Posted by SneakySysadmin, 05-02-2016, 06:58 PM
While IPTables logging will work it's not going to tell you anything you don't already know. Your machine is talking to 22.22.22.22 -- and you already know that. If that's all you want to see there's a much easier way than fumbling around with iptables logging: tcpdump -vvvnni eth0 'host 22.22.22.22' That will show the traffic for that host. It's the same info you'd get from iptables including the port numbers being used. Change 'eth0' if need be to access your primary network interface. On virts that's usually something like venet0 or similar. Method One Method One is the sneaky way to figure out what is happening on your server. Let's proceed under the (wise) assumption that this malicious traffic is being generated as a result of some site you host having their WordPress or Joomla (or some other garbage) compromised. The intruder has uploaded their little shell scripts and are now happily using your server to attack others. By far the fastest and easiest way to figure out what's going on: You will get a listing of all log files :X where X is how many POST requests have been logged by that site. If all your customers have 0 to 20 POSTS and one has Eleventy Billion And Three POSTs? You've found your culprit :p Method Two If you don't get lucky and Method One doesn't work or it's inconclusive (which can happen) and you can't figure it out, the more thorough method is this one, and it will work for any process not just Apache: netstat -latnp | grep ESTAB | grep 22.22.22.22 Of course, the activity has to be happening at the moment you run that, but let's assume for the moment that it is. Your output will look something kinda like this: What you have there now is the exact process id (PID) of what was talking to that 22.22.22.22 host. Let's proceed under the assumption that it is the most likely culprit, which is a compromised virtual host with some rogue PHP script being used by someone. Apache doesn't normally log the process id. To use that, you're going to have to change your servers LogFormat definition and add the '%P' to the line - for both the main server and all virtual hosts. This may or may not confuse whatever stat package you have processing customer log files, but it's something you're going to have to live with temporarily. If you're in luck you'll only need to leave this setting in place for a few hours at most. http://httpd.apache.org/docs/current...og_config.html Check all your apache virtual host configs and find out what LogFormat is in use and modify it to include the PID in the logs for you. You should only have to do this in one place (unfamiliar with CPanel since we use Plesk) but if all your virtual hosts are using LogFormat called "combined" then just modify that one line in the config and apachectl reload. Then just repeatedly run the netstat command from above until you find a connection with a process ID - then grepping your customer logs for that pid, like so: (assuming all your logs are in /etc/httpd/domlogs) Continuing from the above example, let's look for which log file has PID # 5623 in it. find /etc/httpd/domlogs -type f -name \*_log -exec grep -lw 5623 \{\} \; That will show the name of any log where the number 5623 appears by itself on a line (the -w looks for "whole" matches so there's no substring matching) You can then grep that specific log for the pid and omit the '-l' switch this time. You'll see exactly what requests that PID handled - who the culprit was, and most importantly what they were accessing on your server to generate that traffic. You'll probably see a lot of POSTs to a particular script. If it's something other than Apache don't worry - almost all other services on the server log their PID natively without you having to muck about with any configs (Apache is the exception to the rule). Still, Apache is the odds-on favorite to be your culprit in this case so it's a safe bet this method will work for you. From there, it's all over but nuking of the compromised crap and telling your customer "Hey! Guess what!? Your WordPress install is 3 years out-of-date and has been compromised. Dumbass!" ... although you might want to be a bit more diplomatic about it :p Last edited by SneakySysadmin; 05-02-2016 at 07:05 PM.
Posted by newbie_player, 05-02-2016, 08:06 PM
Hi @HelpOps, I have put the port 80 and 443 but there is no success message or something, is it okay? This is what i meant: If it's correct, please let me know how can i see the log in the future when i need it. And how can i delete the log if it's making the disk full? Thank you
Posted by newbie_player, 05-02-2016, 08:10 PM
Thank you for the detailed help. I tried to use tcpdump -vvvnni eth0 'host ip.add.re.ss' but it shown nothing, so i assume they are not happening at this time? For the domain log also i got everything log:0. My server provider also adviced me to use watch -d -n 0.3 "netstat -nputw | grep EST | grep my.ip.addr.ess | grep victim.ip.add.ress" which also usable only when the connection is currently making connection. They also adviced me to use darkstat, but i don't see anything on it about the victim's IP.
Posted by SneakySysadmin, 05-02-2016, 08:30 PM
Correct. It's a real-time peek into what's happening. iptables at least has the benefit of logging the activity so you can look at it later, after the fact. If there are older, processed logs kept they're probably gzipped - you can grep those (same switches) using zgrep. Also note that log file names may be different on your server than what I'm putting into my examples, so you can also just do a grep -cH POST * in the entire directory and see what happens. Edit: In fact, do this instead -- That will exclude all the files that have zero matches, makes it easier to spot things especially if you're looking at a lot of log files.
Posted by HelpOps, 05-02-2016, 08:36 PM
You would need to tail -f /var/log/messages, though insure you do not have port 22 in the list as it will trigger rsyslog and stop logging for a period of time unless you adjust some settings from the default. As you will need something on your end to verify the attack claim, this will be your firewall logs. As they could be using curl or another method that is not logged by apache (wget, curl, perl, python, scripts). If they are not then that is great and you can search through your apache logs as normally an outgoing attack will have someone connecting to a site you host to initiate that attack and view it's progress, take action on certain events, etc. If this is the case keep trying the methods listed by SneakySysadmin For the tcpdump insure you are running it as root, and if needed you can run it as a background process to catch the packets outbound to the victim if the attack reoccurs. If your hosts says the attack reoccurs but you do not see anything in your tcpdump using the ip information your host has given you then you have a bigger problem (rootkit, trojan). It is interesting that you just see log:0, have you made any modifications to tune down the logging? Last edited by HelpOps; 05-02-2016 at 08:45 PM.