So earlier this evening things started to slow down again and upon investigation discovered the attackers were now hitting my DNS servers. I applied the same firewall rules to it as I did to the main server.
Amazon in the meantime responded and told me they had dealt with one of the IP’s for my original report. Another email responding to a question informed me that for additional IPs I have to submit a fresh report.
I don’t fancy having to do this manually each time so decided to give them all the data and do it automatically. I was tempted to do a separate report for each IP but that would have been churlish of me.
So, how to automate this.
The first thing to note is that because I am blocking the entire IP range of Amazon EC2 I can no longer use application logs such as Nginx, Named or Apache etc. I will have to use the firewall to log this.
Here is how I do that.
The first thing to do is create a new chain that logs and drops. It does nothing else and has no actual entries in it. This chain will be called from other chains when a match is found.
iptables -N LOG_DROP iptables -A LOG_DROP -j LOG --log-level warning --log-prefix "INPUT-DROP:" iptables -A LOG_DROP -j DROP
The first line creates the new chain. The second line does the logging. The third line does the drop. It then returns to the calling chain.
UPDATE: I was dropping all connections from Amazon EC2. This turned out to not be a fantastic idea because some services were using EC2 for mail delivery. I have changed the log and drop chain to this now.
iptables -A INPUT -p tcp -m multiport --destination-port 80,443 -j DROP
Even though they are hitting the DNS server now, it is UDP traffic and they do not have enough hosts (approx 8,000) to make a dint on my servers performance.
So then I have to flush the amazon-ec2 chain
iptables -F amazon-ec2
Then re-add all the ip ranges using this new format.
iptables -I amazon-ec2 -s <IP> -j LOG_DROP -w
When a match is found, rather than the previous format command which just drops the connection, it now calls the LOG_DROP chain.
What I get in my /var/log/messages log file is a line for each dropped connection like this.
Jan 25 23:51:25 **** kernel: INPUT-DROP:IN=eth0 OUT= MAC=*:*:*:*:* SRC=18.104.22.168 DST=*.*.*.* LEN=84 TOS=0x00 PREC=0x00 TTL=232 ID=29013 PROTO=UDP SPT=28184 DPT=53 LEN=64
To issue a report I copy all the log entries to a new file and rotate the log file (to make clean matches in future parsing easier).
I take a representative sample of say 5 seconds from the file, so Amazon can see the frequency of the attack and a sample of the ports etc.
I then extract ALL the IP addresses that are taking part in the attack by using this command;
cat /var/log/messages | grep 'INPUT-DROP' | sed -rn -e 's,.* SRC=([0-9.]+).*,\1,p' | sort -t . -k1,1n -k2,2n -k 3,3n -k 4,4n | uniq
So after this I can send the first report.
Now I need to monitor the log file for additional IP addresses that are attacking but remove any that I have already reported. Then create a new report using a new section from the /var/log/message file and only the new IP addresses. I will provide an update after this happens.