Sometimes you may need to filter server logs by date or time, or datetime range. Linux provides many useful tools such as awk, grep and sed to parse files and strings. You can use them to easily filter logs between date range as per your requirement. In this article, we will look at how to filter logs by date/time.
How to Filter Logs Between Date Range in Linux
Let us say you want to filter your Apache server log by date/time. Apache files are typically located at /etc/httpd or /etc/apache2. If you don’t know the location of your Apache log files, use these steps to find Apache log file location.
Here are the most commonly used commands to parse server logs using datetime. We will run them on file access.log. Update the file path below as per your requirement. Also you can update the part in bold to change your timeframe.
Get log records for past 2 hours
$ sudo awk -vDate=`date -d’now-2 hours’ +[%d/%b/%Y:%H:%M:%S` ‘ { if ($4 > Date) print Date FS $4}’ /etc/apache2/access.log
Get most active IPs within the last 2 hours
$ sudo awk -vDate=`date -d’now-2 hours’ +[%d/%b/%Y:%H:%M:%S` ‘ { if ($4 > Date) print $1}’ /etc/apache2/access.log | sort |uniq -c |sort -n | tail
Get most active IPs between 2 and 4 hours
$ sudo awk -vDate=`date -d'now-4 hours' +[%d/%b/%Y:%H:%M:%S` -vDate2=date -d'<strong>now-2 hours</strong>' +[%d/%b/%Y:%H:%M:%S ‘ { if ($4 > Date && $4 < Date2) print Date FS Date2 FS $4}’ /etc/apache2/access.log
Get most active IPs between specific times 15:20 and 15:30
$ sudo awk -vDate=`date -d '15:20' +[%d/%b/%Y:%H:%M:%S` -vDate2=`date -d'15:30' +[%d/%b/%Y:%H:%M:%S` ‘ { if ($4 > Date && $4 < Date2) print $0}’ /etc/apache2/access.log
If you want to automate the above task you can easily create a cron job for the above awk commands and pass their output to another file. Open crontab file as shown below.
$ sudo crontab -e
Add the following line to filter log everyday at 10.a.m to extract records of past 24 hours and copy the result to another file /home/past-day-records.txt. You may update the parts in bold to change the filter timeframe as well as output file location.
0 10 * * * sudo awk -vDate=`date -d'now-24 hours' +[%d/%b/%Y:%H:%M:%S` ‘ { if ($4 > Date) print Date FS $4}’ /etc/apache2/access.log > /home/past-day-records.txt
Save and close the crontab file.
Now, every day at 10.a.m according to server’s time zone, the above awk command will run, get log records for past 24 hours and copy it to your output file. You can easily get last 24 hour records at any time, by simply opening your output file.
$ sudo cat /home/past-day-records.txt
Similarly, you can also create cron jobs for the other awk commands mentioned above to automate log parsing for your server.
As you can see, Linux provides powerful tools to filter server logs and make sense out of them.
Also read:
How to Verify Checksum in Linux
How to Create PDF in Python
How to Redirect IP to Domain in NGINX
How to Fix Mixed Content/Insecure Content in Apache
How to Force HTTPS in .htaccess in Apache