Log files keep accumulating data over time and it becomes really difficult to extract information for a specific time period or date range from them. Sometimes you may need to grep log file within specific time period in Linux, or extract logs within a date range. In this article, we will look at how to extract log records within specific time period in Linux. In fact, awk command is much more useful than grep for this purpose, as you will see below.
How to Grep Log File Within Specific Time Period in Linux
Although the title of this article is about using grep, we strongly recommend using awk command to get log records within specific time period. Let us assume your log file is located at /etc/nginx/nginx-access.log.
Let us say you want to extract log records for the last 2 minutes. Each log file has a different format. First we need to get the date format of our log file. Use the tail command to get the last 2-3 records as shown below.
$ tail -n2 nginx-access.log 207.46.13.51 - - [20/Aug/2021:07:23:07 +0000] "GET /database-blog/wp-content/themes/dazzling/inc/fonts/fontawesome-webfont.woff2?v=4.4.0 HTTP/1.1" 200 66624 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 185.172.169.152 - - [20/Aug/2021:07:23:08 +0000] "GET /robots.txt HTTP/1.1" 200 400 "-" "Screaming Frog SEO Spider/15.2"
In the above log the date format is 20/Aug/2021:07:23:07 that is DD/MMM/YYYY:HH:MM:SS. Now here is the awk command to extract data for the last 2 minutes.
$ awk -vDate=`date -d'now-2 minutes' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' /etc/nginx/nginx-access.log
In the above command, %d/%b/%Y:%H:%M:%S is the format specifier of your date column. You need to update it as per your log file’s date format. Also $4 indicates that the date is present as 4th column in your log file. Again, you need to update this according to your log format.
Once you have these 2 pieces of information you can easily customize your awk command as per your requirement.
Here is the command to get log records for past 2 hours. You just need to replace minutes keyword above with hours keyword.
$ awk -vDate=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' /etc/nginx/nginx-access.log
Here is the awk command to get log results after 07:15.
awk -vDate=`date -d'07:15' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' /etc/nginx/nginx-access.log
Here is the awk command to get log results between 15:20 and 15:30
$ sudo awk -vDate=`date -d ’15:20′ +[%d/%b/%Y:%H:%M:%S -vDate2=date -d’15:30′ +[%d/%b/%Y:%H:%M:%S` ‘ { if ($4 > Date && $4 < Date2) print $0}’ /etc/apache2/access.log
You can add the above commands to shell script to automate such data extraction and store it another file. You can also create a cron job out of it, so that it runs regularly.
That’s it. In this article, we have looked at how to extract log data between specific times.
Also read:
Top Yum Command Examples in Linux
How to Password Protect Folders in Linux
How to Restrict SSH Users to Specific Folders
How to Monitor Log Files in Real Time
Tail Command to Check Log Files in Linux