Log files keep accumulating data over time and it becomes really difficult to extract information for a specific time period or date range from them. Sometimes you may need to grep log file within specific time period in Linux, or extract logs within a date range. In this article, we will look at how to extract log records within specific time period in Linux. In fact, awk command is much more useful than grep for this purpose, as you will see below.
How to Grep Log File Within Specific Time Period in Linux
Although the title of this article is about using grep, we strongly recommend using awk command to get log records within specific time period. Let us assume your log file is located at /etc/nginx/nginx-access.log.
Let us say you want to extract log records for the last 2 minutes. Each log file has a different format. First we need to get the date format of our log file. Use the tail command to get the last 2-3 records as shown below.
$ tail -n2 nginx-access.log 207.46.13.51 - - [20/Aug/2021:07:23:07 +0000] "GET /database-blog/wp-content/themes/dazzling/inc/fonts/fontawesome-webfont.woff2?v=4.4.0 HTTP/1.1" 200 66624 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 185.172.169.152 - - [20/Aug/2021:07:23:08 +0000] "GET /robots.txt HTTP/1.1" 200 400 "-" "Screaming Frog SEO Spider/15.2"
In the above log the date format is 20/Aug/2021:07:23:07 that is DD/MMM/YYYY:HH:MM:SS. Now here is the awk command to extract data for the last 2 minutes.
$ awk -vDate=`date -d'now-2 minutes' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' /etc/nginx/nginx-access.log
In the above command, %d/%b/%Y:%H:%M:%S is the format specifier of your date column. You need to update it as per your log file’s date format. Also $4 indicates that the date is present as 4th column in your log file. Again, you need to update this according to your log format.
Once you have these 2 pieces of information you can easily customize your awk command as per your requirement.
Here is the command to get log records for past 2 hours. You just need to replace minutes keyword above with hours keyword.
$ awk -vDate=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' /etc/nginx/nginx-access.log
Here is the awk command to get log results after 07:15.
awk -vDate=`date -d'07:15' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' /etc/nginx/nginx-access.log
Here is the awk command to get log results between 15:20 and 15:30
$ sudo awk -vDate=`date -d ’15:20′ +[%d/%b/%Y:%H:%M:%S -vDate2=
date -d’15:30′ +[%d/%b/%Y:%H:%M:%S` ‘ { if ($4 > Date && $4 < Date2) print $0}’ /etc/apache2/access.log
You can add the above commands to shell script to automate such data extraction and store it another file. You can also create a cron job out of it, so that it runs regularly.
That’s it. In this article, we have looked at how to extract log data between specific times.
Also read:
How to Password Protect Folders in Linux
Tail Command to Check Log Files in Linux
Related posts:
Delete All Files Except One in Linux
How To Make File Executable in Linux
How to Open Multiple Files in Vim
How to Setup SSH Passwordless Login
How to Use Auto Indent in Vi Editor
How to Convert CRLF to LF in Linux
What does ${} and $() mean in Shell Script
How to Generate & Verify MD5 Hash of File in Linux

Sreeram has more than 10 years of experience in web development, Python, Linux, SQL and database programming.