How to Find & Remove Duplicate Files in Linux

Sometimes you may need to delete duplicate files on your Linux system. There are many third party tools available to accomplish this task. In this article, we will look at 3 different tools to find & remove duplicate files in Linux.

How to Find & Remove Duplicate Files in Linux

Here are the different utilities to remove duplicates on your Linux system.

1. FSlint

FSlist is a free, simple utility that can be used to find and remove different kinds of extraneous data from your system – such as empty folders, duplicate files, temporary files, conflicting/duplicate names, bad symbolic links and more. It is available in both GUI and CLI.

Open terminal and run the following command to install it.

$ sudo apt-get install fslint     [On Debian/Ubuntu]
$ sudo yum install epel-release && $ sudo yum install fslint    [On CentOS/RHEL]
$ sudo dnf install fslint         [On Fedora 22+]
$ sudo pacman -S fslint   [On Arch Linux]

After installation, go to the folder where it is installed, and run the fslint command on the folder in which you want to find duplicates.

$ cd /usr/share/fslint/fslint
$ ./fslint /path/to/directory

Please note, fslint will not remove duplicates from your system, but only list them. You will need to delete them separately using rm command.

2. Fdupes

Fdupes is another popular tool to find duplicates on your system. It is a feature-rich utility with many options. You can ask it to exclude empty files, search folders recursively, find & delete files at the same time. Here is the command to install it.

$ sudo apt-get install fdupes     [On Debian/Ubuntu]
$ sudo yum install epel-release && $ sudo yum install fdupes    [On CentOS/RHEL]
$ sudo dnf install fdupes         [On Fedora 22+]
$ sudo pacman -S fdupes   [On Arch Linux]

You can find duplicates in a folder using the following command.

$ fdupes path/to/dir

If you want to find and delete duplicates use -d option.

$ fdupes -d /path/to/dir

If you want to recursively search for duplicates, use -r option

$ fdupes -r /path/to/dir

If you want to search multiple folders at the same time, you can specify them one after the other in a space-separated manner.

$ fdupes /path/to/dir1 /path/to/dir2

You can always use multiple options at the same time.

3. Rdfind

Rdfind is another popular utility to detect duplicate files. Here is the command to install it.

$ sudo apt-get install rdfind     [On Debian/Ubuntu]
$ sudo yum install epel-release && $ sudo yum install rdfind    [On CentOS/RHEL]
$ sudo dnf install rdfind         [On Fedora 22+]
$ sudo pacman -S rdfind   [On Arch Linux]

Like Fdupes, Rdfind also offers many options. Here is the command to find duplicate files in a folder.

$ rdfind /path/to/dir

rdfind saves the results of your last command as results.txt file in your present working directory.

If you want to only find duplicates without taking any action, use -dryrun option.

$ rdfind -dryrun true /path/to/dir

If you want to replace duplicates with hardlinks, then use the -makehardlinks option

$ rdfind -makehardlinks true /path/to/dir

If you want to find and delete duplicates, use the -deleteduplicates option.

$ rdfind -deleteduplicates true /path/to/dir

In this article, we have learnt about 3 popular tools to find & remove duplicate files in Linux.

How to Find & Remove Duplicate Files in Linux