How to install fdupes in Linux
To install " fdupes" which is used to Find and Delete Duplicate Files in Linux
Fdupes is a program to scan directories for duplicate files. It recognize the duplicates using MD5 signature of files along with a byte-to-byte comparison. List, replace and delete options can be used along with fdupes. It is programmed in C Language and is released under MIT License. Installation of fdupes is explained in this article
fdupes compares the files in the following order.
Size comparison -> Partial MD5 Signature Comparison -> Full MD5 Signature Comparison -> Byte-to-Byte Comparison.
To Install fdupes on Linux
Execute the below command to Install the latest version of fdupes.
On Fedora and CentOS/RHEL based system
Enable epel repository before installing fdupes package. Then run the following command to install.
[root@linuxhelp ~]# yum install fdupes
Important: In fedora, the default yum package manager is replaced by ' dnf' .
On Debian based systems
root@linuxhelp:~# apt-get install fdupes -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libavahi-client-dev libavahi-common-dev libavahi-compat-libdnssd1
libdbus-1-dev libjs-jquery libruby2.1 libyaml-0-2 ruby2.1
rubygems-integration
Use ' apt-get autoremove' to remove them.
The following NEW packages will be installed:
fdupes
0 upgraded, 1 newly installed, 0 to remove and 226 not upgraded.
Need to get 16.8 kB of archives.
After this operation, 69.6 kB of additional disk space will be used.
Get:1 http://in.archive.ubuntu.com/ubuntu/ wily/main fdupes amd64 1.51-1 [16.8 kB]
Fetched 16.8 kB in 0s (25.3 kB/s)
Selecting previously unselected package fdupes.
(Reading database ... 180830 files and directories currently installed.)
Preparing to unpack .../fdupes_1.51-1_amd64.deb ...
Unpacking fdupes (1.51-1) ...
Processing triggers for man-db (2.7.4-1) ...
Setting up fdupes (1.51-1) ...
To use fdupes command
Lets create few duplicate files under ' /home/user1/linuxhelp' .
root@linuxhelp:~# mkdir /home/user1/linuxhelp & & cd /home/user1/linuxhelp & & for i in {1..25} do echo " Welcome to linuxhelp." > file${i}.txt done
Run the following command to verify whether the duplicates files are created or not.
root@linuxhelp:~/linuxhelp# ls -l
total 100
-rw-r--r-- 1 root root 22 May 24 09:25 file1.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file2.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file3.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file4.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file5.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file6.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file7.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file8.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file9.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file10.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file11.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file12.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file13.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file14.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file15.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file16.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file17.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file18.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file19.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file20.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file21.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file22.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file23.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file24.txt
-rw-r--r-- 1 root root 22 May 24 09:25 file25.txt
Now we have created 25 files and all files contains the same data, " Welcome to linuxhelp."
To search for duplicate files
Run the following command to search the duplicate file.
root@linuxhelp:~# fdupes /home/user1/linuxhelp
/home/user1/linuxhelp/file1.txt
/home/user1/linuxhelp/file2.txt
/home/user1/linuxhelp/file3.txt
/home/user1/linuxhelp/file4.txt
/home/user1/linuxhelp/file5.txt
/home/user1/linuxhelp/file6.txt
/home/user1/linuxhelp/file7.txt
/home/user1/linuxhelp/file8.txt
/home/user1/linuxhelp/file9.txt
/home/user1/linuxhelp/file10.txt
/home/user1/linuxhelp/file11.txt
/home/user1/linuxhelp/file12.txt
/home/user1/linuxhelp/file13.txt
/home/user1/linuxhelp/file14.txt
/home/user1/linuxhelp/file15.txt
/home/user1/linuxhelp/file16.txt
/home/user1/linuxhelp/file17.txt
/home/user1/linuxhelp/file18.txt
/home/user1/linuxhelp/file19.txt
/home/user1/linuxhelp/file20.txt
/home/user1/linuxhelp/file21.txt
/home/user1/linuxhelp/file22.txt
/home/user1/linuxhelp/file23.txt
/home/user1/linuxhelp/file24.txt
/home/user1/linuxhelp/file25.txt
Use ' -r' option to Search for duplicate files recursively under every directory including its sub-directories.
root@linuxhelp:~# fdupes -r /home/
/home/user1/.sudo_as_admin_successful
/home/user1/.cache/sso/sso-client-gui.log
/home/user1/.local/share/unity-settings-daemon/input-sources-converted
/home/user1/.config/compiz-1/compizconfig/config
/home/user1/.gconf/apps/%gconf.xml
/home/user1/.local/share/.converted-launchers
/home/user1/.config/unity/first_run.stamp
/home/user1/.cache/motd.legal-displayed
/home/user1/.local/share/zeitgeist/fts.index/flintlock
/home/user1/linuxhelp/file1.txt
/home/user1/linuxhelp/file2.txt
/home/user1/linuxhelp/file3.txt
/home/user1/linuxhelp/file4.txt
/home/user1/linuxhelp/file5.txt
/home/user1/linuxhelp/file6.txt
/home/user1/linuxhelp/file7.txt
/home/user1/linuxhelp/file8.txt
/home/user1/linuxhelp/file9.txt
/home/user1/linuxhelp/file10.txt
/home/user1/linuxhelp/file11.txt
/home/user1/linuxhelp/file12.txt
/home/user1/linuxhelp/file13.txt
/home/user1/linuxhelp/file14.txt
/home/user1/linuxhelp/file15.txt
/home/user1/linuxhelp/file16.txt
/home/user1/linuxhelp/file17.txt
/home/user1/linuxhelp/file18.txt
/home/user1/linuxhelp/file19.txt
/home/user1/linuxhelp/file20.txt
/home/user1/linuxhelp/file21.txt
/home/user1/linuxhelp/file22.txt
/home/user1/linuxhelp/file23.txt
/home/user1/linuxhelp/file24.txt
/home/user1/linuxhelp/file25.txt
To view the size of duplicates found within a folder use the below command.
root@linuxhelp:~# fdupes -S /home/user1/linuxhelp/
22 bytes each:
/home/user1/linuxhelp/file1.txt
/home/user1/linuxhelp/file2.txt
/home/user1/linuxhelp/file3.txt
/home/user1/linuxhelp/file4.txt
/home/user1/linuxhelp/file5.txt
/home/user1/linuxhelp/file6.txt
/home/user1/linuxhelp/file7.txt
/home/user1/linuxhelp/file8.txt
/home/user1/linuxhelp/file9.txt
/home/user1/linuxhelp/file10.txt
/home/user1/linuxhelp/file11.txt
/home/user1/linuxhelp/file12.txt
/home/user1/linuxhelp/file13.txt
/home/user1/linuxhelp/file14.txt
/home/user1/linuxhelp/file15.txt
/home/user1/linuxhelp/file16.txt
/home/user1/linuxhelp/file17.txt
/home/user1/linuxhelp/file18.txt
/home/user1/linuxhelp/file19.txt
/home/user1/linuxhelp/file20.txt
/home/user1/linuxhelp/file21.txt
/home/user1/linuxhelp/file22.txt
/home/user1/linuxhelp/file23.txt
/home/user1/linuxhelp/file24.txt
/home/user1/linuxhelp/file25.txt
To view the size of duplicate files for every directory and subdirectories encountered inside, use ' -Sr' option as follows.
root@linuxhelp:~# fdupes -Sr /home/
0 bytes each:
/home/user1/.sudo_as_admin_successful
/home/user1/.cache/sso/sso-client-gui.log
/home/user1/.local/share/unity-settings-daemon/input-sources-converted
/home/user1/.config/compiz-1/compizconfig/config
/home/user1/.gconf/apps/%gconf.xml
/home/user1/.local/share/.converted-launchers
/home/user1/.config/unity/first_run.stamp
/home/user1/.cache/motd.legal-displayed
/home/user1/.local/share/zeitgeist/fts.index/flintlock
22 bytes each:
/home/user1/linuxhelp/file1.txt
/home/user1/linuxhelp/file2.txt
/home/user1/linuxhelp/file3.txt
/home/user1/linuxhelp/file4.txt
/home/user1/linuxhelp/file5.txt
/home/user1/linuxhelp/file6.txt
/home/user1/linuxhelp/file7.txt
/home/user1/linuxhelp/file8.txt
/home/user1/linuxhelp/file9.txt
/home/user1/linuxhelp/file10.txt
/home/user1/linuxhelp/file11.txt
/home/user1/linuxhelp/file12.txt
/home/user1/linuxhelp/file13.txt
/home/user1/linuxhelp/file14.txt
/home/user1/linuxhelp/file15.txt
/home/user1/linuxhelp/file16.txt
/home/user1/linuxhelp/file17.txt
/home/user1/linuxhelp/file18.txt
/home/user1/linuxhelp/file19.txt
/home/user1/linuxhelp/file20.txt
/home/user1/linuxhelp/file21.txt
/home/user1/linuxhelp/file22.txt
/home/user1/linuxhelp/file23.txt
/home/user1/linuxhelp/file24.txt
/home/user1/linuxhelp/file25.txt
To delete the duplicate files
Use option ' -d' to delete the duplicate files.
All the duplicates are listed and now you can delete, either one by one, all or in certain range.
Except one file, we can delete all the remaining files by running the following command.
root@linuxhelp:~# fdupes -d /home/user1/linuxhelp/
[1] /home/user1/linuxhelp/file1.txt
[2] /home/user1/linuxhelp/file2.txt
[3] /home/user1/linuxhelp/file3.txt
[4] /home/user1/linuxhelp/file4.txt
[5] /home/user1/linuxhelp/file5.txt
[6] /home/user1/linuxhelp/file6.txt
[7] /home/user1/linuxhelp/file7.txt
[8] /home/user1/linuxhelp/file8.txt
[9] /home/user1/linuxhelp/file9.txt
[10] /home/user1/linuxhelp/file10.txt
[11] /home/user1/linuxhelp/file11.txt
[12] /home/user1/linuxhelp/file12.txt
[13] /home/user1/linuxhelp/file13.txt
[14] /home/user1/linuxhelp/file14.txt
[15] /home/user1/linuxhelp/file15.txt
[16] /home/user1/linuxhelp/file16.txt
[17] /home/user1/linuxhelp/file17.txt
[18] /home/user1/linuxhelp/file18.txt
[19] /home/user1/linuxhelp/file19.txt
[20] /home/user1/linuxhelp/file20.txt
[21] /home/user1/linuxhelp/file21.txt
[22] /home/user1/linuxhelp/file22.txt
[23] /home/user1/linuxhelp/file23.txt
[24] /home/user1/linuxhelp/file24.txt
[25] /home/user1/linuxhelp/file25.txt
Set 1 of 1, preserve files [1 - 25, all]: 2-25
[-] /home/user1/linuxhelp/file1.txt
[-] /home/user1/linuxhelp/file2.txt
[-] /home/user1/linuxhelp/file3.txt
[-] /home/user1/linuxhelp/file4.txt
[-] /home/user1/linuxhelp/file5.txt
[-] /home/user1/linuxhelp/file6.txt
[-] /home/user1/linuxhelp/file7.txt
[-] /home/user1/linuxhelp/file8.txt
[-] /home/user1/linuxhelp/file9.txt
[-] /home/user1/linuxhelp/file10.txt
[-] /home/user1/linuxhelp/file11.txt
[-] /home/user1/linuxhelp/file12.txt
[-] /home/user1/linuxhelp/file13.txt
[-] /home/user1/linuxhelp/file14.txt
[-] /home/user1/linuxhelp/file15.txt
[-] /home/user1/linuxhelp/file16.txt
[+] /home/user1/linuxhelp/file17.txt
[-] /home/user1/linuxhelp/file18.txt
[-] /home/user1/linuxhelp/file19.txt
[-] /home/user1/linuxhelp/file20.txt
[-] /home/user1/linuxhelp/file21.txt
[-] /home/user1/linuxhelp/file22.txt
[-] /home/user1/linuxhelp/file23.txt
[-] /home/user1/linuxhelp/file24.txt
[-] /home/user1/linuxhelp/file25.txt
Here, except the file17.txt, all the remaining files are removed.
For safety purpose you may also print the output of " fdupes" to file and then delete using the following command.
root@linuxhelp:~# fdupes -Sr /home > /home/fdupes.txt
Important: You can replace " /home" with the your desired folder. Also use " -r" and " -S" option if you need to search recursively and Print size.
To check the version
Run the following command to check the version.
root@linuxhelp:~# fdupes --version
fdupes 1.51
If you want to know more options in fdupes use ' -h' option.
root@linuxhelp:~# fdupes -h
Usage: fdupes [options] DIRECTORY...
-r --recurse for every directory given follow subdirectories
encountered within
-R --recurse: for each directory given after this option follow
subdirectories encountered within (note the ' :' at
the end of the option, manpage for more details)
-s --symlinks follow symlinks
-H --hardlinks normally, when two or more files point to the same
disk area they are treated as non-duplicates this
option will change this behavior
-n --noempty exclude zero-length files from consideration
-A --nohidden exclude hidden files from consideration
-f --omitfirst omit the first file in each set of matches
-1 --sameline list each set of matches on a single line
-S --size show size of duplicate files
-m --summarize summarize dupe information
-q --quiet hide progress indicator
-d --delete prompt user for files to preserve and delete all
others important: under particular circumstances,
data may be lost when using this option together
with -s or --symlinks, or when specifying a
particular directory more than once refer to the
fdupes documentation for additional information
-N --noprompt together with --delete, preserve the first file in
each set of duplicates and delete the rest without
prompting the user
-v --version display fdupes version
-h --help display this help message
Comments ( 0 )
No comments available