I'm a little late to the party here.. but I wanted to share a solution I came up with to deal with this problem myself, in case it can still be useful to you.
This Python script will compare all files located in or beneath a root directory. If more than one file has the same contents, it reports them as duplicates.
Here's the code:
# for https://www.computerhope.com/forum/index.php/topic,164994.msg973770.html#msg973770
import sys
import argparse
import os
import hashlib
if sys.version_info[0] != 3:
print("This script requires Python 3.")
sys.exit(1)
# Arguments
parser = argparse.ArgumentParser(description='Compare contents of all files from a root directory, and report duplicates.')
parser.add_argument('searchroot', help="Path of the directory where the search/comparison should begin.")
parser.add_argument('-o', '--outputfile', help="Output the report to a file, OUTPUTFILE. If this option is omitted, the report is printed to standard output.", default=None, required=False)
args = parser.parse_args()
if not os.path.isdir(args.searchroot):
print("SEARCHROOT (", args.searchroot, ") is not a directory. Exiting.")
sys.exit(1)
# Functions
def hashfile(path, blocksize = 63356): # arbitrary sane blocksize
""" Compute the MD5 hash of a file, located at path.
Later, the hash digest will be used to quickly compare the contents of files.
"""
hashobj = hashlib.new('md5') # create new md5 hash object
try:
with open(path, 'rb') as fd: # open file in read-binary mode
try:
buf = fd.read(blocksize) # read the first chunk of the file
while len(buf) > 0: # do until we're out of data:
hashobj.update(buf) # append buf to the md5 hash obj
buf = fd.read(blocksize) # read next chunk and loop
except:
print ("Could not read ", path, ", skipping") # e.g. permission denied
return None
except:
print ("Could not open ", path, ", skipping") # e.g. NTFS Junction is walked, but throws an error on open()
return None
return hashobj.hexdigest() # hash the data and return the digest
def build_master_hash(root_folder): # root_folder is root of the search
""" Build a master list of md5 hashes of every file encountered.
Concurrently build a second list containing the duplicates only.
"""
masterHashList = {} # {'hash': ['/path/to/file', ... ], ... }
dupes_only = {} # same, but only the duplicates
for dirpath, subdirs, file_list in os.walk(root_folder, followlinks=False):
print('Processing directory: %s' % dirpath)
for filename in file_list:
path = os.path.join(dirpath, filename) # get full path
if os.path.isfile(path): # operate on regular files only
hash = hashfile(path) # hash file contents
if hash is not None: # proceed only if we got a hash
if hash in masterHashList: # if we've seen it before
masterHashList[hash].append(path) # append path info to that hash
dupes_only[hash] = masterHashList[hash] # and add to dupes
else: # otherwise
masterHashList[hash] = [path] # add new hash & filename
return masterHashList, dupes_only
def generate_report(dupes, outputFile=None):
""" Generate human-readable list of duplicates. """
report = "" # this will be a long text file
for dupe in dupes: # for each entry in dupes
report += "--\r\n" # add a separator line and CRLF
for filename in dupes[dupe]: # for each filename associated with a hash
report += str(filename) + "\r\n" # add it to the report
totalString = "\r\nTotal files with duplicates: " + str(len(dupes)) + "\r\n"
report += totalString # Finally, include a total
if outputFile is not None: # If the -o/--outputfile option was specified
with open(outputFile, 'wt') as fd:
print(totalString, "\r\nWriting report to: ", outputFile)
fd.write(report) # (over)write the report to that file
else: # otherwise
print(report) # print it to the screen (standard output)
# Start
print ("hi")
master, dupes = build_master_hash(args.searchroot) # Recursively hash contents of files starting at searchroot
# NOTE: omit trailing backslash in DOS paths
generate_report(dupes, args.outputfile) # generate report, write to file or screen
# NOTE: if output file exists, it will be overwritten
sys.exit()
# End
To use this program, first install Python 3 if you don't have it installed. You can get it at:
https://www.python.org/downloads/When installing, select option to add Python to your PATH environment variable so you can run python from any directory.
Copy the above code and paste it into a text file (using notepad for instance), and save it as
find-dupes.py.
Run it with:
python find-dupes.py "C:\path\to be\searched" -o outputfile.txtIt will generate a report and write it to outputfile.txt. If you omit
-o outputfile, it will print the report to the screen instead.
I've tested it on Windows 10 and it appears to work correctly
Note that this only identifies duplicates, it doesn't modify any of the files it finds. (Figuring out which file to keep is more difficult to automate, but I'm going to have to figure out a solution for that part -- probably for each file, ask the user which one to keep.)
Another limitation is all the files to be compared have to be on the same drive. This suits my purpose so I left it this way.. but if you need to run this on more than one drive (compare all of C:\ to all of D:\ for example) it could be rewritten that way, where you can provide searchroot1 searchroot2 etc.. let me know
Hope this helps --
