Java command line utility to removes duplicate lines from text files.
From: Canadian Mind Products
DeDup is a Java command-line utilty for removing duplicate
lines in text files.
The dedup process compares adjacent lines only. It does not
sort first. The comparison is case sensitive. It removes
adjacent indentical lines. It replaces the original file, so
do a backup first. You can dedup 1 to N files in one
execution, just put their names on the command line.
DeDup deletes blank lines from both the beginning and end of
the file. It also deletes trailing blanks from each line. It
also cleans up line terminators \r \n \r\n and converts them
all to platform standard. If the file is already deduped,
it does not change the date.
Use it with:
java com.mindprod.dedup.DeDup myfile.txt anotherfile.txt yetanotherfile.txt
java -jar dedup.jar myfile.txt anotherfile.txt yetanotherfile.txt
or, if you have the jar association set up, even even
dedup.jar myfile.txt anotherfile.txt yetanotherfile.txt
It can handle a list of files, or an os-expanded wildcard.
It does not -s to process subdirectories or the use of
directory names to process entire directories.
Changes on the new version:
add icon and pad file
OS Support: Java, Linux, Mac OS X, Unix, Win2000, Win7 x32, Win7 x64, WinServer, WinXP, Win Vista
Hits: 173 visitors
Date added: 15 Aug 2010
Last Update: 24 Jun 2007See full specifications