TIL: Deduplicating Lines With sort -u

Duplicate lines in a file are easy to spot and tedious to remove. sort -u handles both in one pass.

sort -u wordlist.txt

Sorts alphabetically and removes duplicates. Output is a deduplicated sorted list.

sort -u -o wordlist.txt wordlist.txt

In-place deduplication. The -o flag safely writes output back to the input file.

sort -u -f wordlist.txt

The -f flag folds case. “Hello” and “hello” are treated as duplicates. Only the first occurrence is kept.

cat log.txt | sort -u | wc -l

Count unique lines in a log file. Useful for measuring how many distinct error messages appeared.

Does sort -u change line order?

Yes. It sorts first, then deduplicates. Use awk ‘!seen[$0]++’ to remove duplicates without changing order.

Is this faster than uniq?

sort -u does both steps. uniq only removes adjacent duplicates. For most cases sort -u is the right tool.