Sorting and Deduplicating Text with `sort` and `uniq`

Organizing Lines with sort

The sort command does exactly what its name implies: it reads text line by line and outputs a sorted version. By default, it sorts alphabetically. A crucial point to remember is that, like grep and sed, sort sends its output to standard output and does not change the original file.

Let's watch a quick demonstration of its basic usage.

While alphabetical sorting is the default, sort has several options to control its behavior. Here are the most important ones:

• Numeric Sort (-n): When sorting lines that start with numbers, the default alphabetical sort can be misleading (e.g., "10" comes before "2"). The -n flag ensures a proper numerical sort.
• Reverse Order (-r): This flag reverses the sorting order, whether it's alphabetical or numerical.
• Unique (-u): This is a convenient shortcut that removes duplicate lines after sorting. We'll see this is similar to, but distinct from, the uniq command.
• Output File (-o): Instead of redirecting output with > (which you can still do), you can use sort -o output_file.txt input_file.txt. This is a safer way to overwrite the original file with the sorted content (sort -o file.txt file.txt).

Let's see the numeric sort and unique sort options in action.

For a more detailed reference on these options, you can consult this article from Red Hat.

Removing Duplicates with uniq

While sort -u is handy, the uniq command offers more granular control over handling duplicate lines. Its primary function is to filter or report repeated lines.

However, there is one critical rule you must never forget: uniq only detects adjacent duplicate lines. If two identical lines are not next to each other, uniq will not see them as duplicates.

So, if uniq only works on adjacent lines, how do we make it work on an unsorted file? We use the tools you've been learning about: we pipe the output of sort directly into uniq!

The sort file.txt | uniq Pattern

This command pipeline is one of the most common and powerful combinations in the Linux command line. sort first orders the file, placing all identical lines next to each other. Then, uniq can do its job effectively.

Powerful uniq Options

uniq becomes even more useful for analysis with these flags:

• Count Occurrences (-c): Prefixes each line with the number of times it occurred. This is fantastic for summarizing data.
• Show Only Duplicates (-d): Displays only the lines that appeared more than once.
• Show Only Unique Lines (-u): The opposite of -d; it displays only the lines that appeared exactly once.
• Ignore Case (-i): Treats uppercase and lowercase letters as the same when comparing lines.

Your Turn: Hands-on Practice

Time to put sort and uniq to work. As an aspiring Django developer, you'll spend a lot of time looking at application logs. Let's simulate that.

1. Setup
Open your terminal and create a mock log file.

cd ~
mkdir -p text_practice
cd text_practice




# Create a mock application log file
cat << 'EOF' > app.log
WARNING: User authentication failed
ERROR: Database connection timeout
INFO: New user registered
WARNING: User authentication failed
ERROR: Cache server not responding
INFO: New user registered
ERROR: Database connection timeout
WARNING: User authentication failed
INFO: Payment processed successfully
ERROR: Database connection timeout
EOF

2. Exercises
Execute the commands to solve each task.

• Exercise 1: Get an Overview
  Sort the contents of app.log alphabetically to group similar messages together.

  Solution

  sort app.log
  

• Exercise 2: Create a Unique Summary
  You want a clean list of all the unique types of messages that have appeared in the log.

  Solution

  
  
  
  # Remember, you must sort before using uniq!
  sort app.log | uniq
  

• Exercise 3: Count the Incidents
  Now find out how many times each unique message has occurred.

  Solution

  sort app.log | uniq -c
  

  Your output should show that "Database connection timeout" occurred 3 times, for example.

• Exercise 4: Find the Most Frequent Message
  This is a classic sysadmin task. Building on the previous command, find the most frequent error message by sorting the counts. (Hint: You'll need to sort again, numerically and in reverse order).

  Solution

  
  
  
  # A powerful three-command pipeline!
  sort app.log | uniq -c | sort -nr
  

  The output will be a list of messages, ranked from most to least frequent.

• Exercise 5: Case-Insensitive Deduplication
  Let's create a new file with case differences.

  cat << 'EOF' > users.txt
  jdoe
  asmith
  JDOE
  rWilliams
  asmith
  EOF
  

  First, run the standard sort | uniq command on users.txt. Notice that jdoe and JDOE are treated as different. Now, modify the command to get a case-insensitive list of unique usernames.

  Solution

  
  
  
  # We can make sort case-insensitive with -f (fold case)
  # or uniq with -i (ignore case). Using uniq -i is common.
  sort users.txt | uniq -i
  

Conclusion

Congratulations! You've now mastered the essential tools for text processing on the Linux command line: grep (find), sed (replace), sort (order), and uniq (deduplicate). By chaining these commands together with pipes, you can perform surprisingly complex data analysis tasks with just a single line of code.

Key Takeaways:

• sort: Arranges lines of text. Use -n for numbers and -r for reverse order.
• uniq: Removes adjacent duplicate lines.
• The Golden Pipeline: sort | uniq is the standard pattern to get a truly unique list of lines from a file.
• Data Analysis: Use uniq -c to count occurrences and pipe that to sort -nr to find the most frequent items—a powerful technique for analyzing logs and data.

This lesson concludes our module on Text Manipulation. You're now equipped with the fundamental skills to handle text data like a pro. In the next module, we'll shift gears from processing text to managing your system's software with Fedora Package Management using DNF.