Mastering Pipes and Redirection

The Three Standard Streams

Before we can redirect anything, we need to understand what we're redirecting. In Linux, every process you run automatically gets three communication channels, called standard streams.

• Standard Input (stdin): The channel for feeding data into a program. By default, this is your keyboard. Its file descriptor number is 0.
• Standard Output (stdout): The channel for the program's normal output. By default, this is your terminal screen. Its file descriptor number is 1.
• Standard Error (stderr): A separate channel for error messages. By default, this also goes to your terminal screen, which is why you see errors and normal output mixed together. Its file descriptor number is 2.

The video below provides a great conceptual overview of these streams.

This diagram provides a great visual summary of how these streams work.

Redirecting Output to Files

Now that you know about stdout and stderr, let's learn how to control where they go. Instead of printing to the screen, we can send them to a file. This is called output redirection.

Redirecting Standard Output (stdout)

You can redirect the normal output of a command using two main operators:

• >: Redirects stdout to a file. Warning: This will overwrite the file if it already exists.
• >>: Redirects stdout and appends it to the end of the file. This will not delete existing content.

The following video demonstrates this in practice.

Redirecting Standard Error (stderr)

What if you only want to capture the errors? This is extremely useful for debugging. Since stderr is stream 2, we explicitly tell the shell to redirect it using 2>.

• 2>: Redirects stderr to a file (overwrites).
• 2>>: Redirects stderr and appends it to a file.

This next segment of the video shows you exactly how to do that.

Redirecting Both stdout and stderr

Sometimes you want to capture everything—both normal output and errors—in a single log file. There are a couple of ways to do this. A traditional way is > log.txt 2>&1, but since you are on a modern Fedora system using the Bash shell, there's a simpler syntax.

The article below from Fedora Magazine is perfect for learning the modern, preferred way to handle this.

Chaining Commands with Pipes

Redirection is powerful, but the true magic happens when you connect commands to each other using pipes. The pipe operator, |, takes the standard output of the command on its left and uses it as the standard input for the command on its right.

This allows you to build sophisticated workflows by combining small, single-purpose utilities. For example, instead of a single ls command that tries to do everything (listing, sorting, filtering, paging), you can use ls to list, grep to filter, sort to sort, and less to page the output.

Your Turn: Practice with Pipes and Redirects

Time to put this theory into practice! Open your terminal and let's get hands-on.

1. Set up your practice area:
First, navigate to your home directory and create a workspace.

cd ~
mkdir -p command_chaining_practice
cd command_chaining_practice

2. Redirection Practice:

• Save System Info: Run the lscpu command, which prints your CPU architecture information. Save this output to a file called system_info.txt.

  lscpu > system_info.txt
  

• Capture an Error: Now, try to cat a file that doesn't exist and redirect the resulting error message into error_log.txt. Notice that nothing appears on your screen.

  cat non_existent_file.txt 2> error_log.txt
  

  Verify the files were created and have the correct content:

  cat system_info.txt
  cat error_log.txt
  

• Append More Info: Let's add memory information to our system_info.txt file using the append operator >>.

  free -h >> system_info.txt
  

  Now view the file. You should see the CPU info followed by the memory info.

  cat system_info.txt
  

3. Piping Practice:

• Find Python-related Processes: This is a very common task for a developer. List all running processes on your system and use grep to filter for any lines containing "python". This is how you could check if your Django development server is running.

  ps aux | grep python
  

  (Note: You might see the grep command itself in the output. We'll learn how to filter that out in a later lesson!)

• Chain Multiple Commands: Let's create a more complex chain. We'll list the contents of the /usr/bin directory (which contains many executable programs), count how many items there are, and save that count to a file.
  ls /usr/bin - lists the files.
  wc -l - counts the number of lines it receives as input.

  ls /usr/bin | wc -l > file_count.txt
  cat file_count.txt
  

• Challenge: Find the 10th file alphabetically in your /etc directory.
  Hint: You'll need ls, sort, head, and tail all piped together!

  Click to reveal the solution

  ls /etc | sort | head -n 10 | tail -n 1
  

  Explanation: ls /etc lists the files, sort sorts them alphabetically, head -n 10 takes the first 10 from that sorted list, and tail -n 1 takes the last one from that group of 10, which is the 10th file overall.

Conclusion

Excellent work! You've just learned one of the most transformative concepts of the Linux command line. By understanding and using pipes and redirection, you can move beyond single commands and start building powerful, customized data-processing workflows.

Key Takeaways:

• Standard Streams: Every command has an input (stdin), output (stdout), and error (stderr) stream.
• Output Redirection:
  • > overwrites a file with a command's stdout.
  • >> appends stdout to a file.
• Error Redirection:
  • 2> overwrites a file with stderr.
  • 2>> appends stderr to a file.
  • &> redirects both stdout and stderr (modern Bash).
• Pipes (|): The | operator chains commands by sending the stdout of one command to the stdin of the next.

You got a glimpse of the grep command today. It's an incredibly useful tool for filtering text. In our next lesson, we will dive deep into grep and regular expressions, unlocking the ability to search for complex patterns in files and command output—an essential skill for any developer sifting through code or logs.