Constructing Shell Pipelines with Redirection and Text Processing

Hello! In the previous lesson, you protected files by checking ownership and permissions. Now you’ll work with their contents. This lesson shows how to save command output, keep errors separate, and combine small text-processing commands into useful pipelines. You’ll use a local, invented service log—no AWS account or credentials are needed. Allow about 40–45 minutes.

---

Give command output a destination

A command normally reads from standard input and writes results to standard output. It can also write diagnostic messages to standard error. Output and errors usually appear together in your terminal, but they are separate streams. That distinction lets you save a report without accidentally treating an error message as report data.

Shawn Powers demonstrates the streams and the most common redirections in Linux PIPES and REDIRECTS: Command Line Ninja Skillz. Watch the selected portions, then try them in a disposable directory.

For a concise written reference, read the “Standard Output” section of William E. Shotts Jr.’s Learning the shell – Lesson 7: I/O Redirection.

Open Ubuntu. Make a new sandbox under your Linux home directory; if the proposed name already exists, choose another name and use it throughout. Do not continue until pwd shows the new directory.

cd ~/cloud-labs
mkdir pipelines-practice && cd pipelines-practice
pwd
umask 077

The umask sets restrictive starting permissions for files you create in this shell. It is a useful habit for operational output, which might otherwise contain information you did not intend to share. For this lab, create an entirely fictional log:

printf '%s\n' \
  '2026-03-01 INFO api ready' \
  '2026-03-01 ERROR api timeout' \
  '2026-03-01 WARN worker retry' \
  '2026-03-01 ERROR api timeout' \
  '2026-03-02 INFO worker ready' \
  '2026-03-02 ERROR worker denied' \
  '2026-03-02 INFO api ready' > events.log

cat events.log
ls -l events.log

printf writes the seven lines to standard output; > sends that output to events.log instead of the terminal. cat reads the file back. With the restrictive umask, a newly created file should normally be readable and writable only by its owner, just as you practiced previously.

Try saving a small report and then extending it:

printf '%s\n' 'Service event report' > report.txt
printf '%s\n' 'Source: fictional lab log' >> report.txt
cat report.txt

The first command creates or overwrites report.txt. The second adds a line without discarding the first. Before using > on a filename in real work, check that you are comfortable replacing its contents. In particular, never use the same file as both the input and redirected output of a command such as sort < events.log > events.log: the shell can truncate the output file before the command reads its input.

---

Connect commands instead of saving every intermediate file

The pipe character, |, sends one command’s standard output into another command’s standard input. Unlike >, it does not name a destination file. A pipeline is useful when you want to inspect or transform results without creating a succession of temporary files.

Watch the short pipe demonstration, then read the “Standard Input,” “Pipelines,” and “Filters” sections of the written reference.

Here is file input and file output in one command:

sort < events.log > sorted-events.log
cat sorted-events.log

< events.log supplies the file as standard input. sort prints its result to standard output, which > sorted-events.log saves. The original events.log remains unchanged. Many tools, including sort, also accept a filename directly; the < form makes the stream behavior explicit.

Now filter the original log for error lines:

grep -F ' ERROR ' events.log
grep -F ' ERROR ' events.log | wc -l

grep emits only matching lines. Its -F option treats the search text as a literal string rather than a regular-expression pattern; the spaces around ERROR help distinguish the log level from part of another word. wc -l counts the lines it receives, so the second command prints 3. The pipe passes grep’s matches to wc, not to a file.

For a quick look at opposite ends of the log, use head -n 2 events.log and tail -n 2 events.log. Each displays two lines; tail is especially useful when the newest log entries are at the end. These commands can also receive piped input—for example, grep -F ' ERROR ' events.log | tail -n 2 displays the last two matching lines.

A pipe normally carries standard output only. If the first command reports an error on standard error, that message is not automatically input to the next command. You’ll make that separation visible shortly.

---

Turn a log into a small operational summary

A useful pipeline answers a specific question at each stage. Suppose you need to know which service produced the error entries. In this fictional log, fields are separated by single spaces: date is field 1, level field 2, service field 3, and message field 4. First select error lines, then extract service names, group identical names, and count them.

The selected portions of Bash Scripting Course – sed, awk, and grep (pt28) demonstrate field extraction and the sort/uniq combination. The presenter uses a system account file as sample data; you’ll apply the same techniques only to your fictional log.

Build the summary in your sandbox:

grep -F ' ERROR ' events.log | cut -d' ' -f3 | sort | uniq -c | sort -nr

The expected result has api: 2 and worker: 1, with api first. Read the command in stages:

1. grep keeps the three error lines.
2. cut -d' ' -f3 selects the third space-delimited field: the service.
3. sort puts equal service names beside one another.
4. uniq -c counts adjacent equal lines. Without the preceding sort, separated occurrences of api would not form one group.
5. sort -nr orders the numeric counts from largest to smallest.

The fields in this lab are deliberately regular. cut -d' ' expects that single-space delimiter; logs with variable whitespace or a different structure may need another approach. One alternative is awk, which splits fields on whitespace by default:

awk '$2 == "ERROR" {print $3}' events.log | sort | uniq -c

Within the single-quoted awk program, $2 and $3 mean the second and third input fields, not Bash script arguments. Here, awk performs both the level check and field selection. Choose the form whose assumptions are clearest for the data you actually have.

Another filter, sed, can transform a stream without changing its source file:

sed 's/timeout/slow response/g' events.log | grep -F 'ERROR'
grep -F 'timeout' events.log

The first command displays error lines with timeout replaced in its output; the second confirms that events.log still contains the original text. This distinction matters when preparing a report: filtering or transforming output is not the same as editing the source.

---

Preserve results without hiding failures

To see why standard error deserves separate handling, deliberately ask ls for one file that exists and one that does not:

ls events.log missing.log > found.txt 2> errors.txt
cat found.txt
cat errors.txt

The ls command reports a failure, but that failure is intentional here. found.txt receives standard output—the existing filename—while errors.txt receives the diagnostic about missing.log. The 2 in 2> designates standard error. A pipeline that counts the lines in found.txt would not, by itself, tell you that ls also failed.

If you specifically need one combined record, use:

ls events.log missing.log > combined.txt 2>&1
cat combined.txt

Here, > combined.txt directs standard output to the file; 2>&1 then directs standard error to the same current destination as standard output. Order matters in this form. Combining streams can be useful for a diagnostic record, but keep them separate when error messages would corrupt data you intend to count or parse.

Finally, tee lets you retain an intermediate result and pass it to another command:

grep -F ' ERROR ' events.log | tee error-lines.txt | wc -l
cat error-lines.txt

You should see the count 3, while error-lines.txt retains the three matching lines. Like >, plain tee replaces its target file; tee -a appends. Before saving real infrastructure output, consider whether it contains credentials or other sensitive values. Restricted file permissions help, but they do not make saving a secret automatically safe.

---

You can now choose a destination for output with >, >>, or <; connect filters with |; distinguish normal results from errors with 2>; and build a pipeline that selects, extracts, sorts, and counts log data. The central habit is to know what each command receives and what it emits before adding another stage.

In the next lesson, you’ll put commands like these into a Bash script that accepts arguments and handles variables safely.