Sed for Stream Editing

1. What is a Stream Editor?

Unlike a text editor like vim or nano that you use interactively, sed is non-interactive. It reads text from an input stream (either a file or piped data), applies a set of commands to each line, and then sends the modified result to an output stream (by default, your terminal screen).

This line-by-line processing model makes sed incredibly efficient and perfect for use within scripts where you need to make changes programmatically without manual intervention.

To get started, let's explore the fundamental concepts of how sed works.

As you saw in the reading, running sed with an empty command (sed '' BSD) simply prints the file's content, much like the cat command. The real power of sed is unlocked when we start giving it commands to execute.

2. Filtering: Selecting Which Lines to Process

Before you can modify text, you often need to select which lines you want to affect. sed allows you to do this using addresses. An address can be a single line number, a range of lines, or a pattern to search for.

Two fundamental commands for filtering are:

• d: Delete the lines that match the address.
• p: Print the lines that match the address. When using p, you should almost always use the -n command-line option, which suppresses sed's default behavior of printing every line.

Let's continue with the DigitalOcean tutorial to see how this works.

A key takeaway from the reading is that sed does not modify the original file by default. It sends the transformed output to standard output. To save the changes, you can redirect the output to a new file:

sed '/pattern/d' input.log > cleaned.log

You can also edit "in-place" with the -i flag, but this is destructive. It overwrites the original file.

Warning: Use the -i flag with extreme caution. It's best practice to first run your sed command without -i to verify the output. When you are ready to edit in-place, consider creating a backup simultaneously by providing an extension, like sed -i.bak 's/old/new/' file.conf. This will perform the edit on file.conf and save the original content in file.conf.bak.

3. Substitution: Finding and Replacing Text

The most common use of sed is for substitution using the s command. Its basic syntax is 's/find/replace/flags'. This is the workhorse of sed and is invaluable for tasks like updating configuration settings in a script.

Let's look at a simple example. Here, we first use grep to see the original lines, and then use sed to replace "Software" with "SOFTWARE_JOB".

The real power comes from the flags and regular expressions you can use.

As the reading explains, you can replace a specific occurrence of a pattern. For instance, the command sed 's/Bash/Shell/2' file.txt would only replace the second time "Bash" appears on each line.

Using Variables in sed

In your scripts, you'll often need to use variables in your sed command. Since single quotes (') prevent variable expansion, you must use double quotes (").

#!/bin/bash

OLD_VALUE="dev"
NEW_VALUE="prod"
CONFIG_FILE="settings.conf"

echo "server_env=${OLD_VALUE}" > ${CONFIG_FILE}

echo "Original file:"
cat ${CONFIG_FILE}

# Use double quotes to allow variables to be expanded
sed -i "s/server_env=${OLD_VALUE}/server_env=${NEW_VALUE}/" ${CONFIG_FILE}

echo "Modified file:"
cat ${CONFIG_FILE}

Conclusion

You've now added sed to your command-line arsenal. You've learned how to filter data streams and perform powerful find-and-replace operations. This skill is fundamental for automating the modification of text-based files, a cornerstone of configuration management and log analysis in DevOps.

Key Takeaways:

• sed is a stream editor that processes text line-by-line from standard input or a file.
• By default, sed prints its output to the screen and does not alter the original file. Use output redirection (>) to save to a new file or the -i flag to edit in-place (with caution!).
• You can filter lines using addresses (line numbers, ranges, or patterns) combined with commands like d (delete) and p (print with -n).
• The s/find/replace/flags command is the most common. Key flags include g (global) and i (case-insensitive).
• You can use & to reference the entire matched pattern and \1, \2, etc., to reference captured groups for complex transformations.

In our next lesson, we will explore another giant of text processing: awk. While sed excels at whole-line operations, awk is designed to work with structured, field-based data (like columns in a log file), making it the perfect companion to sed.