Sed for Stream Editing

1. Introduction to sed and Basic Substitution

sed works by reading text from a file or standard input, processing it one line at a time according to a script of commands you provide, and then printing the result to standard output. While it has many commands, its most common use by far is for substitution.

Let's start with a video that introduces the core concept and syntax.

As the video demonstrates, the basic syntax for substitution is:

sed 's/pattern/replacement/' filename

Let's break this down:

• s: This is the substitute command.
• /: This is the delimiter. It separates the parts of the command. While / is traditional, you can use almost any other character, which we'll see is very useful later.
• pattern: The text you want to find.
• replacement: The text you want to substitute in.

You may have noticed in the video's example that sed only replaced the first occurrence of the pattern on each line. To replace every occurrence on a line, you need to add the global flag, g, at the end of the command.

sed 's/pattern/replacement/g' filename

2. Saving Changes and Handling Special Characters

By default, sed acts as a filter, printing the modified text to your terminal. This is safe, as it leaves your original file untouched. But what if you want to save the changes? You have two main options.

Option 1: Output Redirection

You can redirect the standard output of sed into a new file.

sed 's/old/new/g' original_file.txt > new_file.txt

This is a safe way to work, as your original file remains intact.

Option 2: In-place Editing

sed can also modify the file directly using the -i (in-place) option. This is very powerful but should be used with care, as it overwrites your original data.

Let's see how this works.

A best practice when using -i is to create a backup of the original file at the same time. You can do this by providing an extension to the -i flag.

sed -i.bak 's/old/new/g' filename.txt

This command will:

1. Create a copy of filename.txt named filename.txt.bak.
2. Perform the substitution and save the changes to the original filename.txt.

Handling Delimiter Conflicts

What happens if your search pattern contains a forward slash, like a file path or a URL?

sed 's/usr/local/bin/\/usr/bin/g' config.txt <- This is confusing and will fail!

The slashes in the path are misinterpreted as delimiters. You could escape them with backslashes (\/), but a much cleaner solution is to use a different delimiter.

By choosing a character that is not in your pattern or replacement, such as #, |, or _, you make your command much more readable.

sed 's#/usr/local/bin#/usr/bin#g' config.txt

3. Advanced Pattern-Based Replacements

So far, we've replaced simple, literal strings. The true power of sed comes from using more complex patterns and referencing parts of what you found.

For this section, a detailed written guide is helpful.

Let's summarize those key advanced features:

• Case-Insensitive (i flag): To replace a word regardless of its case (e.g., "error", "Error", "ERROR"), add the i flag.
  sed 's/error/warning/gi' logfile.txt (This uses both global and case-insensitive flags)

• Referencing the Match (&): The & character in the replacement string is a special character that represents the entire text that was matched by your pattern. This is great for adding prefixes, suffixes, or wrapping text.
  Example: To wrap every instance of the word "Linux" in quotes:
  sed 's/Linux/"&"/g' my_document.txt

• Back-References (\1, \2, etc.): This is one of sed's most powerful features. By wrapping parts of your search pattern in $...$, you create "capture groups." You can then refer to the text captured by these groups in your replacement string using \1 for the first group, \2 for the second, and so on.
  Example: To swap the first two words of every line:
  sed 's/$[^ ]*$ $[^ ]*$/\2 \1/' filename.txt

  • $[^ ]*$: This is the first capture group (\1). It matches a sequence of non-space characters (a word).
  • : Matches the space between the words.
  • $[^ ]*$: This is the second capture group (\2).
  • \2 \1: The replacement pattern, which puts the second captured word first, followed by a space, and then the first captured word.

Conclusion

In this lesson, you have learned the fundamentals of sed, the command-line stream editor. You now have the power not just to select lines and fields, but to perform intricate search-and-replace operations within the text itself.

Key Takeaways:

• sed is a stream editor that processes text line by line.
• The primary command is substitute: s/pattern/replacement/flags.
• Common flags include g for global replacement on a line and i for case-insensitive matching.
• Changes can be saved by redirecting output (>) or editing in-place with -i. Using -i.bak is a safe way to edit in-place while creating a backup.
• When your pattern contains slashes, use a different delimiter (e.g., s|...|...|) for clarity.
• You can reference the matched text in the replacement with & (the whole match) or \1, \2 (captured groups).

You are building a formidable toolkit for text manipulation. We started with find, moved to cut and sort for structural changes, and now have sed for content changes. In our next lesson, we will explore awk, a tool that combines the field-awareness of cut with the pattern-matching of sed and adds its own programming language for even more advanced data extraction and reporting.