`sed` for Basic Text Substitution

1. The Core sed Command: Substitution

The most common use of sed is for substitution. The basic syntax is straightforward and powerful:

sed 's/pattern/replacement/' [file]

Let's break this down:

• s: This is the substitute command.
• /: This is the delimiter. It separates the parts of the command. While / is traditional, you can use other characters, which we'll see later.
• pattern: The text or regular expression you want to find.
• replacement: The text you want to replace the pattern with.
• [file]: The input file. If omitted, sed reads from standard input, making it perfect for pipelines.

A critical point to understand from the start is that sed, by default, does not modify your original file. It reads the file, performs the substitution, and prints the modified result to your terminal (standard output).

Let's watch a short video to see this fundamental concept in action.

2. Controlling the Output: In-Place Editing and Flags

Printing to the terminal is safe, but often you want to save your changes directly back to the file. This is called in-place editing. You can also control how sed substitutes text on each line.

In-Place Editing with -i

To tell sed to modify the file directly, you use the -i (in-place) option.

⚠️ A Word of Caution: Using sed -i will overwrite your original file. There is no "undo." A much safer practice, especially when you're learning or working on critical system files, is to create a backup at the same time. You can do this by providing a suffix to the -i option, like this:

sed -i'.bak' 's/pattern/replacement/' file.txt

This command will:

1. Create a backup copy of your original file named file.txt.bak.
2. Apply the substitution and save the changes to the original file.txt.

This way, if you make a mistake, you can always restore from the .bak file.

Substitution Flags

By default, sed's substitute command only replaces the first match it finds on each line. To change this behavior, you can add flags after the final delimiter:

• g (global): Replaces all occurrences of the pattern on a line, not just the first.
  sed 's/old/new/g' file.txt
• i (ignore case): Performs a case-insensitive match (this is a GNU extension, but standard on most Linux systems).
  sed 's/old/new/i' file.txt
• A number: Replaces only the Nth occurrence.
  sed 's/old/new/2' file.txt (replaces only the second match)

You can combine these flags, for example, gi for a global, case-insensitive replacement.

Let's watch a video that brings these concepts together.

For a solid textual reference on these core features, the DigitalOcean guide is excellent.

3. Practical Techniques for Real-World Use

Now let's cover two techniques that will solve common problems you'll encounter when using sed.

Changing the Delimiter

What happens if you need to replace a file path that contains slashes, like /var/log/?

sed 's//var/log///var/data/' config.ini

This command will fail because sed gets confused by the extra slashes. You could "escape" each slash with a backslash (\/), but this becomes messy and hard to read.

A much cleaner solution is to use a different delimiter. The character immediately following the s becomes the new delimiter. Common choices include |, #, or :.

sed 's|/var/log/|/var/data/|' config.ini

This is far more readable and is standard practice when working with file paths or URLs.

Re-using the Matched Pattern with &

Sometimes, you don't want to completely replace a pattern, but rather add something to it. For instance, you might want to wrap every instance of the word "ERROR" in square brackets to make it [ERROR].

You could do this with sed 's/ERROR/[ERROR]/', but sed provides a more elegant way: the ampersand &. In the replacement string, & is a special character that represents the entire matched pattern.

So, to wrap "ERROR" in brackets, you would write:

sed 's/ERROR/[&]/g'

This is incredibly useful when your pattern is a complex regular expression and you don't want to type it out again in the replacement section.

Finally, remember that sed is a stream editor. It works beautifully in pipelines, receiving input from one command and passing its modified output to the next.

Hands-On Practice

Time to put theory into practice. Imagine you are managing a web server and need to make some configuration changes.

Create a practice file named nginx.conf with the following content:

# Server configuration
server {
    listen 80;
    server_name example.com;
    
    root /var/www/html/test_site;
    index index.html;
    
    location / {
        try_files $uri $uri/ =404;
    }
}

Now, construct the single sed command to accomplish each of the following tasks. Test your command first to see the output, then add the appropriate option to modify the file in-place (with a backup!).

1. The server name is changing from example.com to hacktheplanet.net. Perform this substitution.
2. The web root directory needs to be changed from /var/www/html/test_site to /var/www/html/prod_site. (Hint: You'll need to change the delimiter to avoid a syntax error).
3. You want to comment out the listen 80; line by adding a # at the beginning of that line. (Hint: Use & to add a prefix to the matched line).
4. Challenge: Let's say you want to change all underscores _ in the file to hyphens -. Perform a global replacement to accomplish this.

After running your commands, use cat nginx.conf to check your work and ls to verify that your backup files were created.

Conclusion

Great job! You have now learned the fundamentals of one of the most powerful and ubiquitous text-processing tools in the Linux world. While grep lets you find lines and cut lets you extract columns, sed gives you the power to modify content dynamically.

Key Takeaways:

• The core substitution command is sed 's/pattern/replacement/flags'.
• sed prints to standard output by default. Use the -i option for in-place editing.
• Always use a backup suffix (-i.bak) for in-place edits on important files.
• Use the g flag for global replacement on a line and i for case-insensitivity.
• When your pattern contains slashes (/), avoid messy escaping by choosing a different delimiter (e.g., s|pattern|replacement|).
• Use & in the replacement string as a shorthand for the entire matched pattern.

Next Lesson Preview:
sed is fantastic for stream-based search and replace. But what if your logic needs to be more aware of the data's structure? For instance, what if you only want to perform an action on lines where the third column is a specific value? Or what if you want to perform calculations based on the numbers in different columns? For that level of field-aware processing, we need a more advanced tool: awk. In the next lesson, we will explore how awk allows you to write small but powerful programs that process text field by field.