Sed for Stream-Based Text Transformations

1. The Core of sed: Substitution

The most common use of sed is to substitute text. The command for this is s, and its basic syntax is s/pattern/replacement/flags. sed reads input line by line, applies your command, and prints the result to standard output.

Let's break down the key components.

# A simple example:
# This replaces the *first* occurrence of 'error' with 'warning' on each line
echo "An error occurred. This is a critical error." | sed 's/error/warning/'



# Output: An warning occurred. This is a critical error.

Notice only the first "error" was replaced. To change all instances on a line, you need the global flag (g).

echo "An error occurred. This is a critical error." | sed 's/error/warning/g'



# Output: An warning occurred. This is a critical warning.

This video provides an excellent walkthrough of sed's fundamental commands, with a great focus on substitution.

A common challenge in DevOps is dealing with file paths, which contain slashes (/). Since sed uses / as a delimiter by default, this creates a messy command requiring you to escape every slash.

sed 's/\/var\/log\/nginx/\/data\/logs\/nginx/g' config.file

A much cleaner way is to use an alternative delimiter. You can use almost any character after the s. Common choices are |, #, or @.

sed 's|/var/log/nginx|/data/logs/nginx|g' config.file

2. Deleting Lines

Substitution is about changing text within a line. The d command is for deleting an entire line. You can specify which lines to delete by number, a range, or, most powerfully, a pattern.

This is extremely useful for cleaning up configuration files or logs before processing them.

• Delete by line number: sed '3d' file.txt (deletes the 3rd line)
• Delete by range: sed '1,5d' file.txt (deletes lines 1 through 5)
• Delete lines matching a pattern: sed '/pattern/d' file.txt

Let's see this in a practical context. Imagine you want to process a configuration file but ignore all comments (lines starting with #) and all empty lines.

# Example config.conf
# This is a database configuration
db.host=localhost
db.port=5432




# Enable connection pooling
pool.enabled=true

You could clean this with two sed commands:

1. sed '/^#/d' config.conf removes comment lines. (^ matches the start of the line).
2. sed '/^$/d' config.conf removes empty lines. ($ matches the end of the line, so ^$ matches a line with nothing between the start and end).

You can combine these using a semicolon ; to perform both actions in one pass:

sed '/^#/d; /^$/d' config.conf



# Output:
# db.host=localhost
# db.port=5432
# pool.enabled=true

3. Inserting and Appending Lines

Besides substitution and deletion, sed can also add new lines to the stream.

• The i command inserts text before a matched line.
• The a command appends text after a matched line.

The syntax is sed '/pattern/i \new text' or sed '/pattern/a \new text'.

This is perfect for programmatically adding settings to a configuration file. For instance, to add a timeout setting right after db.port in our example config.conf:

sed '/db.port/a db.timeout=30s' config.conf



# Output:
# # This is a database configuration
# db.host=localhost
# db.port=5432
# db.timeout=30s
# 
# # Enable connection pooling
# pool.enabled=true

For a consolidated view of these core operations, refer to this excellent cheat sheet.

4. Practical Scenario: Editing a Configuration File

By default, sed prints its output to the console. To save the changes, you can either redirect the output to a new file or edit the file in-place using the -i flag.

⚠️ Warning: Using sed -i without a backup is destructive. It overwrites the original file. A safer practice is to create a backup by providing an extension to the -i flag, like sed -i.bak .... This performs the in-place edit on the original file and saves a copy of the original with a .bak extension.

Let's combine everything we've learned into a single, practical task. Imagine you have a docker-compose.yml file and you need to automate an update for a new environment.

Starting docker-compose.yml:

version: '3.8'
services:
  webapp:



    # image: myapp:1.0
    image: myapp:latest
    ports:
      - "80:8080"
    environment:
      - DB_HOST=localhost
      - FEATURE_FLAG_ALPHA=false

Your task is to write a single sed command that:

1. Changes the DB_HOST from localhost to postgres.prod.svc.cluster.local.
2. Uncomments the line for image: myapp:1.0 and comments out image: myapp:latest.
3. Changes FEATURE_FLAG_ALPHA from false to true.
4. Performs these changes in-place and creates a backup file named docker-compose.yml.bak.

Here is the solution. We use the -e option to chain multiple expressions.

sed -i.bak \
  -e 's/DB_HOST=localhost/DB_HOST=postgres.prod.svc.cluster.local/' \
  -e '/image: myapp:1.0/s/^# //' \
  -e '/image: myapp:latest/s/^/# /' \
  -e 's/FEATURE_FLAG_ALPHA=false/FEATURE_FLAG_ALPHA=true/' \
  docker-compose.yml

Let's dissect this powerful one-liner:

• sed -i.bak: Edit in-place and create a backup.
• -e 's/DB_HOST=...': A standard substitution for the database host.
• -e '/image: myapp:1.0/s/^# //': An addressed substitution. It first finds the line containing image: myapp:1.0 (/pattern/), then on that line only, it performs the substitution s/^# //, which finds a hash and a space at the beginning of the line (^# ) and replaces it with nothing.
• -e '/image: myapp:latest/s/^/# /': Similar logic, but this time it adds a comment by replacing the start of the line (^) with # .
• -e 's/FEATURE_FLAG_ALPHA=...': Another simple substitution.

This example showcases the kind of powerful, automated text manipulation that is central to DevOps. For more practical examples, the following article is an excellent resource.

Conclusion

You have now learned how to use sed to perform the three fundamental stream-editing operations. This tool allows you to programmatically modify files and data streams, a crucial capability for any DevOps engineer.

Key Takeaways:

• sed transforms text streams, distinguishing it from grep which only filters them.
• The most common commands are s (substitute), d (delete), i (insert), and a (append).
• Substitution flags like g (global) and i (case-insensitive) are essential.
• Use alternative delimiters (e.g., s|...|...|) to cleanly handle patterns containing slashes.
• Address your commands by line number (10s/.../) or pattern (/pattern/d) to target specific lines.
• The -i flag modifies files in-place; always consider using -i.bak for safety.

Preview of the Next Lesson:
sed is brilliant for line-oriented edits. But what if your data is structured in columns, like a CSV file or the output of ls -l? For that, we need a tool that understands fields. In our next lesson, we will explore awk, a powerful utility designed to extract and format structured, column-based text data.