Stream Editing with Sed

The Basic Substitution

At its heart, sed works by applying a command to each line of input it receives. The most fundamental command is s for "substitute."

The syntax looks like this: sed 's/pattern/replacement/'

Let's break this down:

• s: The substitute command.
• /: A delimiter character. The slash is traditional, but we'll see later that you can use others.
• pattern: The text or regular expression to search for (just like in grep).
• replacement: The text to replace the pattern with.

By default, sed reads from standard input, performs the substitution, and prints the result to standard output, leaving the original source untouched.

Global, In-Place, and Case-Insensitive Changes

You probably noticed in the video that sed only replaced the first occurrence of "kubernetes" on each line. To replace all occurrences, we add the g flag for global replacement.

• Global Replacement (g): sed 's/pattern/replacement/g'

What if you actually want to modify the file itself? Redirecting output with > works, but sed has a more direct way: the -i flag for in-place editing.

• In-Place Editing (-i): sed -i 's/pattern/replacement/g' filename

⚠️ A Word of Caution: The -i flag modifies the file directly. There is no "undo." It's a common and good practice to create a backup of the original file. You can do this automatically by providing an extension to the -i flag, like so: sed -i.bak 's/pattern/replacement/g' filename. This will perform the replacement in filename and save the original content in filename.bak.

Finally, just like grep's -i flag, you can make sed's search case-insensitive. In GNU sed (which is standard on Fedora), you add the I flag.

• Case-Insensitive (I): sed 's/pattern/replacement/gI'

Let's see a video that covers saving output and making backups.

The Delimiter Problem and Its Solution

As a developer, you'll often need to replace file paths or URLs, like /home/user/project/ or http://example.com. What happens when your pattern contains the / delimiter?

You would have to "escape" every slash, which quickly becomes ugly and hard to read:
sed 's/http:\/\/old.com/http:\/\/new.com/g'

There's a much better way. sed allows you to use almost any character as a delimiter. Just be consistent. Using #, |, or ; is common.

sed 's#/old/path#/new/path#g'

This is much cleaner and is the preferred method for working with text containing slashes.

Unleashing Full Power with Back-references

Here is where sed truly shines, especially when combined with the regular expression knowledge you gained from the last lesson. You can capture parts of your search pattern and reuse them in the replacement string. This is done with capturing groups () and back-references \1, \2, etc.

• Anything matched inside the first () in your pattern is stored in \1.
• Anything in the second () is stored in \2, and so on.

This allows you to reformat text, not just replace it. For example, to swap the first and last name:

echo "Doe, John" | sed -E 's/(\w+), (\w+)/\2 \1/'
Output: John Doe

• (\w+) captures the first word ("Doe") into \1.
• , matches the comma and space.
• (\w+) captures the second word ("John") into \2.
• The replacement \2 \1 reorders them.

Note: To use capturing groups () without escaping them, it's best to use the -E flag for Extended Regular Expressions, just as you did with grep.

For a more detailed read on this powerful feature, please review the following guide.

Your Turn: Hands-on Practice

Time to put this into practice with a scenario you'll definitely encounter: modifying configuration files for a web application.

1. Setup
Open your terminal. First, let's create a practice directory and a mock Django settings file.

cd ~
mkdir -p sed_practice
cd sed_practice




# Create a mock Django project settings file
cat << 'EOF' > myproject/settings.py



# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = []

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'db.sqlite3',
    }
}




# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.2/howto/static-files/
STATIC_URL = 'static/'
EOF

2. Exercises

For each exercise, try to write the command yourself before checking the solution.

• Exercise 1: Simple Find and Replace
  For production, you must turn DEBUG off. Use sed to change DEBUG = True to DEBUG = False. The change should only be printed to the terminal, not saved to the file.

  Solution

  sed 's/DEBUG = True/DEBUG = False/' myproject/settings.py
  

• Exercise 2: In-place Editing
  Now, make that change permanent using in-place editing. Also, add your domain name "mydomain.com" to ALLOWED_HOSTS. The line should become ALLOWED_HOSTS = ['mydomain.com'].

  Solution

  
  
  
  # Note the use of single quotes inside double quotes for the shell
  sed -i "s/ALLOWED_HOSTS = \[\]/ALLOWED_HOSTS = ['mydomain.com']/" myproject/settings.py
  sed -i 's/DEBUG = True/DEBUG = False/' myproject/settings.py
  
  
  
  
  # You can verify with:
  cat myproject/settings.py
  

• Exercise 3: Changing the Delimiter
  You've decided to change your static files URL. Replace STATIC_URL = 'static/' with STATIC_URL = '/assets/v1/'. Since the pattern and replacement contain slashes, you'll need to use a different delimiter.

  Solution

  
  
  
  # Using '#' as the delimiter
  sed -i 's#STATIC_URL = '\''static/'\''#STATIC_URL = '\''/assets/v1/'\''#' myproject/settings.py
  
  
  
  
  # You can verify with:
  cat myproject/settings.py
  

• Exercise 4: Using Back-references for Refactoring
  Your database settings are changing. You need to transform the database NAME from 'db.sqlite3' to 'NAME': '/var/data/myproject.sqlite3'. Use a back-reference to capture the NAME key and part of the line.

  Solution

  
  
  
  # The -E flag allows us to use () without escaping
  # We capture "'NAME':" and replace it with itself (\1) plus the new path.
  sed -i -E "s/('NAME': ).*/\1'\/var\/data\/myproject.sqlite3',/" myproject/settings.py
  
  
  
  
  # Verify the change
  cat myproject/settings.py
  

• Exercise 5: Recursive Code Refactoring
  This is a classic developer task. Let's say you used a temporary variable name, temp_user_id, in your project, and now you want to rename it to customer_id. You need to do this across all Python files.

  First, let's create another file for the example:

  mkdir -p myproject/accounts
  cat << 'EOF' > myproject/accounts/utils.py
  def get_user(request):
      temp_user_id = request.session.get('user')
      return temp_user_id
  EOF
  

  Now, use find and sed together to replace all instances of temp_user_id with customer_id in all .py files inside the myproject directory.

  Solution

  find myproject -type f -name "*.py" -exec sed -i 's/temp_user_id/customer_id/g' {} +
  
  
  
  
  # Verify the changes in both files
  echo "--- settings.py ---"
  grep "customer_id" myproject/settings.py # Should show no results
  echo "--- accounts/utils.py ---"
  grep "customer_id" myproject/accounts/utils.py # Should show the changed line
  

  This find command is extremely powerful. It finds every file (-type f) ending in .py and executes sed on them. The {} + part efficiently passes the list of found files to sed.

Conclusion

Excellent work! You've just added sed to your command-line toolkit, moving from simply finding text with grep to actively manipulating it. This is a massive step forward in your ability to automate tasks and manage your development projects efficiently.

Key Takeaways:

• Core Syntax: sed 's/pattern/replacement/g' is the foundation of find-and-replace.
• In-Place Editing: The -i flag modifies files directly. Always be careful and consider using -i.bak to create a backup.
• Delimiter Flexibility: Don't be afraid to change your delimiter from / to # or | when working with paths and URLs.
• Back-references: Using () to capture parts of your pattern and \1, \2 to reuse them in the replacement is the key to complex text reformatting.
• Automation: Combining sed with find allows you to perform project-wide refactoring with a single command.

In our next lesson, we will explore sort and uniq. Now that you can find (grep) and modify (sed) text, you'll learn how to organize and de-duplicate it, another essential skill for processing logs and data.