Looping Through Files, Directories, and Command Output

1. The for Loop: Iterating Over a List

A for loop is your go-to tool when you have a known, finite set of items you want to process one by one. This "set of items" could be a list of numbers, a series of filenames, or the output of another command.

The basic structure is:

for variable_name in item1 item2 item3; do



    # Commands to execute for each item
    echo "Processing $variable_name"
done

Let's watch a video that introduces the concept and syntax.

Iterating Over Files with Globbing

Manually typing out a list is fine for simple examples, but the real power of for loops comes from using shell "globbing" patterns (wildcards like *) to generate the list of items for you. This is how you iterate over files and directories.

For example, to process every file in the current directory, you could write:

for file in *; do
    echo "Found file: $file"
done

The * is expanded by the shell into a list of all filenames in the directory, which the for loop then iterates through.

A fantastic practical application of this is to automate system administration tasks, such as archiving log files.

An Important Best Practice: What happens if your globbing pattern (e.g., *.log) doesn't match any files? By default, the loop will run once with the literal string *.log as the item, which is usually not what you want. To prevent this, you can set a shell option.

2. The while Loop: Looping Based on a Condition

Unlike a for loop, a while loop doesn't iterate over a fixed list. Instead, it continues to execute a block of code as long as a certain condition remains true. This is ideal when you don't know in advance how many times you need to loop.

The basic structure uses the same test expressions you learned about in the if/else lesson:

while [[ condition ]]; do



    # Commands to execute as long as the condition is true
done

Let's watch a video that breaks this down.

Reading Input Line-by-Line

One of the most powerful and common uses of a while loop is to read a file or the output of a command line by line. This pattern is essential for processing data, log files, or lists of items generated by other tools.

The standard, robust syntax for this is:

while IFS= read -r line; do
    echo "Processing line: $line"
done < /path/to/your/file.txt

This might look complex, but it's a vital pattern to learn:

• while ... : The loop structure.
• IFS=: This prevents the shell from splitting lines on spaces and tabs.
• read -r: The read command reads one line of input. The -r option prevents it from mangling backslashes.
• line: The variable that will hold the content of each line.
• < file.txt: This redirects the content of file.txt into the while loop's input.

You can also pipe the output of another command directly into a while loop. This is incredibly powerful for combining tools. For example, you can use find to locate specific files and then process each one with a while loop.

Do-It-Yourself: Create a Backup & Cleanup Script

Now it's time to apply what you've learned. You will create a script named cleanup.sh that automates the process of archiving old log and temporary files. This is a very common task for system administrators.

Setup Your Test Environment:

First, run these commands in your terminal to create a directory and some dummy files to work with:

# Create a directory to work in
mkdir ~/automation_practice
cd ~/automation_practice




# Create some files with different extensions
touch app-2024-01.log
touch app-2024-02.log
touch tempfile_123.tmp
touch data.csv
touch important_notes.txt

Your Task:

Create a cleanup.sh script inside the ~/automation_practice directory. The script must perform the following actions:

1. Define a target directory: Create a variable TARGET_DIR and set it to the current directory (.).
2. Define an archive directory: Create a variable ARCHIVE_DIR and set it to archive.
3. Check for the archive directory: Use an if statement (like you learned in the last lesson) to check if the $ARCHIVE_DIR exists within $TARGET_DIR. If it does not, the script should create it and print a message like "Created archive directory.".
4. Use a for loop: Write a for loop that iterates over all files ending in .log and .tmp in the $TARGET_DIR.
5. Archive the files: Inside the loop, for each file found, the script should:
   • Print a message, e.g., Archiving app-2024-01.log...
   • Move the file into the $ARCHIVE_DIR.
6. Add shopt -s nullglob at the beginning of your script to handle cases where no log or temp files are found.

How to Test Your Script:

1. Run ./cleanup.sh. You should see messages about the archive directory being created and the .log and .tmp files being archived.
2. Use ls and ls archive to verify that the files have been moved correctly.
3. Run ./cleanup.sh again. This time, it shouldn't create the directory again, and it should find no files to archive.

#!/usr/bin/env bash




# Handle cases where no files match
shopt -s nullglob

TARGET_DIR="."
ARCHIVE_DIR="archive"




# 1. Check if the archive directory exists. If not, create it.
if [[ ! -d "$TARGET_DIR/$ARCHIVE_DIR" ]]; then
    echo "Creating archive directory..."
    mkdir "$TARGET_DIR/$ARCHIVE_DIR"
fi

echo "Searching for .log and .tmp files to archive..."




# 2. Loop through all .log and .tmp files
for file in "$TARGET_DIR"/*.log "$TARGET_DIR"/*.tmp; do



    # 3. Print message and move the file
    echo "Archiving $file..."
    mv "$file" "$TARGET_DIR/$ARCHIVE_DIR/"
done

echo "Cleanup complete."

# This would replace your 'for' loop
find "$TARGET_DIR" -maxdepth 1 -type f \( -name "*.log" -o -name "*.tmp" \) -print0 | while IFS= read -r -d '' file; do
    echo "Archiving $file..."
    mv "$file" "$TARGET_DIR/$ARCHIVE_DIR/"
done

Conclusion

Excellent work! You have now mastered two fundamental building blocks of automation: for loops for iterating over known lists and while loops for repeating actions based on a condition. You've also seen how to combine these loops with other commands like tar, mv, and find to create practical and powerful scripts.

Key Takeaways:

• for loops are for iterating through a predefined set of items, often generated by filename globbing (e.g., *.txt).
• while loops are for executing code as long as a condition is true, perfect for tasks where the number of iterations is unknown.
• The while IFS= read -r line pattern is the standard, robust way to process files or command output line by line.
• Combining loops with conditionals (if) and other command-line tools is the key to creating sophisticated automation scripts.

In our next lesson, we will explore functions. Functions allow you to organize your code into reusable, named blocks. Instead of rewriting your cleanup logic everywhere you need it, you could simply define a cleanup_logs function and call it whenever necessary, making your scripts much cleaner, more readable, and easier to maintain.