File I/O Operations

1. The Core Packages: os and io

Go's standard library provides two main packages for file operations:

• os: This is your primary toolkit for interacting with the operating system. It contains functions to open, create, read, write, and remove files, as well as get file metadata.
• io: This package provides the fundamental interfaces for I/O operations. The two most important are io.Reader and io.Writer.

Understanding io.Reader and io.Writer is crucial because they provide a powerful abstraction. Many things in Go can be treated as a source of bytes (a Reader) or a destination for bytes (a Writer), not just files. This includes network connections, in-memory buffers, and standard input/output. An *os.File object, which you get when you open a file, happens to implement both interfaces.

The following video from the Code & Learn channel gives a clear explanation of these two fundamental interfaces.

2. Writing and Creating Files

Let's start by creating and writing to a file. This is a common task for generating configuration, logging output, or creating reports.

A great way to get a practical overview of all basic file operations is by watching the following video. We will refer back to it throughout the lesson.

Let's summarize the key functions for writing:

Creating and Overwriting with os.Create

The simplest way to create a file is with os.Create(path).

package main

import (
	"log"
	"os"
)

func main() {
	file, err := os.Create("config.txt")
	if err != nil {
		log.Fatalf("failed to create file: %s", err)
	}
	defer file.Close() // CRITICAL: Always close the file!

	_, err = file.WriteString("API_KEY=mysecretkey\n")
	if err != nil {
		log.Fatalf("failed to write to file: %s", err)
	}
}

Key Points:

• os.Create opens a file for writing. If the file doesn't exist, it's created. If it already exists, its contents are erased (truncated).
• defer file.Close() is the idiomatic way to ensure a file is closed when the function exits, whether it completes successfully or panics. This releases the file handle back to the OS. Forgetting to close files can lead to resource leaks.
• file.WriteString() and file.Write() (which takes a []byte) are used to write data.

Appending to a File with os.OpenFile

Often, you don't want to overwrite a file; you want to add to it. This is essential for log files. For this, you need the more powerful os.OpenFile function, which takes flags to control its behavior.

package main

import (
	"log"
	"os"
)

func main() {
	// Flags: Write-only, append data, create if it doesn't exist.
	flags := os.O_WRONLY | os.O_APPEND | os.O_CREATE
	// File permissions (0644 is standard: user can read/write, others can only read).
	permissions := 0644

	file, err := os.OpenFile("app.log", flags, os.FileMode(permissions))
	if err != nil {
		log.Fatalf("failed to open file: %s", err)
	}
	defer file.Close()

	_, err = file.WriteString("INFO: Application started successfully.\n")
	if err != nil {
		log.Fatalf("failed to write to log file: %s", err)
	}
}

The flags are combined using the bitwise OR operator |. The most common flags are detailed in this guide.

3. Reading Files

Now that we've created some files, let's read them. Your choice of reading method depends on the file size and what you need to do with the data.

Reading an Entire File

For small files like configurations or short scripts, the easiest way is to read the whole file into memory at once using os.ReadFile.

package main

import (
	"fmt"
	"log"
	"os"
)

func main() {
	data, err := os.ReadFile("config.txt")
	if err != nil {
		log.Fatalf("failed to read file: %s", err)
	}
	fmt.Println(string(data))
}

This is simple and effective, but not suitable for very large files (e.g., multi-gigabyte logs) as it could exhaust your system's memory.

Reading a File Line-by-Line

A more memory-efficient and common pattern, especially for processing log files or data streams, is to read a file line by line. The bufio package is perfect for this.

Here is a summary of the pattern:

package main

import (
	"bufio"
	"fmt"
	"log"
	"os"
)

func main() {
	file, err := os.Open("app.log")
	if err != nil {
		log.Fatalf("failed to open file: %s", err)
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	// scanner.Split(bufio.ScanLines) is the default.

	for scanner.Scan() {
		fmt.Println(scanner.Text()) // Print each line
	}

	if err := scanner.Err(); err != nil {
		log.Fatalf("error occurred during scanning: %s", err)
	}
}

This approach uses a scanner to read the file in manageable chunks, making it highly efficient for files of any size.

4. Other Essential File Operations

Beyond basic reading and writing, DevOps tools often need to manage files and their metadata.

Here's a quick summary of those operations:

• Check for Existence: Use os.Stat(path). This function returns file info and an error. If you want to specifically check if a file does not exist, use os.IsNotExist(err).

  if _, err := os.Stat("non_existent_file.txt"); os.IsNotExist(err) {
      fmt.Println("File does not exist.")
  }
  

• Get File Info: os.Stat(path) returns a FileInfo struct if the file exists, which contains useful metadata.

  info, err := os.Stat("app.log")
  if err != nil {
      log.Fatal(err)
  }
  fmt.Printf("File size: %d bytes\n", info.Size())
  fmt.Printf("Last modified: %s\n", info.ModTime())
  

• Copy a File: The most efficient way to copy a file is using io.Copy(destination, source). This beautifully demonstrates the power of the io.Writer and io.Reader interfaces.

  sourceFile, _ := os.Open("app.log")
  defer sourceFile.Close()
  
  destFile, _ := os.Create("app.log.backup")
  defer destFile.Close()
  
  bytesCopied, err := io.Copy(destFile, sourceFile)
  

• Delete a File: A straightforward call to os.Remove(path).

  err := os.Remove("app.log.backup")
  

Conclusion

You now have a solid grasp of the fundamental file system operations in Go. These are the building blocks you'll use constantly when writing automation scripts, configuration managers, and other DevOps tools.

Key Takeaways:

• The os package is your go-to for most file system interactions like creating, opening, reading, and deleting files.
• For simple, small files, os.ReadFile and os.WriteFile are convenient helpers.
• For larger files or line-by-line processing (like logs), use os.Open with a bufio.Scanner.
• os.OpenFile with flags gives you fine-grained control over file access, such as opening in append mode.
• defer file.Close() is a non-negotiable best practice to prevent resource leaks.
• The io.Reader and io.Writer interfaces provide a powerful abstraction that makes functions like io.Copy incredibly versatile.

Next Up

Now that your tools can read from and write to the file system, the next logical step is to have them interact with other programs. In our next lesson, we will learn how to execute and manage external commands, capturing their input and output streams. This will enable you to write Go programs that can orchestrate tools like git, docker, kubectl, or any other CLI application.