Managing External Commands: Input and Output Streams

1. The Three Standard Streams: A Quick Refresher

Before we write any Go code, let's quickly revisit the fundamental concept that governs all command-line programs: the standard I/O streams. In Linux and other Unix-like systems, every process gets three default data streams:

• Standard Input (stdin): The stream for data flowing into the program.
• Standard Output (stdout): The stream for the program's primary, successful output.
• Standard Error (stderr): The stream for error messages and diagnostics.

Your proficiency in Bash means you've used these extensively, for example, when piping output from one command to another (ps aux | grep nginx) or redirecting errors to /dev/null (command 2> /dev/null).

Go's os/exec package gives us structured, programmatic access to these same three streams, along with the command's exit code. To refresh your memory on these concepts from a shell perspective, the following video provides a concise overview.

2. Executing Basic Commands with os/exec

The core of running external commands in Go is the os/exec package. The main entry point is the exec.Command function, which prepares a command for execution but doesn't run it immediately.

cmd := exec.Command("ls", "-la", "/var/log")

Here, exec.Command takes the command name as the first argument, followed by any additional arguments as separate strings. Go does not use a shell to run the command, so shell features like globbing (*) or pipes (|) won't work directly.

The article below provides excellent examples that we will build on.

Capturing Combined Output

For simple commands where you just need the output and don't need to distinguish between stdout and stderr, you can use cmd.CombinedOutput(). It runs the command and returns its combined output as a byte slice. If the command returns a non-zero exit code, it also returns an error.

package main

import (
	"fmt"
	"log"
	"os/exec"
)

func main() {
	cmd := exec.Command("git", "version")
	output, err := cmd.CombinedOutput()
	if err != nil {
		log.Fatalf("cmd.CombinedOutput() failed with %s\n", err)
	}
	fmt.Printf("Git version: %s", string(output))
}

Capturing stdout and stderr Separately

For any serious automation tool, you need to handle stdout and stderr independently. For example, you might want to log errors but parse the standard output. To do this, you can assign io.Writers to the Stdout and Stderr fields of the Cmd struct. The bytes.Buffer is a perfect in-memory io.Writer for this task.

After setting up the writers, you call cmd.Run(), which starts the command and waits for it to complete.

package main

import (
	"bytes"
	"fmt"
	"log"
	"os/exec"
)

func main() {
	// Let's try a command that will produce an error.
	cmd := exec.Command("ls", "/path/to/nonexistent/dir")

	var stdout, stderr bytes.Buffer
	cmd.Stdout = &stdout
	cmd.Stderr = &stderr

	err := cmd.Run() // .Run() starts the command and waits for it to complete.

	// Check for a non-zero exit code.
	if err != nil {
		// The error from .Run() is often of type *exec.ExitError,
		// which indicates the command ran but exited with an error.
		log.Printf("Command finished with error: %v", err)
		log.Printf("Stderr:\n%s", stderr.String())
		return
	}
    
	// This part will not be reached in this example, but would be for a successful command.
	fmt.Printf("Stdout:\n%s", stdout.String())
}

This pattern gives you complete control to inspect, log, or parse the different output streams from the commands you execute.

3. Handling Input and Streaming Output

Providing Input to a Command (stdin)

Just as you can capture output, you can also provide input. This is done by setting the Stdin field of the Cmd struct, which is an io.Reader. This beautifully links back to the interfaces you learned about in the last lesson. A simple strings.NewReader is often sufficient.

This is useful for commands that process data from stdin, like grep, sed, or even kubectl apply -f -.

Here's a condensed example:

package main

import (
	"bytes"
	"fmt"
	"log"
	"os/exec"
	"strings"
)

func main() {
	// A common DevOps task: filter some text data.
	inputData := "server_status=OK\nmetrics_p99=54ms\nserver_status=DEGRADED\n"
	cmd := exec.Command("grep", "DEGRADED")
	
	// Provide the input data to the command's stdin.
	cmd.Stdin = strings.NewReader(inputData)

	var stdout bytes.Buffer
	cmd.Stdout = &stdout

	if err := cmd.Run(); err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Found matching line: %s", stdout.String())
}

Streaming Output for Long-Running Commands

The Run() and Output() methods are blocking; they wait for the command to finish. This isn't suitable for long-running processes like ping, tail -f, or starting a server, where you want to see the output as it's generated.

For this, you use a two-step process: cmd.Start() and cmd.Wait().

• cmd.Start() launches the command in a non-blocking way.
• cmd.Wait() blocks until the command has finished.

In between, the command runs in the background, and you can process its output in real-time. The simplest way to see this is to connect the command's output directly to your program's output.

package main

import (
	"log"
	"os"
	"os/exec"
)

func main() {
	cmd := exec.Command("ping", "-c", "4", "google.com")

	// Pipe the command's stdout directly to our program's stdout.
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	// Start the command
	if err := cmd.Start(); err != nil {
		log.Fatalf("cmd.Start failed with %s\n", err)
	}

	log.Printf("Waiting for command to finish...")
	
	// Wait for the command to exit
	err := cmd.Wait()
	log.Printf("Command finished with error: %v", err)
}

When you run this, you'll see the ping output appear line-by-line, just as it would in your terminal.

4. Advanced Control: Timeouts and Termination

A robust automation script must be able to handle commands that hang or run for too long. Forgetting to do this can lead to stuck processes and resource leaks. The modern and idiomatic way to manage a command's lifecycle in Go is with the context package.

By creating a command with exec.CommandContext, you link the command's lifetime to a context. If the context is cancelled (e.g., due to a timeout), the command is automatically sent a signal to terminate.

Here's how you can implement a timeout:

package main

import (
	"context"
	"log"
	"os/exec"
	"time"
)

func main() {
	// Create a context that will be cancelled after 2 seconds.
	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
	defer cancel() // Good practice to call cancel, even on success.

	// This command will sleep for 5 seconds, so it should be killed by the context.
	cmd := exec.CommandContext(ctx, "sleep", "5")

	log.Println("Executing command with 2-second timeout...")
	output, err := cmd.CombinedOutput()

	if err != nil {
		// When the context times out, it sends a kill signal.
		// The error will reflect this. On Linux, it's often "signal: killed".
		log.Printf("Command failed with error: %v", err)
	}

	log.Printf("Output: %s", string(output))
}

A Note on Process Groups

For most use cases, CommandContext is sufficient. However, in complex scenarios, especially when using sh -c to run shell pipelines (e.g., cmd1 | cmd2), you might find that killing the main shell process doesn't terminate its children. Unix-like systems solve this with process groups.

The Managing Go Processes article delves into this, explaining how to use syscall.SysProcAttr with Setpgid: true to ensure that a signal sent to the parent terminates the entire group of child processes. This is an advanced topic you don't need to master now, but it's important to know it exists for when you build highly complex orchestration tools.

Conclusion

You now have the tools to make your Go programs execute and interact with any command-line application. This is a massive step forward, bridging the gap between standalone Go code and the rich ecosystem of CLI tools that power modern infrastructure.

Key Takeaways:

• The os/exec package is the standard library for running external commands.
• Use exec.Command to define the command and its arguments.
• For simple cases, cmd.CombinedOutput() or cmd.Output() can run a command and capture its output.
• For robust tools, capture stdout and stderr separately by assigning io.Writers (like bytes.Buffer) to the cmd.Stdout and cmd.Stderr fields, then use cmd.Run().
• Provide input to commands via the cmd.Stdin field, which accepts any io.Reader.
• Use the cmd.Start() / cmd.Wait() pattern to handle streaming output from long-running commands.
• Always use exec.CommandContext with a context to manage command lifetimes and implement timeouts, preventing hung processes.

Next Up

Many modern CLI tools and APIs (like kubectl, aws cli, or the GitHub API) can return data structured as JSON or YAML. Now that you can execute commands and capture their output, the next logical step is to parse that structured data to make your tools even more intelligent. In the next lesson, we will focus on how to parse and generate JSON and YAML for application configuration.