JSON & YAML for App Configuration

1. A Tale of Two Formats: JSON and YAML

In the DevOps world, you'll constantly encounter two primary data serialization formats: JSON and YAML. While both serve to structure data in a human-readable way, they are optimized for different use cases.

• JSON (JavaScript Object Notation): The lingua franca of web APIs. It is compact and strictly defined, making it ideal for machine-to-machine communication.
• YAML (YAML Ain't Markup Language): Favored for configuration files (think Kubernetes, Ansible, CI/CD pipelines). Its use of indentation and support for comments makes it easier for humans to read and write.

This image provides an excellent side-by-side comparison of the same data represented in both formats.

As a DevOps engineer, you need to be proficient in handling both. Let's see how Go makes this process straightforward.

2. Working with JSON: The encoding/json Package

Go's standard library provides the powerful encoding/json package for all your JSON needs. The two fundamental operations you'll perform are:

• Unmarshalling: Parsing JSON data (typically a []byte) into a Go struct.
• Marshalling: Generating JSON data from a Go struct.

The key to this process is defining a Go struct that mirrors the structure of your JSON data.

The following video provides a great hands-on walkthrough of reading, parsing, and writing JSON files. We'll break down the key concepts from it.

Unmarshalling: From JSON to Go Structs

Let's say you've executed a command like gcloud compute instances describe my-instance --format=json and captured the output. To work with this data in Go, you first define a struct.

// A simplified struct for a VM instance
type VMInstance struct {
    Name        string
    MachineType string
    Status      string
    Disks       []string
}

The process to unmarshal a JSON byte slice jsonData into this struct is a single function call:

var instance VMInstance
err := json.Unmarshal(jsonData, &instance)
if err != nil {
    log.Fatalf("Error unmarshalling JSON: %v", err)
}

fmt.Printf("Instance Name: %s, Status: %s\n", instance.Name, instance.Status)

Notice that json.Unmarshal takes a pointer (&instance) as the second argument. This is because the function needs to modify the instance variable in memory to populate it with the parsed data.

Struct Tags: Mapping JSON to Go

What if your JSON keys don't perfectly match your Go field names? For instance, JSON often uses camelCase or snake_case, while Go's convention for exported fields is PascalCase. This is where struct tags come in. They are string literals placed after a field's type that provide metadata to the json package.

Let's refine our VMInstance struct with tags:

type VMInstance struct {
    Name        string   `json:"name"`
    MachineType string   `json:"machineType"`
    Status      string   `json:"status"`
    Disks       []string `json:"disks"`
}

Now, json.Unmarshal knows to map the JSON key "machineType" to the Go field MachineType. It's a best practice to always use struct tags for explicit mapping.

Marshalling: From Go Structs to JSON

Generating JSON is just as easy. The json.Marshal function takes a Go struct and returns its JSON representation as a byte slice.

updatedInstance := VMInstance{
    Name:        "my-instance",
    MachineType: "e2-medium",
    Status:      "TERMINATED", // We've updated the status
    Disks:       []string{"disk-1"},
}

// Marshal to a compact, single-line JSON string
jsonData, err := json.Marshal(updatedInstance)
if err != nil {
    // handle error
}

// For human-readable output, use MarshalIndent
prettyJSON, err := json.MarshalIndent(updatedInstance, "", "  ") // "" prefix, "  " indent
if err != nil {
    // handle error
}

fmt.Println(string(prettyJSON))

The output of MarshalIndent would be:

{
  "name": "my-instance",
  "machineType": "e2-medium",
  "status": "TERMINATED",
  "disks": [
    "disk-1"
  ]
}

type ServiceStatus struct {
    ServiceName     string `json:"service_name"`
    HealthyReplicas int    `json:"healthy_replicas"`
    IsCritical      bool   `json:"is_critical"`
}

3. Handling Configuration with YAML and Viper

While you can parse YAML using a similar marshal/unmarshal pattern with a third-party library like gopkg.in/yaml.v3, a more robust and common approach for application configuration is to use a dedicated library. Viper is the de-facto standard in the Go ecosystem.

Viper is powerful because it can:

• Read from YAML, JSON, TOML, and other file formats.
• Read from environment variables.
• Read from command-line flags.
• Set default values.

This layered approach is perfect for DevOps tools, where you might want a default configuration in a file that can be overridden by environment variables in a CI/CD pipeline.

The following resources will guide you through setting up Viper to handle YAML configuration.

To reinforce what you see in the video, here is a great article that covers the same concepts in text form.

The Viper Workflow

Let's summarize the typical workflow for reading a config.yaml file.

1. Define your config struct, using mapstructure tags (Viper's preferred tag name, though it often falls back to others like yaml).

   // config.yaml
   server:
     host: "localhost"
     port: 8080
   database:
     user: "admin"
     password: "password"
   

   // config.go
   type Config struct {
       Server struct {
           Host string `mapstructure:"host"`
           Port int    `mapstructure:"port"`
       } `mapstructure:"server"`
       Database struct {
           User     string `mapstructure:"user"`
           Password string `mapstructure:"password"`
       } `mapstructure:"database"`
   }
   

2. Configure and run Viper to read the file and unmarshal it.

   import (
       "fmt"
       "log"
       "github.com/spf13/viper"
   )
   
   func LoadConfig() (Config, error) {
       var config Config
   
       // Tell Viper where to look for the config file
       viper.AddConfigPath(".") // Look in the current directory
       viper.SetConfigName("config") // Name of the config file (without extension)
       viper.SetConfigType("yaml") // Type of the config file
   
       // Attempt to read the config file
       if err := viper.ReadInConfig(); err != nil {
           return Config{}, fmt.Errorf("failed to read config file: %w", err)
       }
       
       // Unmarshal the configuration into our struct
       if err := viper.Unmarshal(&config); err != nil {
           return Config{}, fmt.Errorf("unable to decode into struct: %w", err)
       }
   
       return config, nil
   }
   
   func main() {
       cfg, err := LoadConfig()
       if err != nil {
           log.Fatalf("Error loading config: %v", err)
       }
   
       fmt.Printf("Server will run on %s:%d\n", cfg.Server.Host, cfg.Server.Port)
   }
   

This pattern gives you a strongly-typed Config object that you can pass around your application, making your code clean, testable, and easy to reason about.

Conclusion

You now have the skills to handle the two most common data formats in the cloud-native and DevOps landscape. This ability to parse and generate structured data transforms your Go programs from simple script replacements into sophisticated components of your automation and infrastructure stack.

Key Takeaways:

• Go's encoding/json package provides Marshal and Unmarshal functions for seamless JSON handling.
• Structs are the foundation for working with structured data in a type-safe way in Go.
• Struct tags (json:, yaml:, mapstructure:) are essential for mapping keys from your data format to your Go struct fields.
• For application configuration, especially with YAML, Viper is the industry-standard library that provides a robust, layered approach to reading from files, environment variables, and more.
• The most reliable pattern is to unmarshal your configuration directly into a Go struct, giving you type safety and code completion.

Next Up

In this lesson, we saw how Viper can read from files. We also mentioned its ability to read from environment variables. In the next lesson, we will dive deeper into environmental interactions. You will learn to interact with the OS via environment variables and the path/filepath package. This will complete your foundational knowledge for building CLI tools that are fully aware of and configurable by their runtime environment.