File Handling in Python: JSON and CSV

1. Working with CSV Files

CSV (Comma-Separated Values) is a simple, text-based format for storing tabular data, like a spreadsheet. Each line in the file is a data record (a row), and each record consists of one or more fields (columns), separated by commas. You might encounter CSV files when exporting reports from monitoring systems, managing user lists, or cataloging cloud resources.

While you could try to parse these files by splitting strings, this approach is brittle and fails when values themselves contain commas. Python's built-in csv module is the correct and robust way to handle this.

Reading from a CSV File

The csv module provides a reader object that intelligently iterates over lines in the CSV file, yielding each row as a list of strings.

Let's walk through an example. Imagine you have a file named servers.csv with the following content:

hostname,ip_address,location
web-01,192.168.1.10,us-east-1
db-01,192.168.1.15,us-east-1

You can read this file with the following Python code:

import csv

try:
    with open('servers.csv', 'r') as csv_file:
        csv_reader = csv.reader(csv_file)
        
        # Skip the header row
        header = next(csv_reader)
        print(f"Header: {header}")
        
        # Iterate over the remaining rows
        for row in csv_reader:
            # Each 'row' is a list of strings
            print(f"Hostname: {row[0]}, IP: {row[1]}")
            
except FileNotFoundError:
    print("Error: servers.csv not found.")

Notice the use of with open(...), which ensures the file is automatically closed, and the try...except block we learned about last lesson to handle a missing file.

Writing to a CSV File

To write data, you use a csv.writer object and its writerow() method.

Here's how you could generate a new CSV file. Note the newline='' argument in open()—this is important to prevent blank rows from being inserted between your data on some operating systems.

import csv

# Data to be written (list of lists)
server_data = [
    ['hostname', 'os', 'status'],
    ['app-01', 'Ubuntu 20.04', 'online'],
    ['db-01', 'RHEL 8', 'offline']
]

with open('inventory_report.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    
    # writer.writerow() writes one row
    # writer.writerows() writes all rows from an iterable
    writer.writerows(server_data)

print("inventory_report.csv created successfully.")

A Better Way: DictReader and DictWriter

Accessing data by index (e.g., row[0]) can make your code hard to read and maintain. What if a column is added or removed? A much better practice is to use csv.DictReader and csv.DictWriter, which treat each row as a dictionary, using the header row for keys.

Here is the reading example refactored to use DictReader:

import csv

try:
    with open('servers.csv', 'r') as csv_file:
        # DictReader uses the first row as keys by default
        csv_reader = csv.DictReader(csv_file)
        
        for row in csv_reader:
            # Now access data by key name!
            print(f"Hostname: {row['hostname']}, IP: {row['ip_address']}")
            
except FileNotFoundError:
    print("Error: servers.csv not found.")

This version is much clearer and more robust against changes in column order.

2. Working with JSON Files

JSON (JavaScript Object Notation) is the de facto standard for data exchange on the web, especially for APIs. Its syntax maps almost perfectly to Python's dictionaries and lists, making it extremely easy to work with. You'll use JSON for everything from cloud provider API responses to application configuration files.

Loading JSON (from Files or Strings)

Python's json module provides two main functions for loading data:

1. json.load(): Reads from a file object and converts it into a Python object.
2. json.loads(): Reads from a string (load-s) and converts it into a Python object.

This is a frequent point of confusion, so this diagram is helpful:

Imagine a configuration file app_config.json:

{
  "appName": "WebApp1",
  "version": "1.2.0",
  "max_connections": 100,
  "features_enabled": [
    "feature_A",
    "feature_B"
  ],
  "monitoring": {
    "enabled": true,
    "endpoint": "http://monitor.svc.local/api"
  }
}

Here's how you'd load it in Python:

import json

try:
    with open('app_config.json', 'r') as f:
        config = json.load(f) # Use json.load() for files
    
    # 'config' is now a Python dictionary
    print(f"App Name: {config['appName']}")
    print(f"Monitoring enabled: {config['monitoring']['enabled']}")

except FileNotFoundError:
    print("Error: app_config.json not found.")
except json.JSONDecodeError:
    print("Error: The file is not valid JSON.")

Dumping JSON (to Files or Strings)

Similarly, there are two functions for writing data:

1. json.dump(): Writes a Python object to a file object as JSON.
2. json.dumps(): Converts a Python object into a JSON-formatted string.

Here's how to create a new JSON file from a Python dictionary:

import json

new_config = {
    'service_name': 'auth-service',
    'version': '2.0.1',
    'replicas': 3,
    'ports': [8080, 8081]
}

with open('auth_service_config.json', 'w') as f:
    # Use json.dump() to write the dict to the file.
    # indent=4 makes it pretty and readable.
    json.dump(new_config, f, indent=4)

print("auth_service_config.json created successfully.")

3. Practical Application: Converting CSV to JSON

A common task is to convert data from one format to another. For example, you might get an inventory of servers in a CSV file but need to feed it into an automation tool that consumes JSON.

This is a perfect way to combine what you've learned. The process is:

1. Open the CSV file and use csv.DictReader to read the data into a list of dictionaries.
2. Open a new JSON file for writing.
3. Use json.dump to write the list of dictionaries to the JSON file.

Here is the code you can run yourself. First, create the servers.csv file from our earlier example, then run this script:

import csv
import json

def convert_csv_to_json(csv_path, json_path):
    """Reads data from a CSV file and writes it to a JSON file."""
    data = []
    try:
        with open(csv_path, 'r', newline='') as csv_file:
            # DictReader converts each row to a dictionary
            reader = csv.DictReader(csv_file)
            for row in reader:
                data.append(row)
        
        with open(json_path, 'w') as json_file:
            # Dump the list of dictionaries to the JSON file
            json.dump(data, json_file, indent=4)
        
        print(f"Successfully converted {csv_path} to {json_path}")
        
    except FileNotFoundError:
        print(f"Error: The file {csv_path} was not found.")
    except Exception as e:
        print(f"An error occurred: {e}")

# --- Execute the conversion ---
convert_csv_to_json('servers.csv', 'servers.json')

After running this, you will have a new servers.json file containing your server data, perfectly formatted for another program to consume.

Conclusion

You now have the fundamental skills to handle the two most prevalent data formats in the DevOps world. This ability to read, manipulate, and write structured data is a cornerstone of automation.

Key Takeaways:

• Use the csv module for CSV files. Prefer DictReader and DictWriter for more readable and robust code.
• Use the json module for JSON data. Its structure maps naturally to Python dictionaries and lists.
• Remember the s: load and dump work with files, while loads and dumps work with strings.
• Always use with open(...) to ensure files are closed properly, and wrap file operations in try...except blocks to handle potential errors.
• Use the indent argument when writing JSON files to keep them human-readable.

Next Up

You've learned how to create and parse JSON data. But where does that data often come from? Web services and APIs! In our next lesson, you will use Python libraries like 'requests' to interact with REST APIs, fetching live data from the internet and putting your JSON handling skills to immediate practical use.