Reusable Record Transformations with Python Functions and Generators

Hello! Last lesson, you converted CSV fields into typed Python values and checked that those values survived a CSV–JSON round trip. The conversion worked, but it lived inside one script’s reading loop. Today you’ll turn that work into reusable functions and use a generator to process records one at a time—an important pattern when a batch input is too large to keep in a list.

You’ll continue in the same project using only Python’s standard library. Allow about 40 minutes, including the short resource study and the coding walkthrough.

---

Give each transformation a clear contract

A record transformation is easier to reuse when its inputs and output are explicit. For this project, we can separate three jobs:

1. Convert one raw CSV row into a typed order.
2. Convert one typed order into a smaller sale record.
3. Take a sequence of typed orders, select paid ones, and produce sale records.

The first two jobs are ordinary functions: one record goes in and one new record comes out. Neither needs to open a file or change the input dictionary. That separation lets the same business transformation work with an order read from CSV, JSON, or another source.

Read the focused passages in Python’s Functional Programming HOWTO. You do not need the formal-proof discussion; concentrate on why small, input-to-output functions are useful in practical Python.

Here, “input-to-output” does not mean every Python function must be mathematically pure. It means that, given the expected input record, our transformation’s result does not depend on a file handle, a global counter, or a previous call. We’ll leave deliberate file writing to the outer script.

---

Why an iterable is not necessarily a list

In formats.py, you collected every converted order in orders. That was useful for inspecting two rows. For a much larger file, collecting all rows may be unnecessary: a writer usually needs only the next transformed record.

An iterable can supply an iterator. An iterator supplies items one at a time when requested; a for loop makes those requests for you. Lists are iterable, but they already hold all their items. A csv.DictReader can instead provide rows as you loop without first building a list of the whole file.

The distinction is illustrated here:

Read the “Iterators” section of the Python HOWTO for the precise distinction. Pay particular attention to its warning that an iterator is normally forward-only.

A generator function is a convenient way to make an iterator. Unlike an ordinary function that returns one completed result, it uses yield to supply a value, pause, and resume when another value is requested. Watch Corey Schafer change a list-producing function into a generator, then compare a list comprehension with a generator expression.

For a tiny dataset, a list is often simpler. The benefit of a generator is not that it makes each record’s calculation disappear; it is that the caller can consume results without storing the entire result set at once.

---

Build a record-by-record pipeline

From your project root, create transforms.py. Keep formats.py intact so its earlier round-trip demonstration still works. This new script retains the CSV type policy you already established: IDs remain strings, amounts remain integer cents, lowercase CSV true and false become booleans, and a blank coupon means None.

import csv
import json


def parse_order(row):
    """Return a typed order from one CSV dictionary row."""
    if row["paid"] not in ("true", "false"):
        raise ValueError("paid must be true or false")

    return {
        "order_id": row["order_id"],
        "product": row["product"],
        "quantity": int(row["quantity"]),
        "amount_cents": int(row["amount_cents"]),
        "paid": row["paid"] == "true",
        "coupon_code": row["coupon_code"] or None,
    }


def paid_sale(order):
    """Return an output record from one typed, paid order."""
    return {
        "order_id": order["order_id"],
        "product": order["product"].strip(),
        "amount_cents": order["amount_cents"],
        "used_coupon": order["coupon_code"] is not None,
    }


def transform_paid_orders(orders):
    """Yield one sale record for each paid order."""
    for order in orders:
        if order["paid"]:
            yield paid_sale(order)


def main():
    with open("data/orders.csv", newline="", encoding="utf-8") as source:
        with open(
            "data/paid_sales.jsonl", "w", encoding="utf-8"
        ) as destination:
            raw_rows = csv.DictReader(source)
            typed_orders = (parse_order(row) for row in raw_rows)

            for sale in transform_paid_orders(typed_orders):
                destination.write(
                    json.dumps(sale, ensure_ascii=False) + "\n"
                )


if __name__ == "__main__":
    main()

Run it from the project root:

python transforms.py

The output file, data/paid_sales.jsonl, should contain one line:

{"order_id": "0007", "product": "Notebook, ruled", "amount_cents": 1599, "used_coupon": false}

This is JSON Lines: each line is a separate JSON object. It is not the single JSON array you wrote last lesson. Writing one object per line lets the script write each sale as soon as it is produced. The unpaid order is absent; the paid order has used_coupon: false because its blank CSV coupon became None.

Look closely at the boundaries between functions. parse_order() accepts raw CSV strings. paid_sale() accepts a typed order and returns a new, smaller dictionary. transform_paid_orders() accepts an iterable of typed orders and yields zero or one sale for each input order. It does not care whether those typed orders originated in a file or a list.

The generator expression assigned to typed_orders also produces values on demand. No CSV row is parsed merely because that line is assigned. The for sale ... loop begins consumption: it requests a typed order, checks whether it is paid, and writes a sale if one is yielded. The input file stays open throughout that loop because consumption happens inside its with block. If you constructed this pipeline inside with but tried to iterate over it after leaving the block, its deferred reads would meet a closed file.

This pipeline does not build a list proportional to the number of input rows. Calling list(typed_orders) or list(transform_paid_orders(typed_orders)) would change that memory behavior. It is fine to do so for a small inspection sample, but not if the purpose is to stream a large file.

---

Reuse the business transformation with another source

Last lesson’s data/orders.json contains a small array of already typed orders. You should not pass those dictionaries through parse_order(): that function expects CSV text such as "true", not the JSON-decoded boolean True. You can pass them directly to transform_paid_orders().

From the project root, run this short check:

python - <<'PY'
import json
from transforms import transform_paid_orders

with open("data/orders.json", encoding="utf-8") as source:
    orders = json.load(source)

print(list(transform_paid_orders(orders)))
PY

It should print a list containing the same paid sale record. json.load() and list() hold data in memory here only for this two-order reuse check; the CSV-to-JSON-Lines script remains record-by-record.

There is one more practical consequence of the iterator contract: a generator is usually single-use. Once a loop has consumed it, another loop over that same generator will produce nothing. To run a file-based transformation again, reopen the source and create a fresh reader and generators. Also, because conversion happens when records are requested, an invalid value might raise an error during the writing loop, rather than when typed_orders is created. For now, leave that error visible; the next lesson will handle malformed records explicitly.

---

Conclusion

You now have reusable functions with distinct contracts: one converts a raw CSV row, one shapes a typed paid order for output, and one generator filters and yields sale records. Keeping file I/O outside the per-record functions makes the transformation usable with another source, while consuming generators inside the open-file block lets the CSV pipeline avoid collecting every record.

Next, you’ll decide what should happen when a record violates those contracts, using validation and exception handling rather than letting an unexplained conversion error interrupt the batch.