File Management with pathlib: Traversal, Encoding, and Error Handling

Hello! In the previous lesson, you built a parser that turns JSON text into a validated repository inventory and then produces a summary. The example supplied that JSON directly in a Python string. This lesson gives the parser a filesystem boundary: you will find inventory files, read them as text, and save summaries, while making missing paths and invalid text encoding visible rather than silently treating them as empty data.

Plan on about 40–50 minutes, including the resources and implementation. Work from the project root, using the environment and src/repo_assistant/inventory.py from the previous lessons.

Paths represent locations, not necessarily files

A Path object lets you describe a location before anything exists there. For example, Path("data") / "incoming" / "demo.json" constructs a path; it does not create demo.json. A relative path starts from the process’s current working directory, which is why running examples from the project root matters.

Watch the selected portions of Corey Schafer’s Python Tutorial: Pathlib - The Modern Way to Handle File Paths. The aim is to recognize which operations merely construct or inspect a path and which ones visit the filesystem.

For this application, the traversal methods have distinct jobs. root.iterdir() yields immediate children. root.glob("*.json") matches JSON-named entries immediately under root. root.rglob("*.json") searches beneath root as well. These methods produce Path objects; filter with is_file() if you want files rather than any matching directory. Sorting the results gives your program a predictable processing order.

One subtlety matters for error handling: an empty search result need not mean “a valid directory containing no matching files.” If the input directory is missing, establish that explicitly before reporting an empty inventory.

Read and write text deliberately

Your parser accepts JSON text, so read_text() is a suitable boundary for a small inventory file. Its counterpart, write_text(), can save the summary. Both open and close the file for you.

Watch the short read_text() and write_text() demonstration in Net Ninja’s Working with Files in Python #5 - Pathlib to Read & Write Files.

Encoding determines how file bytes become characters and vice versa. Choosing UTF-8 explicitly makes the intended format clear across machines. It does not guarantee that an arbitrary input file is UTF-8: invalid bytes can raise UnicodeDecodeError. Nor does a successful decode prove you chose the right encoding; some mismatches produce plausible-looking but garbled text.

Read the relevant portions of Files and directories from Python Basics for the encoding behavior and the alternative of reading through a file object.

For a small JSON document, reading the entire string is convenient. For a large source file that you only need to inspect line by line, use with path.open(encoding="utf-8") as file: and iterate over file instead. Keep the default strict decoding behavior for inventory JSON: replacing or ignoring invalid bytes could change the data before validation sees it.

Add a filesystem boundary to the package

Create src/repo_assistant/file_io.py:

from pathlib import Path

from repo_assistant.inventory import (
    InventorySummary,
    RepositoryPayload,
    parse_inventory,
)


def find_inventory_files(root: Path) -> list[Path]:
    if not root.exists():
        raise FileNotFoundError(f"Inventory directory not found: {root}")
    if not root.is_dir():
        raise NotADirectoryError(f"Expected inventory directory: {root}")

    return sorted(
        path for path in root.rglob("*.json") if path.is_file()
    )


def load_inventory(path: Path) -> RepositoryPayload:
    try:
        raw = path.read_text(encoding="utf-8")
    except FileNotFoundError as exc:
        raise FileNotFoundError(
            f"Inventory file not found: {path}"
        ) from exc
    except UnicodeDecodeError as exc:
        raise ValueError(
            f"Inventory file is not valid UTF-8: {path}"
        ) from exc

    return parse_inventory(raw)


def save_summary(
    summary: InventorySummary, destination: Path
) -> None:
    destination.parent.mkdir(parents=True, exist_ok=True)
    destination.write_text(
        summary.model_dump_json(indent=2) + "\n",
        encoding="utf-8",
    )

There are three different boundaries here. find_inventory_files() traverses a directory and rejects a missing root instead of presenting it as an empty one. load_inventory() reads and decodes a file, then passes its text to the validator you already built. save_summary() creates missing parent directories before writing JSON text; write_text() can create the file, but it cannot create its parents.

The exception choices preserve useful distinctions:

• A missing input file raises FileNotFoundError with its path in the message. The from exc clause keeps the original failure available in the traceback.
• Invalid UTF-8 becomes a ValueError describing the file. It is a decoding problem, not a failed Pydantic field rule.
• Syntactically broken JSON or invalid inventory fields still fail in parse_inventory(). This function does not disguise those validation errors as missing files.

Notice that load_inventory() does not call exists() before reading. A file could disappear between that check and the read, so handling the read’s FileNotFoundError is more reliable. The check on the directory serves a different purpose: making a missing search root distinguishable from a valid search with zero matches. Other filesystem failures, such as a permission error, remain visible rather than being incorrectly reported as “not found.”

Finally, write_text() overwrites an existing destination. This is appropriate only if that destination is a report you intend to regenerate. Do not point it at an input or another valuable file.

Run the complete file-to-summary path

Create try_inventory_files.py at the project root:

from pathlib import Path

from repo_assistant.file_io import (
    find_inventory_files,
    load_inventory,
    save_summary,
)
from repo_assistant.inventory import summarize_inventory


incoming = Path("data") / "incoming"
incoming.mkdir(parents=True, exist_ok=True)

sample = incoming / "demo.json"
sample.write_text(
    '{"repository":"demo","files":['
    '{"path":"src/math_ops.py","lines":42,"symbols":['
    '{"name":"mean","kind":"function","line":5}'
    ']}]}\n',
    encoding="utf-8",
)

for path in find_inventory_files(incoming):
    inventory = load_inventory(path)
    summary = summarize_inventory(inventory)

    destination = Path("data") / "reports" / f"{path.stem}-summary.json"
    save_summary(summary, destination)
    print(
        f"{path.name}: {summary.file_count} files, "
        f"{summary.symbol_count} symbols"
    )

print(destination.read_text(encoding="utf-8"))

try:
    load_inventory(incoming / "absent.json")
except FileNotFoundError as exc:
    print(exc)

bad_encoding = Path("data") / "bad-encoding.json"
bad_encoding.write_bytes(b"\xff")

try:
    load_inventory(bad_encoding)
except ValueError as exc:
    print(exc)

Run python try_inventory_files.py from the project root. You should see one processed inventory, a saved summary containing one file and one symbol, then distinct messages for the missing file and the file whose bytes cannot be decoded as UTF-8. The deliberately malformed file sits outside data/incoming, so the next run will not accidentally include it in the inventory search.

The sample uses write_text() to create known UTF-8 input and write_bytes() solely to simulate a decoding failure. Keep that distinction in mind: missing path, unreadable text, and invalid JSON data are different problems, even though all three can prevent a summary from being produced.

Where this leaves the application

You can now construct paths, search a directory recursively, read UTF-8 input, and write a summary into a directory that may not yet exist. More importantly, your code does not mistake a missing search root for an empty result or corrupt input bytes for valid JSON. You have kept filesystem concerns separate from the inventory’s validation and transformation logic.

Next, you will put application logic like this behind a command-line interface, so a user can supply paths as arguments instead of editing a script.