Constructing a Representative Golden Dataset

Good to see you again. Last lesson, you froze one model response to test whether generate_offer_insight() and its validation boundary work together. That fixture answers a narrow engineering question: “Does this controlled response pass through the application correctly?” It does not tell you whether the application handles the range of requests people are likely to make.

Today you’ll build a small, version-controlled golden dataset of expected ecommerce insight requests. The deliverable is a reviewed JSONL file that records the input your application receives and what a satisfactory response must get right. You can build and review a first version in about 40 minutes, without a paid API or a running model.

Decide what this dataset represents

Scope this first dataset to the behavior you have been building: generating an OfferInsight from a business note and trusted metric values. Do not mix in questions for a future SQL tool, document search, or a general-purpose chatbot. A failed case should point you back toward this application path.

A golden case needs more than a plausible question. It needs a reviewed expectation. For example, if a note says repeat purchase rate moved from 0.32 to 0.28, a satisfactory insight must not call that an increase, mistake a four-percentage-point decline for a four-percent relative decline, or invent a cause that the note does not establish. It may express the insight in more than one acceptable way; you need not prescribe its exact wording.

Read the relevant parts of Langfuse’s Golden dataset evaluation: build and maintain LLM test sets. Focus on the distinction between collected examples and trusted, reviewed cases—not on its product-specific UI or SDK.

For a quick view of how to start when genuine interactions are scarce, watch this portion of AI with Lena Hall’s How to Build AI Products That Work | LLM Evaluation Guide.

The supplied Building a golden dataset diagram makes a useful distinction among human-crafted cases, sanitized production requests, and reviewed synthetic expansions:

Treat its sample counts as planning heuristics, not statistical guarantees. Ten carefully selected cases can expose an obvious regression, but they cannot establish the performance of your application across all production traffic. Conversely, hundreds of near-duplicates do not make a dataset representative.

Build a first set from the application’s actual input

Your previous integration test used NOTE and separate, trusted expected_metric, previous_value, and current_value arguments. Preserve that boundary here. The dataset’s input should contain values that can actually be passed into that path; its expected field describes behavior to review later. Do not turn the accepted JSON response from last lesson into ten lightly reworded copies. That would vary the response fixture, not the user requests.

Start with the following candidate generator. It writes ten invented, non-sensitive requests to evals/offer_insight_candidates.jsonl. The cases vary period, segment, phrasing, direction, numerical interpretation, and requests for causal explanations. They deliberately stay with repeat_purchase_rate, the metric used in your running example. If your application supports other metrics, replace some cases with requests for those metrics before promoting the file to a golden set.

# scripts/seed_offer_insight_cases.py
import json
from pathlib import Path

cases = [
    (
        "feb_decline", "baseline",
        "Why did repeat purchase rate drop from 0.32 in January "
        "to 0.28 in February?",
        0.32, 0.28,
    ),
    (
        "spring_increase", "terse request",
        "Quick read on repeat purchase rate: March 0.28; April 0.30.",
        0.28, 0.30,
    ),
    (
        "email_cohort", "segment",
        "Email cohort repeat purchase rate moved from 0.22 to 0.25. "
        "Summarize this.",
        0.22, 0.25,
    ),
    (
        "mobile_decline", "segment",
        "Mobile repeat purchase rate was 0.41, now 0.37. What changed?",
        0.41, 0.37,
    ),
    (
        "promotion_question", "unsupported cause",
        "After the spring promotion, repeat purchase rate fell from "
        "0.34 to 0.31. Did the promotion cause it?",
        0.34, 0.31,
    ),
    (
        "west_region", "segment",
        "West region repeat purchase rate rose from 0.18 to 0.21. "
        "Summarize.",
        0.18, 0.21,
    ),
    (
        "points_vs_percent", "numerical interpretation",
        "Our repeat purchase rate went from 0.31 to 0.29. Is that "
        "a 2 percent or 2 percentage point drop?",
        0.31, 0.29,
    ),
    (
        "quarter_comparison", "reversed period order",
        "Repeat purchase rate for Q3 was 0.26 versus 0.24 for Q2; "
        "give a quick insight.",
        0.24, 0.26,
    ),
    (
        "investigation", "next step",
        "November repeat purchase rate 0.36, October 0.38. "
        "What should we investigate?",
        0.38, 0.36,
    ),
    (
        "returning_shoppers", "segment",
        "Repeat purchase rate for returning shoppers: old 0.52, "
        "new 0.49. Please keep it concise.",
        0.52, 0.49,
    ),
]

path = Path("evals/offer_insight_candidates.jsonl")
path.parent.mkdir(parents=True, exist_ok=True)

with path.open("w", encoding="utf-8") as file:
    for case_id, scenario, note, previous, current in cases:
        item = {
            "id": case_id,
            "input": {
                "note": note,
                "expected_metric": "repeat_purchase_rate",
                "previous_value": previous,
                "current_value": current,
            },
            "expected": {
                "metric": "repeat_purchase_rate",
                "direction": (
                    "increase" if current > previous else "decrease"
                ),
                "change_percentage_points": round(
                    (current - previous) * 100, 2
                ),
                "criteria": [
                    "Does not contradict the supplied metric or rates",
                    "Does not assert an unverified cause",
                ],
            },
            "metadata": {
                "scenario": scenario,
                "source": "human_crafted_seed",
                "reviewed": False,
            },
        }
        file.write(json.dumps(item) + "\n")

print(f"Wrote {len(cases)} candidates to {path}")

Run it from the project root with your locked environment:

uv run --locked python scripts/seed_offer_insight_cases.py

The percentage-point value is a reference for reviewing a response, not an assertion that OfferInsight has a field with that name. If your schema does not report a numerical change, assess whether any change it does state is correct. Similarly, the criteria do not prove your current validator enforces causal restraint; they record a business requirement that may reveal a gap when you later run evaluations.

Notice the promotion_question case. “After the promotion” establishes timing, not causation. An acceptable response can identify the decline and say that further evidence would be needed to attribute it to the promotion. That distinction is more useful than storing one supposedly perfect sentence for the model to imitate.

Turn candidates into a reviewed golden file

The generator creates candidates, not automatically trusted labels. Spend the remaining build time inspecting every JSONL line. For each one, check the following against your real application contract:

1. Replayability: Can its input values be passed through your existing generate_offer_insight() call? If a request asks for behavior your application has not been designed to provide, narrow the expectation, mark it for a separate dataset, or remove the case. Do not quietly count an unsupported feature as a current pass requirement.
2. Business correctness: Do the period ordering, metric, direction, and percentage-point change agree? For points_vs_percent, 0.31 to 0.29 is a two-percentage-point fall; the relative percentage decrease is a different calculation.
3. Coverage: Are you testing distinct situations rather than ten near-identical phrasings? The scenario tag makes gaps visible. If your application serves gross-margin or conversion-rate requests too, replace some repeat-purchase cases with supported examples and review their expected values separately.
4. Provenance and privacy: These starter cases are human-crafted. Do not relabel them as production examples. When you eventually incorporate a real user request, remove personal or confidential details and have someone who understands the metric review its expected behavior.

Set metadata.reviewed to true only after reviewing and, where necessary, editing each case. Save the reviewed result as evals/offer_insight_golden.v1.jsonl; keep it in version control. A compact check catches missing reviews and duplicated IDs before you call it done:

import json
from pathlib import Path

path = Path("evals/offer_insight_golden.v1.jsonl")
items = [
    json.loads(line)
    for line in path.read_text(encoding="utf-8").splitlines()
    if line.strip()
]

ids = [item["id"] for item in items]
assert len(ids) == len(set(ids)), "Duplicate case IDs"
assert all(item["metadata"]["reviewed"] is True for item in items)
print(f"{len(items)} reviewed cases")
print(sorted({item["metadata"]["scenario"] for item in items}))

Run that check in a Python session from the project root, or save it as a small script. It checks dataset integrity, not model quality. No model calls are needed today.

Be precise about what you have built: this is a representative first set for a scoped behavior, chosen from expected requests. It is not yet an estimate of production success rates. As sanitized real requests and failures become available, compare their distribution with this set, add missing cases after review, and record a new dataset version. When comparing two application versions, use the same pinned dataset version so a changed score is not merely the result of changed cases.

Takeaway

You now have a path from one controlled response fixture to a reviewed collection of request-and-expectation pairs. Each case records replayable input, a business-grounded expectation, a coverage category, and honest provenance. The review step is what earns the word golden; the case count alone does not.

This completes the first module’s application-hardening foundation. In the next module you’ll run an open-weight instruction model locally or in a hosted notebook. Your dataset can then give you concrete business requests to revisit as you learn how to control and compare model behavior.