Reconciling Mismatched Sequential Files

The Classic COBOL File Matching Algorithm

For any developer with your experience, the traditional approach to file matching in COBOL is likely familiar. It involves reading from two pre-sorted sequential files in a synchronized loop and comparing their keys to decide which record to process next.

The core logic, often implemented within a PERFORM UNTIL loop, looks something like this:

1. Read one record from File 1.
2. Read one record from File 2.
3. Compare the keys of the two current records.
   • If Key 1 < Key 2: The record from File 1 is unique. Process it, then read the next record from File 1 only.
   • If Key 1 > Key 2: The record from File 2 is unique. Process it, then read the next record from File 2 only.
   • If Key 1 = Key 2: The records match. Process the match, then read the next record from both files.
4. Repeat until one or both files reach their end. Handle the remaining records in the file that is not yet at its end.

The following flowchart illustrates this fundamental process.

Let's look at how this logic is implemented in code.

Limitations of the Classic Approach

While this COBOL-centric approach is perfectly valid and clear, it has a significant drawback when dealing with the "large sequential files" specified in our scenario. The process is I/O-bound, meaning its performance is limited by the speed of reading from disk. For files with millions of records, a program that performs a READ for nearly every record in both files can be inefficient and consume substantial CPU time and I/O resources.

Furthermore, this simple logic assumes the keys and record layouts are identical. Handling mismatched formats and keys within this single COBOL program would require complex conditional logic, making the code harder to write, debug, and maintain. As a senior developer, your goal is to find a more efficient and elegant solution.

A More Powerful Tool: DFSORT's JOINKEYS

For high-volume file reconciliation on the mainframe, the industry-standard tool is DFSORT (or a compatible sort utility). Its JOINKEYS feature is specifically designed to perform high-speed, key-based comparisons and joins on large files, offloading the heavy I/O and comparison work from your COBOL program.

The strategy shifts from doing everything in COBOL to a two-stage process:

1. Use DFSORT to handle the comparison, key matching, and reformatting.
2. Use COBOL to process the clean, pre-processed output from DFSORT.

This division of labor leverages the strengths of each tool: DFSORT for raw data manipulation speed and COBOL for implementing complex business rules.

Let's dive into how JOINKEYS works.

Solving the "Mismatched" Scenario

Now we can address the specific complexities of our scenario—mismatched formats and keys—using the power of JOINKEYS.

Handling Mismatched Keys

The problem states our files have mismatched keys. This could mean several things, all of which JOINKEYS can handle.

• Keys are in different positions: This is the simplest case. The FIELDS parameter on the JOINKEYS statement allows you to specify the key's position, length, and format independently for each file.

  //SYSIN DD *
    JOINKEYS F1=IN1,FIELDS=(10,8,CH,A)  // Key starts at pos 10 in File 1
    JOINKEYS F2=IN2,FIELDS=(5,8,CH,A)   // Key starts at pos 5 in File 2
    ...
  /*
  

• Keys have different formats (but represent the same value): Imagine one file stores an account number as PIC X(8) and the other as PIC 9(8) PACKED-DECIMAL (4 bytes). You cannot join them directly. The solution is a pre-processing step. You would use DFSORT's INREC or OUTREC statement in a prior step to reformat one of the keys so they match.

  For example, you could convert the packed decimal key to a character representation before the join step.

• Keys are composite or need derivation: If a key is made of multiple, non-contiguous fields, you can use DFSORT's INREC statement to build a new, contiguous key at the beginning of the record in a preliminary step. Both files are pre-processed this way, and JOINKEYS then operates on the standardized key.

Handling Mismatched Record Formats

This is where the REFORMAT statement in JOINKEYS becomes invaluable. It allows you to construct a new output record by cherry-picking fields from both input files.

When a match is found, you have access to the record from F1 and the record from F2. In the REFORMAT statement, you can create a new, consolidated record layout.

For example, let's say File 1 has CUST-ID and CUST-ADDRESS, and File 2 has CUST-ID and CUST-LAST-ORDER-DATE. You can create a single output record containing all three fields.

//SYSIN DD *
  JOINKEYS F1=CUSTMAST,FIELDS=(1,10,A)
  JOINKEYS F2=ORDHIST,FIELDS=(1,10,A)
  
  // Create a new record from the matched pair
  REFORMAT FIELDS=(F1:1,10,   // CUST-ID from File 1
                  F1:11,50,  // CUST-ADDRESS from File 1
                  F2:11,8)   // CUST-LAST-ORDER-DATE from File 2
  
  JOIN UNPAIRED,F,F // Process all records
  
  SORT FIELDS=COPY
  
  // Split output into Adds, Deletes, and a consolidated "Changes" file
  OUTFIL FNAMES=ADDS,INCLUDE=(1,10,SS,EQ,C'??????????')
  OUTFIL FNAMES=DELETES,INCLUDE=(61,8,SS,EQ,C'????????')
  OUTFIL FNAMES=CHANGES,SAVE
/*

In the OUTFIL statements above, ?????????? is DFSORT's default placeholder for a key field from a file that did not participate in an unpaired match. By checking for this placeholder, you can route records to different output datasets. Unmatched F1 records go to DELETES, unmatched F2 records go to ADDS, and all others (the matched pairs) are SAVEd to the CHANGES file.

Architecting the Complete Solution

A robust, production-quality solution would be a multi-step JCL job:

1. STEP010 - SORTF1: Sort the first input file on its reconciliation key.
2. STEP020 - SORTF2: Sort the second input file on its reconciliation key. (These steps can be omitted if the files are guaranteed to be in the correct sequence).
3. STEP030 - RECONCILE: Use PGM=SORT with JOINKEYS control cards.
   • Define the JOINKEYS for F1 and F2 with their respective key definitions.
   • Use REFORMAT to define the layout of the output record, combining fields as needed.
   • Use JOIN UNPAIRED,F,F to process every record.
   • Use multiple OUTFIL statements to create separate output files for:
     • Records only in File 1 ("deletes" or "unmatched master").
     • Records only in File 2 ("adds" or "unmatched transaction").
     • Matched records, now in a single, consistent format.
4. STEP040 - PROCESS: A COBOL program that reads one or more of the output files from STEP030 to apply final business logic (e.g., generate a formatted report, create database update records). This COBOL program is now much simpler because the complex task of synchronization and reformatting has already been done.

This hybrid DFSORT + COBOL architecture is a powerful pattern. It demonstrates a senior developer's ability to select the most efficient tool for each part of a problem, leading to solutions that are faster, more scalable, and often easier to maintain than a pure COBOL approach.

Conclusion

In this lesson, we tackled a complex data reconciliation scenario. We moved from the traditional COBOL file-matching algorithm to a far more powerful and efficient architecture using DFSORT's JOINKEYS feature. This approach is the professional standard for handling large-scale file comparisons on the mainframe.

Key Takeaways:

• Classic vs. Modern: While the classic COBOL algorithm is fundamental, it is not performant for large files.
• Use the Right Tool: DFSORT is optimized for high-speed I/O and data manipulation. Offload the heavy lifting of comparison and reformatting to DFSORT.
• JOINKEYS is Your Friend: It handles the core logic of finding adds, deletes, and matches between two pre-sorted files.
• Solve Mismatches with JOINKEYS and REFORMAT: Define keys independently for each file using the FIELDS parameter, and create a unified output record layout using the REFORMAT statement.
• Architect a Multi-Step Solution: A complete solution typically involves JCL with steps for sorting, JOINKEYS reconciliation, and final COBOL-based business rule processing.

In our next lesson, we will continue exploring complex file processing by tackling another common scenario: how to process a multi-record-type file with internal sorting and control break logic.