Custom Classifiers for Specialized Data

1. The Role of Custom Classifiers

When a crawler runs, it iterates through a list of classifiers to determine the data's format. The key rule to remember is that custom classifiers are always invoked before built-in classifiers.

The process works as follows:

1. The crawler tries each custom classifier you've specified, in the order you've listed them.
2. Each classifier returns a "certainty score" between 0.0 and 1.0. A score of 1.0 means the classifier is 100% confident it has recognized the data.
3. If any custom classifier returns a certainty of 1.0, the crawler uses it to generate the schema and stops.
4. If no custom classifier returns 1.0, the crawler then tries the built-in classifiers.
5. If no classifier (custom or built-in) achieves a certainty score above 0.0, the data is classified as UNKNOWN.

There are four types of custom classifiers you can create, each for a different scenario:

• Grok: For parsing line-based, unstructured or semi-structured text files, such as application logs or custom text formats. It uses named regular expressions.
• CSV: For handling delimited files that the built-in classifier can't parse, for example, files using a non-standard delimiter (like | or #), a different quote character, or needing explicit column header definitions.
• JSON: For defining a table schema based on a specific part of a JSON document (e.g., an array of objects nested within a larger structure) using a JsonPath expression.
• XML: For specifying which XML tag should be treated as a "row" when parsing XML data.

Let's dive into the most common and powerful types: Grok and CSV.

2. Parsing Complex Text with Grok Classifiers

Grok is a powerful tool for giving structure to unstructured text data. At its core, Grok uses named regular expressions to parse strings. Given your background in computer science, you can think of this as defining a formal grammar for your text data.

The basic syntax for a Grok pattern is:
%{PATTERN_NAME:field_name:data_type}

• PATTERN_NAME: The name of a built-in or custom pattern (e.g., NUMBER, WORD).
• field_name: The name you want to assign to the resulting column in your table schema.
• data_type: An optional data type to cast the column to (e.g., int, float). If omitted, it defaults to string.

For example, a log line like [INFO] 192.168.1.1 User login successful could be parsed by a Grok pattern like:
$$%{WORD:log_level}$$ %{IP:client_ip} %{GREEDYDATA:log_message}

Here, WORD, IP, and GREEDYDATA are built-in patterns that AWS Glue provides. Sometimes, however, you need to parse a format for which no built-in pattern exists. In that case, you define your own custom pattern, which is simply a name you assign to a regular expression.

Now, let's see this in action. The following video provides a comprehensive, hands-on demonstration of creating a Grok classifier to parse a custom Apache error log format.

3. Handling Non-Standard Delimited Files with CSV Classifiers

Another common scenario is receiving delimited files that don't follow the standard comma-separated format. Perhaps they use a semicolon or hash as a delimiter, or they have unusual quoting rules. In these cases, the built-in CSV classifier will fail, resulting in an UNKNOWN classification or an incorrect schema.

The solution is to create a custom CSV classifier. Here, instead of writing a complex Grok pattern, you simply specify the file's properties:

• Column delimiter: The character separating columns (e.g., #, ;, |).
• Quote symbol: The character used to enclose fields that may contain the delimiter.
• Column headings: You can either tell Glue to detect them, confirm they exist, or provide a manual list of header names.
• Custom datatypes: You can optionally provide a comma-separated list of data types to enforce a specific schema.

Let's watch a practical example.

As the video also briefly demonstrates (from 19:09 onwards), it's often possible to solve the same problem with a Grok classifier. However, for delimited files, using a custom CSV classifier is almost always simpler and more direct. For the exam, know when to choose the right tool for the job.

4. A Quick Look at JSON and XML Classifiers

While Grok and CSV classifiers are the most common, it's good to know the purpose of JSON and XML classifiers.

Conclusion

You've now learned how to make AWS Glue Crawlers smarter by teaching them to understand your unique and specialized data formats. This is a powerful feature that moves you from being a user of Glue to an engineer who can tailor it to specific business needs.

Key Takeaways:

• Custom classifiers are used when AWS Glue's built-in classifiers cannot infer the schema of your data.
• They run before built-in classifiers and are chosen based on a certainty score.
• Grok classifiers use named regular expressions to parse semi-structured text and log files. You define a Grok pattern and optional Custom patterns.
• CSV classifiers are for delimited files with non-standard delimiters, quote characters, or headers. They are configured with simple parameters, not patterns.
• JSON and XML classifiers are used to specify which part of a document (JSON path or Row tag) should be used to define the table's schema.
• Modifying a custom classifier requires re-running the crawler on the data source to create or update the table metadata. It does not retroactively change tables created by previous crawler runs.

Next Up:
So far in this module, we have focused on cataloging data. We've defined a Data Catalog, used crawlers to populate it with standard data, and now we've taught our crawlers to handle custom formats. With our data fully discovered and its schema understood, we are ready for the next step in the ETL process: transformation. In the next lesson, we will design a Glue ETL job using the Spark environment for scalable data transformation.