Debugging AST Creation

The Path from Parser to AST

Before we dive into the debugger, let's solidify our mental model of what we expect to see. As we've discussed, the Clang frontend is a collaboration primarily between the Parser and the Sema (Semantic Analysis) libraries.

1. The Parser consumes tokens and recognizes the grammatical structure of the code. For example, it sees identifier, equals, literal, plus, literal, semicolon.
2. When it recognizes a complete grammatical construct (like a binary expression), it calls an ActOn... method in Sema. For a + operation, this would be Sema::ActOnBinOp.
3. Sema then performs semantic checks (e.g., "Are these two operands compatible with addition?") and, if everything is valid, it creates the final AST node that represents the operation.

This entire workflow is a core part of Clang's design.

To visualize this, imagine the call stack deepening as the compiler processes your code. The following video gives an excellent animated overview of this process.

The Debugging Journey: From Code to AST Node

Let's begin. We'll use a slightly more interesting test case than last time to give us a binary operation to trace.

1. The Test Case
Create a file named test.cpp with the following content:

int main() {
    int x = 42 + 10;
    return 0;
}

Our target is the expression 42 + 10 and the BinaryOperator AST node that will represent it.

2. Launch the Debugger
Following the procedure from the last lesson:

1. Generate the cc1 command: /path/to/your/build/bin/clang -### test.cpp
2. Launch LLDB (or GDB) with the full cc1 command: lldb -- "<full cc1 command>"
3. Set the initial breakpoint: (lldb) b clang::Parser::ParseTopLevelDecl
4. Run the program: (lldb) run

You should now be paused at the entry point of ParseTopLevelDecl, ready to begin our trace.

Phase 1: Navigating to the Statement

Our breakpoint is at the function that parses any top-level declaration. Our file contains one: the main function. We need to step through the parsing of this function to get to its body, where our target expression lives.

• n (next): Use the next command several times to step over local variable initializations at the start of ParseTopLevelDecl.
• You will eventually reach a call to ParseExternalDeclaration. This is the main dispatcher for parsing things in the global scope.
• s (step): Use the step command to enter ParseExternalDeclaration.
• Again, use a combination of n and s to follow the logic. You will descend through a series of calls like ParseDeclaration, ParseDeclGroup, and finally ParseFunctionDefinition. Each of these functions is responsible for a more specific part of the grammar.
• Inside ParseFunctionDefinition, you will find a call to ParseCompoundStatementBody. This is the function that parses the code inside the {...} of our main function. Use s to step into it.

You are now inside the part of the parser that handles the body of the main function.

Phase 2: Parsing the Declaration

The first thing inside main is the statement int x = 42 + 10;.

• Inside ParseCompoundStatementBody, the parser will loop to consume statements. The first call in this loop will likely be to ParseStatementOrDeclaration. Step into it with s.
• This function is complex, as it can handle many kinds of statements. It will parse the type int, then the declarator which includes the variable name x and its initializer. The key function for parsing the ... = initializer; part is ParseDeclarator.
• As you step through, you'll see the parser consume the = token and then call a function to parse the expression on the right-hand side. This is often ParseAssignmentExpression. Step into it.

Phase 3: Parsing the Binary Expression and the Hand-off to Sema

You are now at the heart of expression parsing. Clang's parser is a recursive-descent parser that respects C++ operator precedence.

• The function ParseAssignmentExpression will in turn call other functions for expressions with higher precedence. You'll descend through a chain like ParseConditionalExpression, ParseBinaryExpression, and so on, until you reach ParseAdditiveExpression.
• Inside ParseAdditiveExpression, the parser will see the + token. After parsing the left-hand side (42) and the right-hand side (10), it will make the crucial call to Sema.
• Keep stepping until you see a line that looks like this:
  Result = SemaRef.ActOnBinOp(getCurScope(), Tok.getLocation(), Tok.getKind(), LHS, RHS);
• This is the hand-off. The Parser has done its job of recognizing the syntax of a binary operation. It now passes the operands (LHS and RHS) to Sema via ActOnBinOp.
• Use s to step into Sema::ActOnBinOp.

Phase 4: Witnessing AST Node Creation

You are now inside the semantic analysis engine.

• ActOnBinOp will perform type checking and other semantic validation. It often delegates the actual AST creation to a helper function. In this case, it will likely call CreateBuiltinBinOp. Step into it.
• Inside CreateBuiltinBinOp, you will finally find the line of code that allocates and constructs the AST node. It will look something like this:
  return new (Context) BinaryOperator(LHS, RHS, Opcode, ResultTy, ...);
• This is the moment of creation. Let's inspect the result.
  1. Use n to step over this line. The BinaryOperator has now been created, and a pointer to it is stored in a local variable (often called Result or something similar).
  2. Use the print command to inspect this variable.
     (lldb) print Result
     The debugger will print the memory address and type of the object: (clang::BinaryOperator *) $0 = 0x...
  3. Now, let's look inside it. You can dereference the pointer to see its members.
     (lldb) print *Result
     This will show you the contents of the BinaryOperator object, including pointers to its left-hand side (LHS) and right-hand side (RHS), and its opcode (e.g., BO_Add).
  4. For an even better view, you can ask the debugger to execute the node's dump() method, which is designed for this purpose.
     (lldb) expr Result->dump()

You have successfully traced an expression from raw text to a fully-formed, in-memory AST node and inspected its state.

Verifying with High-Level Tools

This low-level debugging view is incredibly powerful, but it's also useful to verify your understanding with higher-level tools that show the final result.

Using -ast-dump
You can ask Clang to print its entire AST for a given file. This shows you the final structure that your step-by-step process helped to build.

/path/to/your/build/bin/clang -Xclang -ast-dump -fsyntax-only test.cpp

The output for 42 + 10 will look something like this, clearly showing the BinaryOperator with its opcode and children:

`-BinaryOperator 0x... <col:13, col:20> 'int' '+'
  |-IntegerLiteral 0x... <col:13> 'int' 42
  `-IntegerLiteral 0x... <col:20> 'int' 10

Using Compiler Explorer
For quick experiments, the online Compiler Explorer tool is invaluable. It lets you type code and instantly see the corresponding AST.

Conclusion

In this lesson, you moved beyond pausing the compiler and learned how to actively trace its execution. By stepping through the code, you have witnessed the intricate dance between the Parser and Sema and pinpointed the exact moment an AST node is born. This ability to follow the flow of data and control through the codebase is the foundational skill for diagnosing bugs and understanding how to implement new features.

Key Takeaways:

• The path to creating an expression's AST node typically flows from Parser::Parse... functions to a Sema::ActOn... call.
• Inside Sema, a Build... or Create... helper function performs semantic checks and allocates the new AST node (e.g., new (Context) BinaryOperator(...)).
• The debugger's step, next, and print commands are your essential tools for tracing this flow.
• You can use expr Node->dump() in the debugger to get a nicely formatted view of an AST node you've paused on.
• Higher-level tools like clang -Xclang -ast-dump and Compiler Explorer are excellent for verifying the final structure of the AST.

You now have a solid understanding of Clang's AST and the process that creates it. In our next module, we will build on this foundation. We will shift from just observing the AST to actively using it, as you learn to write your own standalone Clang tools for code analysis and transformation.