Python AST for RAG Chunking

Describe how ingestion, chunking, embeddings, vector search, retrieval, and generation interact in a basic RAG pipeline.

Explain how embedding vectors and similarity search support retrieval without requiring an embedding model to understand Python syntax directly.

Explain how chunk boundaries affect retrieval quality and the context ultimately supplied to a language model.

Contrast character-based, token-based, and syntax-aware strategies for chunking Python source code.

Assess a code chunk for semantic cohesion, contextual completeness, size, and retrievability.

Distinguish a chunk's verbatim source text from metadata and optionally added contextual text.

Distinguish Python source modules, packages, and package initializers when scanning a repository.

Interpret absolute imports, relative imports, aliases, and from-import statements in Python source.

Recognize `async def` as a distinct function definition and identify its syntactic boundaries without executing it.

Trace the lexical nesting of classes, functions, methods, and nested functions in a Python file.

Explain how local, enclosing, global, and built-in scopes affect the meaning of a name reference.

Interpret the effects of global and nonlocal declarations on name resolution.

Construct stable qualified names for top-level and nested Python definitions.

Explain the relationship among Python source text, tokens, grammar, concrete syntax, and an abstract syntax tree.

Parse valid Python source with ast.parse and inspect the resulting ast.Module root.

Use ast.dump to investigate node fields and attributes in a readable tree representation.

Recognize AST nodes for classes, synchronous and asynchronous functions, and lambda expressions.

Recognize AST nodes for imports, assignments, annotated assignments, and control-flow statements.

Identify interpreter-version differences that can affect supported Python syntax and AST node shapes.

Handle SyntaxError information when a source file cannot be parsed.

Apply parent, child, ancestor, descendant, depth, and subtree terminology to a Python AST.

Enumerate immediate child nodes with ast.iter_child_nodes.

Use ast.walk when traversal order and ancestry are not required.

Implement a recursive depth-first traversal of an AST.

Create an ast.NodeVisitor with handlers for selected node types.

Maintain an ancestor stack during traversal to recover scopes and qualified names.

Distinguish direct members of a class or function from definitions nested deeper inside statements.

Interpret lineno, end_lineno, col_offset, and end_col_offset on AST nodes.

Account for UTF-8 byte-based column offsets when mapping AST positions to source text.

Extract a node's original text with ast.get_source_segment.

Extract complete line-oriented spans while preserving original line endings.

Include decorators when calculating the source span of a class or function.

Identify module, class, and function docstrings from their AST representation.

Implement a `tokenize`-based policy for recovering and associating comments, including shebang and encoding lines that are absent from the AST.

Compare verbatim span extraction with ast.unparse for code-chunking use cases.

Define a framework-independent chunk record containing source text, file location, symbol identity, span, and chunk type.

Classify top-level classes, synchronous and asynchronous functions, imports, module docstrings, and other module-level statements as chunk candidates.

Group adjacent import statements into a coherent import-context chunk.

Group suitable adjacent module-level statements without crossing definition boundaries.

Prevent duplicate chunk candidates when multiple traversal rules encounter the same definition.

Emit one source-faithful chunk for each ordinary top-level function or class.

Attach module paths and qualified symbol names as retrieval metadata.

Generate deterministic chunk identifiers from stable source attributes.

Measure chunk size with both source-character counts and model-token counts.

Apply configurable minimum and maximum size policies, merging undersized chunks only when semantically appropriate and never discarding source content.

Split an oversized class at direct method and nested-class boundaries while preserving its header, docstring, class-level preamble, and symbol identity.

Split an oversized function at top-level statement boundaries while preserving every statement and the function's identity.

Apply a token-based fallback splitter when an individual structural unit exceeds the size limit.

Implement statement- or line-level overlap between adjacent fallback subchunks without duplicating structural headers.

Create contextual headers that identify a chunk's module, enclosing symbols, and source location.

Keep verbatim code separate from synthetic context so consumers can choose what to embed or display.

Extract imported module names, imported symbols, and aliases from import nodes.

Distinguish name definitions, loads, stores, and deletions using AST context fields.

Collect names referenced within a chunk without incorrectly attributing names from nested scopes.

Identify function and method calls represented by Name and Attribute nodes.

Associate decorators, base classes, return annotations, and parameter annotations with definition metadata.

Represent lightweight definition-to-reference relationships without claiming full static type resolution.

Explain the limits imposed by dynamic imports, runtime attribute access, monkey patching, and generated code.

Discover Python source files with pathlib while excluding configurable virtual-environment, build, cache, and generated-code directories.

Read Python files using tokenize.open so declared source encodings are respected.

Normalize repository-relative paths for stable metadata across machines.

Record parse failures without aborting processing of the remaining repository.

Cache file hashes so unchanged files can be skipped during incremental re-indexing.

Process untrusted source without importing or executing it.

Serialize chunk records to JSON Lines for downstream indexing.

Prepare separate embedding text and display text from a structured chunk record.

Store chunks in a generic vector index while retaining filterable metadata.

Retrieve candidate chunks for a natural-language code question using vector similarity.

Assemble retrieved chunks into a context window under a token budget.

Use qualified names, file paths, and chunk types to filter or rerank retrieval results.

Define representative retrieval questions and expected source targets for evaluating a Python code corpus.

Measure retrieval success with recall at k on a small labeled question set.

Compare naive fixed-size chunking against AST-aware chunking on the same evaluation set.

Organize parsing, traversal, span extraction, splitting, metadata, and serialization behind clear Python interfaces.

Express chunking behavior through a typed configuration object.

Implement a command-line interface that scans a repository and emits configured chunk records as JSON Lines.

Create unit fixtures covering decorated, asynchronous, nested, annotated, version-sensitive, and malformed Python constructs.

Test that emitted spans map byte-for-byte to the intended regions of the original source.

Test that splitting policies respect size limits without losing structural units and introduce only explicitly configured overlap.

Add structured diagnostics for skipped files, parse failures, fallback splits, and emitted chunk counts.

Tune structural boundaries, context enrichment, and size thresholds using retrieval-evaluation evidence.