Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 9

Extracting Text from PDFs and Text Files in Python

Recommended resources

Welcome to the third module of our course. In the previous module, we established a solid foundation for understanding semantic search. You now know how to represent text as vectors and measure their similarity, and you can clearly distinguish between traditional keyword search and modern semantic search. This sets the stage for building the "Retrieval" component of our RAG system.

This lesson marks our first step into the practical construction of a RAG pipeline: preparing the knowledge base. Before we can retrieve relevant information, we first need to ingest it from its source. Your goal for this lesson is to learn how to load and extract text from common file formats like PDFs and plain text files using standard Python libraries and frameworks. This is the foundational I/O operation upon which the entire RAG process is built.

The First Hurdle: Getting Data In

A RAG system's power comes from its ability to access external knowledge. This knowledge often resides in a collection of documents—research papers, internal wikis, product manuals, legal contracts—which are frequently stored as PDF or text files. Our first task is to programmatically read these files and extract their raw textual content into a format our application can work with.

As a software engineer, you can think of this as the data ingestion or ETL (Extract, Transform, Load) phase of our system. We need a reliable way to extract text from various file formats.

This image shows a Python snippet using LangChain's `PyPDFLoader`. It demonstrates loading a PDF and printing the resulting `Document` object, which contains the extracted text content from a page.

This task is not always trivial. Different file formats have unique internal structures. Fortunately, we can rely on specialized Python libraries to handle the low-level parsing details.

Under the Hood: Direct Text Extraction with pypdf

Before we jump into high-level frameworks, it's useful to understand the core libraries they are often built upon. For PDFs, pypdf is a widely used library that allows for direct text extraction. Understanding its mechanics will give you a clearer picture of what the abstractions are doing for you.

The process is straightforward: you open the PDF file, create a PdfReader object, access individual pages, and then call a method to extract the text from each page.

Extract text from PDF File using Python

This article from GeeksforGeeks provides a concise example of using the pypdf library directly.

Please read the first section, Extracting text from a PDF file using the pypdf library. Focus on the code example and the step-by-step explanation of how the PdfReader object is used to access and extract text from each page. This demonstrates the fundamental logic of PDF parsing in Python.

While using libraries like pypdf directly is perfectly viable, RAG development frameworks like LangChain and LlamaIndex offer a more integrated and standardized approach that simplifies the entire process.

Abstraction and Standardization: Document Loaders

Frameworks designed for building LLM applications provide a powerful abstraction called Document Loaders. A loader is a component specifically designed to handle the complexities of a particular file format (PDF, TXT, CSV, HTML, etc.) and return the data in a standardized structure. This allows you to switch between data sources with minimal code changes and ensures the output is compatible with the next stages of the RAG pipeline, such as text splitting and embedding.

Let's explore how this works in both LangChain and LlamaIndex.

Loading with LangChain

LangChain offers a rich collection of document loaders. For our purposes, PyPDFLoader and TextLoader are the most relevant. The workflow is simple: instantiate the appropriate loader with the file path and call the .load() method.

The article "Loading and Splitting Documents with LangChain" provides a clear walkthrough.

Loading and Splitting Documents with LangChain

This guide demonstrates the core principles of document loading in LangChain.

Please read the following parts: Start with the section on LangChain Document Loaders to get an overview. Note the simple initialization for PyPDFLoader and TextLoader. Follow the complete example of loading a document to see how the load() method is used. Finally, review how to inspect the loaded documents. This is crucial for understanding the output format.

As you've just read, the .load() method doesn't return raw text. It returns a list of Document objects. This is a key concept. For PyPDFLoader, each Document object in the list typically corresponds to a single page of the PDF.

This short video provides another quick, practical demonstration of this process.

Python RAG Tutorial (with Local LLMs): AI For Your PDFs

The video "Python RAG Tutorial (with Local LLMs)" shows a straightforward implementation of document loading with LangChain.

Watch the segments on loading documents from a folder and then inspecting the output. This will visually reinforce what a loaded Document object contains.

Loading with LlamaIndex

LlamaIndex provides a similar, and in some ways more convenient, approach. Its SimpleDirectoryReader can automatically ingest all supported files from a specified directory, detecting their types and using the appropriate parser without you needing to select a specific loader for each file.

The following video introduces the LlamaIndex pipeline and demonstrates this powerful feature.

Introduction to LlamaIndex with Python (2025)

This "Introduction to LlamaIndex" video gives an excellent overview of its data ingestion pipeline.

First, watch the practical implementation from creating a data folder up to loading the documents using SimpleDirectoryReader. Notice how a single reader can ingest the contents of an entire folder. Next, jump to the detailed breakdown of the resulting Document object from what the documents look like. Pay close attention to the structure, especially the metadata and text properties.

The Document Object: A Universal Data Structure

You've now seen document loading in action using two major frameworks. A critical pattern emerges: the output is consistently a Document object (or a list of them). This object is a standardized container for the extracted data.

Regardless of the framework, a Document object almost always contains two key pieces of information:

  1. Content: The actual text extracted from the source file. In LangChain, this is stored in the page_content attribute. In LlamaIndex, it's the text attribute.
  2. Metadata: A dictionary of data about the content. This is invaluable and typically includes:
    • source or file_path: The original file the text came from.
    • page: The page number (for PDFs).
    • Other details like creation date, file type, etc.

Treating ingested data as these structured objects is a powerful design choice. The content is what we will embed and search, while the metadata is essential for filtering results (e.g., "only find information from documents published after 2022") and for citing sources in the final RAG response, which enhances trustworthiness.

Conclusion

In this lesson, you have mastered the first practical step in building a RAG system. You now understand how to bridge the gap between raw files on a disk and the structured data your application needs.

Here are your key takeaways:

  • Document loading is the process of extracting text from source files like PDFs and TXT files.
  • Low-level libraries like pypdf provide the core extraction functionality.
  • RAG frameworks like LangChain and LlamaIndex offer high-level Document Loaders (PyPDFLoader, SimpleDirectoryReader) that abstract away file-specific details.
  • The standard output of a loader is one or more Document objects, which are structured containers holding the extracted text content and crucial metadata (like the source file and page number).

We have successfully loaded our documents. However, a single loaded Document (e.g., a full page of a dense PDF) is often still too large to be used effectively with an embedding model. In our next lesson, we will address this by exploring why and how to split these documents into smaller, semantically meaningful chunks.