Managing Dependencies with pip and requirements.txt

1. The requirements.txt Workflow Revisited

As a quick refresher from our last session, the core of Python dependency management revolves around pip (the package installer) and a simple text file, requirements.txt. This file acts as a manifest, listing all the external libraries your project needs to run.

The workflow is straightforward:

1. Generate the file: You capture the packages from your active virtual environment.
2. Recreate the environment: Another developer (or a deployment server) uses this file to install the exact same packages.

The blog post from NamasteDev provides a good summary of these fundamental concepts and commands.

The image below shows what this looks like in practice. First, pip freeze > requirements.txt creates the file. Then, pip install -r requirements.txt reads the file and installs the listed packages. In this case, since the packages are already present, it confirms that the "Requirement [is] already satisfied."

The resulting requirements.txt file is a simple list of packages and their versions, which you will commit to your Git repository.

2. Version Pinning: The Key to Reproducibility

Notice the == symbol in the requirements.txt file. This is called version pinning. It instructs pip to install the exact version specified (e.g., requests==2.28.2).

Why is this so important?

• Consistency: It eliminates the "it works on my machine" problem by ensuring every environment (developer, testing, production) uses the exact same code.
• Stability: A new version of a library, even a minor one, could introduce a bug or a breaking change. Pinning versions prevents your application from breaking unexpectedly due to an automatic update of a dependency.

For building applications (like the backend systems at Infilect), pinning exact versions is the standard and recommended practice. While other operators exist (>= for minimum version, ~= for compatible release), they are more suited for libraries intended to be used by other developers, where some flexibility is desired. For your applications, always aim for the predictability that == provides.

3. The Problem with pip freeze

While pip freeze is effective, it has one significant drawback: it's not very smart. It dumps every single package installed in your virtual environment into the file, including the dependencies of your dependencies (known as transitive dependencies).

For example, when you install requests, pip also installs its dependencies, such as charset-normalizer, idna, and urllib3. When you run pip freeze, your requirements.txt will contain all of them.

This creates two problems:

1. Noise: The file becomes cluttered, making it difficult to see what your project's direct dependencies are.
2. Maintenance: If you want to upgrade requests, which of its dependencies should you also upgrade or remove? It's not clear.

A much cleaner approach is to specify only the top-level packages your project directly imports and let pip resolve the sub-dependencies during installation.

4. A Better Way: Generating Requirements from Imports

Fortunately, there are tools that can generate a clean requirements file by scanning your source code for import statements. One of the most straightforward tools for this is pipreqs.

The following video clearly demonstrates the problem with pip freeze and how pipreqs solves it.

By using pipreqs, you create a requirements.txt file that serves as a clear declaration of your project's direct needs, which is much easier to read and maintain.

5. Hands-On Practice

Let's put this into practice and see the difference firsthand.

1. Set up your project:
   Open your terminal, create a new project directory, and cd into it.

   mkdir my-api-client
   cd my-api-client
   

2. Create and activate a virtual environment:

   
   
   
   # On Mac/Linux
   python3 -m venv venv
   source venv/bin/activate
   
   
   
   
   # On Windows
   python -m venv venv
   .\venv\Scripts\activate
   

3. Create a simple Python script:
   Create a file named main.py and add the following code. This script will use the requests library to fetch some data from a public API.

   import requests
   
   def get_user_data(user_id):
       url = f"https://jsonplaceholder.typicode.com/users/{user_id}"
       try:
           response = requests.get(url)
           response.raise_for_status()  # Raises an exception for bad status codes (4xx or 5xx)
           print("User Data:")
           print(response.json())
       except requests.exceptions.RequestException as e:
           print(f"An error occurred: {e}")
   
   if __name__ == "__main__":
       get_user_data(1)
   

4. Install the package and generate requirements with pip freeze:
   First, install the requests package. Then, use pip freeze to generate a requirements file.

   pip install requests
   pip freeze > requirements_frozen.txt
   

   Now, open requirements_frozen.txt. You will see requests along with several other packages it depends on (like certifi, charset-normalizer, idna, urllib3).

5. Generate clean requirements with pipreqs:
   Now, let's try the better way. First, install pipreqs itself. Then, run it in your project directory.

   pip install pipreqs
   pipreqs .
   

   This command will scan main.py, find the import requests line, and create a new requirements.txt file.

   Now, open requirements.txt. You will see it contains only one line: requests==.... This is a much cleaner and more accurate representation of your project's direct dependencies.

When a collaborator clones your project and runs pip install -r requirements.txt, pip will automatically fetch requests and all its necessary sub-dependencies. The outcome is the same, but your dependency manifest is far more maintainable.

Conclusion

You have now moved beyond the basics and learned the professional standard for managing Python dependencies. This is a critical building block for creating reliable and scalable backend applications.

Key Takeaways:

• A requirements.txt file is essential for reproducible environments, a cornerstone of professional software development.
• Pinning versions with == is the standard for applications to ensure stability and consistency.
• pip freeze is a blunt tool that includes all transitive dependencies, creating noise.
• Tools like pipreqs provide a much cleaner solution by generating requirements based on your project's actual import statements.
• Always commit your requirements.txt to Git, but add your venv folder to .gitignore.

In the next lesson, we will shift our focus from the local development environment to the outside world. We'll explore the HTTP request-response cycle, the fundamental communication protocol of the web. This will lay the groundwork for understanding how your future Django applications will interact with clients and other services.