Downloading Files with curl and wget

1. Introduction to wget and curl

Before diving into the commands, let's establish what these tools are and check that you have them. wget and curl are both command-line utilities for transferring data using various network protocols, most commonly HTTP and HTTPS.

• wget (Web Get) is a straightforward tool whose primary purpose is to download files from the internet. It's simple, robust, and excellent for downloading entire websites or large files.
• curl (Client for URL) is a more versatile tool. While it can download files, it's often used for more general data transfer, such as interacting with APIs or viewing the raw content of a webpage directly in your terminal. Think of it as a Swiss Army knife for data transfer.

As an Arch Linux user, you likely have these installed. You can verify with which wget and which curl. If not, you can easily install them:

sudo pacman -S wget curl

2. wget: The Reliable Downloader

Let's start with wget. Its design philosophy is simple: get the file and save it. This makes it very intuitive for its main purpose.

Key wget Commands

Based on the video, here are the essential wget commands you should know:

3. curl: The Versatile Data Transfer Tool

Now let's turn to curl. Its default behavior is different and highlights its flexibility: instead of saving a file, it prints the content directly to your terminal's standard output. This is perfect for a quick peek at a file or for piping data to other commands.

Key curl Commands

Here's a summary of the curl commands from the video:

4. wget vs. curl: Choosing the Right Tool

You might be wondering when to use wget and when to use curl. While they have overlapping features, they are designed for different primary use cases.

Here is a quick summary of the key differences:

• Primary Use:

  • wget is a downloader. It's built for reliably getting files from the web.
  • curl is a data transfer tool. It's built for interacting with URLs and is often the backbone of scripts that test APIs.

• Default Behavior:

  • wget <URL> downloads a file.
  • curl <URL> displays the content to standard output.

• Recursive Downloading:

  • wget is the king of recursive downloads (wget -r). It can mirror entire websites.
  • curl does not support this natively.

• Underlying Library:

  • curl is powered by libcurl, a library used in thousands of applications (including cars, TVs, and more) to handle data transfers. wget is a standalone program.

Rule of thumb:

• Use wget when your goal is to download files, especially large ones, multiple files, or entire directories.
• Use curl when you want to see the content of a URL quickly, test an API endpoint, or use its output in a command pipeline.

5. Hands-On Lab

Time to put this into practice. Open your terminal and let's download some files. I suggest creating a dedicated directory for this practice, for example: mkdir -p ~/practice/downloads && cd ~/practice/downloads.

Activity 1: Download a Hacking Tool with wget
A common task is to download security scripts. Let's grab LinPEAS, a popular privilege escalation checker.

1. Use wget to download it.

   wget https://github.com/carlospolop/PEASS-ng/releases/latest/download/linpeas.sh
   

2. Use ls -l to verify the file linpeas.sh has been downloaded.

Activity 2: Resuming a Large Download
Let's simulate downloading a large file that gets interrupted. We'll download the Arch Linux ISO.

1. Start the download but limit the speed so you have time to cancel it.

   wget --limit-rate=500k http://mirror.cs.pdx.edu/archlinux/iso/latest/archlinux-x86_64.iso
   

2. After about 10-15 seconds, press Ctrl+C to interrupt the download.
3. Use ls -l. You'll see a partially downloaded .iso file.
4. Now, resume the download. wget will see the partial file and pick up where it left off.

   wget -c http://mirror.cs.pdx.edu/archlinux/iso/latest/archlinux-x86_64.iso
   

5. Let it run for a few more seconds, then you can cancel it again with Ctrl+C. You have now successfully practiced resuming a download.

Activity 3: Inspecting Web Content with curl
Let's see curl's default behavior.

1. Run the following command. You'll see the raw HTML of the example website printed to your screen.

   curl http://example.com
   

2. Now, let's handle a redirect. http://google.com will redirect to https://www.google.com.

   
   
   
   # This will just show a "301 Moved" message
   curl http://google.com
   
   
   
   
   # This will follow the redirect and show the actual page content
   curl -L http://google.com
   

Activity 4: Saving Files with curl
Now let's use curl to save the same linpeas.sh script.

1. Save it with a custom name using -o.

   curl -o linpeas_from_curl.sh -L https://github.com/carlospolop/PEASS-ng/releases/latest/download/linpeas.sh
   

   (We use -L because GitHub release URLs often involve redirects).
2. Now save it with its original name using -O.

   curl -O -L https://github.com/carlospolop/PEASS-ng/releases/latest/download/linpeas.sh
   

3. You'll get a warning that linpeas.sh already exists. curl by default won't overwrite. If you wanted to, you'd remove the file first (rm linpeas.sh) before running the command.

Conclusion

You've now mastered two essential command-line tools for interacting with web resources. This skill is critical for everything from downloading security tools and exploit code to scripting interactions with web applications during a penetration test.

Key Takeaways:

• wget is your go-to tool for downloading files. Its key flags are -O (rename), -P (directory), -c (resume), and -i (input file).
• curl is a versatile data transfer tool. Its default is to print to the screen. Key flags are -o (save to file), -O (save with original name), and -L (follow redirects).
• Understanding the fundamental difference in their default behavior (save vs. print) is the key to choosing the right tool for the job.

Next Lesson Preview:
We've found open ports (ss), and now we can download files from them (wget, curl). But what's happening on the wire? How does the data actually look as it travels across the network? In our next lesson, we will dive into network packet analysis by learning how to capture and inspect network packets using tcpdump with basic filters. This will give you an even deeper view into network communications.