Gathering OSINT with theHarvester and Recon-ng

1. TheHarvester: The OSINT Multi-Tool

theHarvester is a classic open-source intelligence (OSINT) tool. Its job is to gather emails, subdomains, hosts, employee names, open ports, and banners from various public sources like search engines and public databases (e.g., certificate transparency logs). Think of it as an automated script that runs dozens of Google Dorks and other queries for you, all at once.

Understanding theHarvester's Commands

Since you're comfortable with command-line tools, the best way to understand theHarvester is to look at its help menu. It's pre-installed on Kali Linux, so you can bring this up in your terminal with theHarvester -h.

The most important flags for a basic scan are:

• -d: The target domain (e.g., tesla.com).
• -b: The data source to use. You can specify one (e.g., bing) or multiple sources (bing,google,crtsh). Using all will query every available source.
• -l: Limits the number of search results from sources like search engines.
• -f: Saves the output to an HTML and/or XML file for your records.

Hands-on Lab 1: Running Your First Harvest

Time to put this into practice in your Kali VM. We'll use a public bug bounty target for this exercise, like tesla.com, to ensure our activities are directed at an organization that permits reconnaissance.

1. Open your terminal.

2. Run a broad scan. Let's start by telling theHarvester to query all its sources for tesla.com and limit the results to 200.

   theHarvester -d tesla.com -l 200 -b all
   

   You will see many sources being queried. Some will yield results, while others will report missing API keys. This is expected. The goal here is to see the breadth of sources the tool can check.

3. Run a focused scan and save the results. Now, let's run a more targeted scan using sources that are known to be effective without API keys and save the output.

   theHarvester -d tesla.com -l 500 -b bing,crtsh,duckduckgo,dnsdumpster -f tesla-harvest
   

   • bing, duckduckgo: General search engines.
   • crtsh: Searches certificate transparency logs, an excellent source for finding subdomains.
   • dnsdumpster: A popular service that maps out a domain's DNS records.
   • -f tesla-harvest: This will create tesla-harvest.html and tesla-harvest.xml in your current directory. Open the HTML file in a browser to see a nicely formatted report of the subdomains and emails found.

Examine the output. You have just automated a significant amount of the manual work from the previous lesson, discovering dozens or even hundreds of potential hosts in minutes.

2. Recon-ng: The Reconnaissance Framework

If theHarvester is a multi-tool, Recon-ng is a full workshop. It's a powerful, modular framework written in Python with an interactive shell that will feel familiar after working with Linux. Its key advantage is its structure:

• Workspaces: You create separate workspaces for each target, keeping your findings organized.
• Modular Design: It uses individual modules for specific tasks (e.g., a module for querying Bing, another for WHOIS, another for brute-forcing subdomains).
• Database Backend: All data collected by modules is automatically saved into a structured database within the workspace. This allows you to easily query, manage, and build upon your findings.

Hands-on Lab 2: A Structured Recon Workflow

The best way to learn Recon-ng is to use it. This lab will walk you through a standard workflow: setting up a workspace, finding and installing modules, and running them to gather data.

Step 1: Setup and Module Installation

First, we'll start recon-ng, create a workspace for our target, and install the modules we need from the marketplace.

Now, perform these actions in your terminal:

1. Start the tool: recon-ng
2. Create and load a new workspace: workspaces create tesla_recon
3. Search the marketplace for modules related to domains: marketplace search domain
4. Install the following modules. Remember to use the full path as shown in the marketplace search results:
   • marketplace install recon/domains-contacts/whois_pocs
   • marketplace install recon/domains-hosts/google_site_web
   • marketplace install recon/domains-hosts/bing_domain_web
   • marketplace install recon/domains-hosts/brute_hosts

Step 2: Harvesting Contacts with the whois_pocs Module

This module automates the WHOIS lookups we did manually in the last lesson to find Points of Contact (POCs).

Follow along in your terminal:

1. Load the module: modules load recon/domains-contacts/whois_pocs
2. View information and options for this module: info
3. Set the target domain: options set SOURCE tesla.com
4. Run the module: run
5. After it finishes, view the results: show contacts

The module automatically queries WHOIS records and neatly populates the contacts table in your database with any email addresses it finds.

Step 3: Enumerating Subdomains with Search Engines

Next, we'll use modules that automate Google and Bing dorking to find subdomains.

Now it's your turn:

1. Return to the main prompt: back
2. Load the Google module: modules load recon/domains-hosts/google_site_web
3. Set the target (it might be pre-filled from the last module, but it's good practice to check with info): options set SOURCE tesla.com
4. Run it: run
5. View the results. Notice how new entries are added to the hosts table: show hosts
6. Repeat the process for the Bing module (recon/domains-hosts/bing_domain_web). You will see that different sources often yield different results, which is why using multiple modules is so important.

Step 4: Brute-Forcing Subdomains with brute_hosts

Our final module takes a more active approach. It uses a wordlist of common subdomain names (like www, mail, dev, api) and attempts to resolve each one to see if it exists.

Let's run it:

1. Return to the main prompt: back
2. Load the brute-force module: modules load recon/domains-hosts/brute_hosts
3. Check the options with info. The SOURCE should still be tesla.com. Notice it uses a default wordlist.
4. Run it: run
5. Check the hosts table again with show hosts. You will likely see even more subdomains discovered by this module.

You have now completed a structured reconnaissance workflow, gathering data from multiple sources and consolidating it all into a single, queryable database.

Conclusion

Congratulations! You've successfully automated your passive reconnaissance process using two industry-standard tools. You've moved beyond single manual queries to efficient, large-scale data gathering.

Key Takeaways:

• theHarvester is a fantastic tool for a quick, initial sweep. It's simple to use and queries many sources at once, making it ideal for getting a broad overview of a target.
• Recon-ng is a professional-grade framework for deep, methodical reconnaissance. Its workspace and database structure allow you to organize and build upon your findings over time, which is crucial for complex penetration tests.
• The goal of these tools is to build a list of potential targets—subdomains, IP addresses, and people. This list forms the foundation for the next, more active phases of a security assessment.

Next Lesson Preview:
So far, our reconnaissance has been entirely passive (with the slight exception of the DNS brute-forcing). We've gathered a list of potential hostnames, but we don't know if they are actually online and responsive. In the next lesson, we will begin the Scanning phase. You will learn to perform host discovery on a network using Nmap ping scans, a fundamental technique for turning your list of potential targets into a list of confirmed, live systems.