Python for Demographic and Market Data Analysis

Hello! Welcome back.

In our last lesson, we constructed a weighted scoring model to bring structure and objectivity to the complex process of comparing investment properties. We saw how criteria related to the market, the property itself, and the deal could be weighted and scored to produce a final, comparable number. A key part of that model relies on assessing market factors like population trends and economic strength.

Today, we transition from theory to practice and tackle the "how" of gathering that market data. This lesson's outcome is to write a Python script to fetch and analyze demographic and market data for a specific area using a public API. As a software engineer, you'll find this topic very familiar. We're essentially going to build the data pipelines that feed the "Market" section of the scoring model we just designed, making your analysis repeatable, scalable, and data-driven.

To start, let's consider why we need this data. Long-term demographic trends are a fundamental driver of real estate value. A growing population increases demand for housing, while a declining one can lead to stagnation.

1. The API Workflow: From Request to DataFrame

An Application Programming Interface (API) is a standard way for different software applications to communicate. Many data providers, from government statistics agencies to real estate portals, offer APIs that allow us to request and receive data in a structured format, most commonly JSON.

There are generally two ways to interact with an API in Python:

1. Using a Wrapper Library: A developer creates a Python package that handles the complexities of authentication and request formatting, giving you simple functions to call (e.g., get_population_data('Madrid')).
2. Direct HTTP Requests: You use a library like requests to manually construct and send HTTP requests and then parse the raw JSON response yourself.

We'll start by learning the process with a wrapper library, as it's often the most efficient method.

The key workflow you saw in the video can be summarized as:

1. Authentication: Obtain an API key and instantiate the API client object.
2. Discover: Search for the specific data series you need to find their unique IDs.
3. Fetch: Request the data for one or more series IDs.
4. Process: Load the data into a Pandas DataFrame for cleaning, analysis, and visualization.

2. A Spanish Example: Demographic Data from INE

Now, let's apply that exact workflow to a Spanish data source. The equivalent of FRED in Spain is the Instituto Nacional de Estadística (INE), the official source for demographic and economic data.

Fortunately, there's a Python wrapper library for the INE's 2021 Census API, making our job much easier.

Let's put this into practice. Say we want to get the total population for the provinces of Madrid and Barcelona to feed into our scoring model. Based on the censosine21 documentation, the process is straightforward.

First, install the library:
pip install censosine21

Now, we can write a script to fetch the population data. After exploring the available tables (using the static methods mentioned in the docs), we'd find that table 36321 contains population by province.

import pandas as pd
from censosine.ine import CensosINE21

# 1. Instantiate the CensosINE21 object
ine_api = CensosINE21()

# 2. Define the parameters for our request
# We want the 'Population' metric from table '36321'
# The variable 'PROV' corresponds to provinces
params = {
    'table': '36321',
    'metrics': 'Población',
    'variables': {
        'PROV': ['28 Madrid', '08 Barcelona']
    }
}

# 3. Make the request to the INE API
try:
    ine_api.post(params)
    
    # 4. The data is stored in the .data attribute as a list of dictionaries
    if ine_api.data:
        # Convert the list of dictionaries to a Pandas DataFrame
        df = pd.DataFrame(ine_api.data)
        
        # Clean up and display the data
        df_cleaned = df[['Nombre', 'Valor']]
        df_cleaned = df_cleaned.rename(columns={'Nombre': 'Province', 'Valor': 'Population'})
        
        print("2021 Population Data from INE:")
        print(df_cleaned)
    else:
        print("No data returned from the API.")

except Exception as e:
    print(f"An error occurred: {e}")

# Example Output:
# 2021 Population Data from INE:
#        Province  Population
# 0   28 Madrid     6751251
# 1  08 Barcelona     5714730

This script is a tangible tool that automates the collection of a key demographic indicator for your market analysis. You could easily expand it to pull other variables like age distribution, household size, or housing tenure, all of which are available through the INE API.

3. Direct API Interaction: Getting Property Data from Idealista

Wrapper libraries are great, but they don't exist for every API. For many services, especially commercial ones, you'll need to handle the HTTP requests and JSON parsing directly. Your software engineering background makes this a standard task.

Before diving into a real estate example, let's quickly review the core logic of handling JSON from an API response in Python.

Now, let's apply this direct-request method to the most relevant data source for us: Idealista, Spain's largest real estate portal. Idealista provides a developer API that gives access to its listings.

Note: The Idealista API requires you to register for a key. We will walk through the process using a detailed guide, and you can follow the steps to register on your own time to get your own credentials.

The process described in the article is a standard and powerful way to gather real-time market data. Let's summarize the key Python snippets from the article into a logical flow.

Step 1: Get Access Token (Authentication)
You send your credentials to get a temporary token.

# Simplified from the article
import requests
import base64

apikey = 'YOUR_API_KEY'
client_secret = 'YOUR_CLIENT_SECRET'

credentials = f"{apikey}:{client_secret}"
encoded_credentials = base64.b64encode(credentials.encode()).decode()

token_url = "https://api.idealista.com/oauth/token"
data = {"grant_type": "client_credentials"}
headers = {"Authorization": f"Basic {encoded_credentials}", ...}

response = requests.post(token_url, data=data, headers=headers)
token = response.json()["access_token"]

Step 2: Build Search URL and Make Request
You use the token to ask for the actual data.

# Simplified from the article
search_url = (
    "https://api.idealista.com/3.5/es/search?" +
    "operation=sale" +
    "&propertyType=homes" +
    "&center=40.4167,-3.70325" + # Madrid Center
    "&distance=5000" # 5km radius
)

headers = {'Authorization' : 'Bearer ' + token}
content = requests.post(search_url, headers=headers)
results = content.json()

# The property listings are inside the 'elementList' key
property_df = pd.DataFrame(results['elementList'])
print(property_df.head())

This property_df DataFrame would contain a wealth of information for all listings in your defined area: price, size (m²), number of rooms, location, and more. You could use this to calculate the average price per square meter in a neighborhood, analyze the distribution of property types, or identify outliers—all essential components of market analysis.

property_df['price_per_sqm'] = property_df['price'] / property_df['size']

Conclusion

In this lesson, you've added a powerful, practical skill to your real estate analysis toolkit. You now have the ability to go beyond static reports and programmatically fetch live market and demographic data, which is the foundation of any serious investment analysis.

Key Takeaways:

• You can access public data using Python via wrapper libraries (like censosine21 for INE data) or through direct HTTP requests (like for the Idealista API).
• The general workflow is to authenticate (get a key/token), build a query with your desired parameters, fetch the data, and parse the JSON response into a Pandas DataFrame.
• With these skills, you can automate the data collection for the "Market" criteria of your scoring model, pulling fresh data on population, housing stock, and active property listings.

Preview of the Next Module:
This lesson concludes our module on Market Analysis and Due Diligence. You now have a robust framework for finding and evaluating potential deals. Our next module, "Time Value of Money and Cash Flow Projections," shifts our focus from the present state of a property to its future performance. We will begin by calculating the present and future value of cash flows, the foundational concept for modeling an investment's returns over its entire holding period.