K-Means Clustering: Unsupervised Learning

1. What is Clustering and How Does K-Means Work?

Clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar to each other than to those in other groups. Imagine you have a dataset of customer information (like age and spending habits) without any labels. A clustering algorithm could help you automatically identify distinct customer segments, like "young high-spenders" or "older budget-shoppers."

K-Means is one of the most popular algorithms for this task. Its goal is to partition data into a pre-specified number of clusters, $k$.

The K-Means Algorithm

The algorithm is surprisingly straightforward and works in an iterative fashion. It's often described as an "Expectation-Maximization" process.

Here's a summary of those steps:

1. Choose k: You decide how many clusters you want to find.
2. Initialize Centroids: Randomly select $k$ data points from your dataset to act as the initial "centers" of your clusters. These are called centroids.
3. Assign Points to Clusters: For every data point, calculate its distance to each of the $k$ centroids. Assign the data point to the cluster of the nearest centroid.
4. Update Centroids: After assigning all points, recalculate the position of the $k$ centroids. The new centroid is the mean (average) of all data points assigned to that cluster.
5. Repeat: Repeat the assignment (Step 3) and update (Step 4) steps until the centroids no longer move significantly. When this happens, the algorithm has converged, and you have your final clusters.

2. How to Choose the Right Number of Clusters (K)?

The biggest challenge with K-Means is that you have to choose the value of $k$ yourself. How do you know if you should look for 3 clusters or 5? Choosing a good value for $k$ is a key part of the art and science of clustering.

The most common technique is the Elbow Method.

The core idea is to run the K-Means algorithm multiple times with a different number of clusters (e.g., $k=1,2,3,...,10$) and measure the "quality" of the clustering for each $k$. For K-Means, a common quality metric is the Sum of Squared Errors (SSE), also known as inertia.

SSE (Inertia) is the sum of the squared distances between each data point and its assigned centroid.

A lower SSE means the points are, on average, closer to the center of their clusters, which is good. As you increase $k$, the SSE will always decrease. If $k$ is equal to the number of data points, SSE will be 0, but that's not useful clustering.

We are looking for the point of diminishing returns—the "elbow" in the plot of SSE vs. $k$.

The point where the SSE curve bends sharply, like an elbow, is a good candidate for the optimal number of clusters. Before this point, adding a new cluster significantly reduces the error. After this point, adding more clusters doesn't provide much benefit.

3. Implementing K-Means in Python

Now, let's put theory into practice. We'll use the scikit-learn library, which provides an efficient implementation of K-Means. We'll follow a practical example that also highlights a critical data preprocessing step: feature scaling.

Since K-Means relies on distance calculations, it's very sensitive to the scale of your features. If one feature (e.g., income in dollars) has a much larger range than another (e.g., age in years), the distance calculation will be dominated by the larger feature. To prevent this, we must scale our data first.

We will follow the codebasics video tutorial, breaking it down into a few parts.

Part A: A First Attempt (Without Scaling)

First, let's see what happens when we apply K-Means to unscaled data.

As you saw, the clusters are not well-formed. The green and black clusters seem to be mixed incorrectly. This is a classic symptom of features being on different scales. The y-axis (income) has a range in the tens of thousands, while the x-axis (age) has a range of a few dozen. The algorithm is giving too much weight to income.

Part B: The Solution - Feature Scaling

To fix this, we'll use MinMaxScaler from scikit-learn to scale both features to a range between 0 and 1.

Part C: K-Means on Scaled Data

With our data properly scaled, we can now run K-Means again and visualize the much-improved results, including the final centroid locations.

The difference is night and day! The clusters are now clearly separated and make intuitive sense. This demonstrates that feature scaling is a crucial preprocessing step for K-Means.

Part D: Implementing the Elbow Method in Code

Finally, let's write the code to generate the elbow plot we discussed earlier to programmatically find the best value for $k$.

The resulting plot clearly shows an "elbow" at $k=3$, confirming that our choice was a good one for this dataset.

Conclusion

Congratulations! You have just learned one of the most important algorithms in unsupervised machine learning. It's a powerful tool for discovering hidden groups in your data when you don't have the benefit of labels.

Key Takeaways:

• K-Means is an unsupervised algorithm that partitions data into $k$ distinct clusters.
• It works through an iterative process of assigning data points to the nearest centroid and then updating the centroid's position.
• The Elbow Method is a common technique for choosing the optimal number of clusters, $k$, by plotting the Sum of Squared Errors (SSE or inertia) for a range of $k$ values.
• Feature scaling is essential for K-Means, as it is a distance-based algorithm. Features on different scales can lead to poor clustering results.
• In scikit-learn, you use sklearn.cluster.KMeans, and you can find the SSE value in the .inertia_ attribute after fitting the model.

Preview of the Next Lesson:
In our practical example, we worked with data that had only two features (age and income), which was easy to visualize. But what happens when you have hundreds or even thousands of features, as is common in data science? This is known as the "curse of dimensionality." In the next lesson, we will tackle this problem by learning about Principal Component Analysis (PCA), a powerful technique for dimensionality reduction.