Implementing Linear Regression from Scratch

1. The Goal of Simple Linear Regression

Simple linear regression aims to model the relationship between two variables by fitting a straight line to the observed data. One variable is considered the independent variable (the feature, $x$), and the other is the dependent variable (the target, $y$).

The core idea is to find the "best-fit" line that passes through the data points, which can then be used to make predictions.

The equation for this line is likely familiar to you from basic algebra:

$y=b_0+b_{1}x$

• $y$ is the predicted value of the target variable.
• $x$ is the value of the input feature.
• $b_1$ is the slope of the line. It tells us how much $y$ changes for a one-unit increase in $x$.
• $b_0$ is the y-intercept. It's the predicted value of $y$ when $x$ is zero.

Our entire task is to find the optimal values for the coefficients $b_0$ and $b_1$ that result in the "best-fit" line. But what makes a line the "best"?

2. Finding the "Best" Line: The Method of Ordinary Least Squares (OLS)

The "best" line is the one that minimizes the total error between its predictions and the actual data points. We can't just sum the errors ($y_{actual}-y_{predicted}$), because positive and negative errors would cancel each other out.

Instead, we use the Sum of Squared Errors (SSE), also known as the cost function. We square each error before summing them up. This ensures all error contributions are positive and penalizes larger errors more heavily.

$SSE={\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2={\sum }_{i=1}^n(y_i-(b_0+b_1{x}_i){)}^2$

Here, $y_i$ is the actual value and ${\hat{y}}_i$ is the predicted value for the $i$-th data point.

Our goal is to find the values of $b_0$ and $b_1$ that minimize this SSE function. This is a classic optimization problem that can be solved with calculus. Since our SSE function depends on two variables ($b_0$ and $b_1$), we find the minimum by taking the partial derivative with respect to each variable, setting both derivatives to zero, and solving the resulting system of equations.

The following video provides an excellent, detailed walkthrough of this mathematical derivation from first principles. It's a fantastic way to see the "why" behind the formulas we'll be using.

The Resulting Formulas

After all the calculus, we are left with two elegant, closed-form equations for our coefficients. These formulas tell us exactly how to calculate the optimal slope ($b_1$) and intercept ($b_0$) directly from our data.

The slope ($b_1$) is the covariance of x and y divided by the variance of x:

$b_1=\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}$

The intercept ($b_0$) is calculated to ensure the line passes through the center of our data (the point $(\bar{x},\bar{y})$):

$b_0=\bar{y}-b_1\bar{x}$

Where:

• $\bar{x}$ is the mean of the input values.
• $\bar{y}$ is the mean of the output values.

This approach is powerful because it gives us a direct analytical solution, avoiding the need for iterative approximation methods like gradient descent (which we'll use for more complex models).

3. A Step-by-Step Numerical Example

Before we write any Python code, let's solidify our understanding by calculating the coefficients for a tiny dataset by hand. This will make the subsequent code implementation much clearer.

Let's use the simple example of study hours vs. test scores from the article "Simple Linear Regression: Complete Guide".

Let's recap the key steps from that example here:

Data:

1. Calculate Means:

   • $\bar{x}=(1+2+3+4+5)/5=3$
   • $\bar{y}=(65+70+80+85+90)/5=78$

2. Calculate Slope ($b_1$):

   • Numerator: $\sum (x_i-\bar{x})(y_i-\bar{y})$
     • $(1-3)(65-78)+(2-3)(70-78)+(3-3)(80-78)+(4-3)(85-78)+(5-3)(90-78)=26+8+0+7+24=65$
   • Denominator: $\sum (x_i-\bar{x}{)}^2$
     • $(1-3{)}^2+(2-3{)}^2+(3-3{)}^2+(4-3{)}^2+(5-3{)}^2=4+1+0+1+4=10$
   • $b_1=65/10=6.5$

3. Calculate Intercept ($b_0$):

   • $b_0=\bar{y}-b_1\bar{x}=78-(6.5\times 3)=78-19.5=58.5$

Our final model is: Test Score = 58.5 + 6.5 * Study Hours.

4. Implementing Simple Linear Regression in Python

Now for the most exciting part: translating these formulas into a working Python model. We'll create a class with fit and predict methods, mimicking the structure of scikit-learn models. This provides great practice for building more complex algorithms later.

The same CampusX video we used for the math derivation also includes a clear, from-scratch implementation.

Here is a full code block that puts everything together. It defines the class as shown in the video and then tests it on a sample dataset.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression as SklearnLinearRegression

# Our from-scratch Linear Regression class
class SimpleLinearRegression:
    def __init__(self):
        self.b0 = None  # Intercept
        self.b1 = None  # Slope

    def fit(self, X_train, y_train):
        """
        Calculates the optimal b0 and b1 coefficients using OLS.
        X_train should be a 1D NumPy array or Pandas Series.
        y_train should be a 1D NumPy array or Pandas Series.
        """
        # Ensure inputs are NumPy arrays
        X_train = np.array(X_train)
        y_train = np.array(y_train)

        x_mean = np.mean(X_train)
        y_mean = np.mean(y_train)

        # Calculate b1 (slope)
        numerator = np.sum((X_train - x_mean) * (y_train - y_mean))
        denominator = np.sum((X_train - x_mean)**2)
        
        self.b1 = numerator / denominator
        
        # Calculate b0 (intercept)
        self.b0 = y_mean - (self.b1 * x_mean)
        
        print(f"Model trained: b0 (intercept) = {self.b0:.4f}, b1 (slope) = {self.b1:.4f}")

    def predict(self, X_test):
        """
        Makes predictions for new data points.
        X_test should be a 1D NumPy array or Pandas Series.
        """
        if self.b0 is None or self.b1 is None:
            raise Exception("Model has not been trained yet. Call fit() first.")
        
        return self.b0 + self.b1 * np.array(X_test)

# --- Let's test our implementation ---

# 1. Create a sample dataset
# Let's use a dataset where we know the relationship: y = 2 + 3x + noise
X = 2 * np.random.rand(100, 1)
y = 2 + 3 * X + np.random.randn(100, 1)

# Split the data
# Note: For simple linear regression, X should be 1D. We use .ravel() to flatten it.
X_train, X_test, y_train, y_test = train_test_split(X, y.ravel(), test_size=0.2, random_state=42)

# 2. Train our from-scratch model
my_model = SimpleLinearRegression()
my_model.fit(X_train, y_train)

# 3. Train scikit-learn's model for comparison
sklearn_model = SklearnLinearRegression()
sklearn_model.fit(X_train, y_train.reshape(-1, 1)) # sklearn expects a 2D array for X
print(f"Scikit-learn: b0 = {sklearn_model.intercept_[0]:.4f}, b1 = {sklearn_model.coef_[0][0]:.4f}")

# 4. Make a prediction on a new data point
new_x = np.array([1.5])
my_prediction = my_model.predict(new_x)
sklearn_prediction = sklearn_model.predict(new_x.reshape(-1, 1))

print(f"\nPrediction for x = {new_x[0]}:")
print(f"Our model's prediction: {my_prediction[0]:.4f}")
print(f"Scikit-learn's prediction: {sklearn_prediction[0][0]:.4f}")

When you run this code, you'll see that the coefficients calculated by our SimpleLinearRegression class are virtually identical to those found by scikit-learn's highly optimized LinearRegression model. This confirms that our from-scratch implementation correctly applies the theory of Ordinary Least Squares.

Conclusion

Congratulations! You have successfully built a machine learning model from scratch. This is a significant milestone. You not only know how to use linear regression but also why it works, right down to its calculus-based origins.

Key Takeaways:

• Simple Linear Regression models a linear relationship between a single feature ($x$) and a target ($y$).
• The "best-fit" line is found by minimizing the Sum of Squared Errors (SSE). This method is called Ordinary Least Squares (OLS).
• Using calculus, we can derive direct formulas for the optimal slope ($b_1$) and intercept ($b_0$) without needing iterative methods.
• These formulas, based on the means, variance, and covariance of the data, can be implemented directly in Python to create a fully functional regression model.

Preview of the Next Lesson:
Our from-scratch model is a fantastic learning tool, but it's limited to a single input feature. Most real-world problems are more complex and involve multiple predictors. In our next lesson, we will extend linear regression to multiple features. We'll explore how the math generalizes and why using libraries like scikit-learn becomes essential for handling higher-dimensional data efficiently.