1.2. Loss Functions and the Constant Model

This is a real dataset, collected by Joseph Hearn, except he lived in Seattle, not Metro Detroit. The full dataset contains more columns than are shown above, but we’ll focus on these few for now.

Our goal is to predict commute times, stored in the minutes column. This is our $y$ variable. The natural first input variable, or feature, to consider, is departure_hour. This is our $x$ variable.

We’ll use the subscript $i$ to index the $i$th data point, for $i = 1, 2, \ldots, n$. Using the dataset above, $x_1 = 10.816667$ and $y_1 = 68$, for instance.

Departure hours are stored as decimals, but correspond to times of the day. For example, 7.75 corresponds to 7:45 AM, and 10.816667 corresponds to 10:49 AM.

Before we get any further, we should look at our data. Since we’re working with two quantitative variables, we should draw a scatter plot.

There’s a general downward trend: the later the departure time, the lower the commute time tends to be.

Again, our goal is to predict commute time given departure hour. That is, we’d like to build a useful function $h$ such that:

This is a regression problem, because the variable we are predicting – commute time – is quantitative.

To build this function, the approach we’ll take is the machine learning approach – that is, to learn a pattern from the dataset that we’ve collected. (This is not the only approach one could take – we could build the function $h$ however we want.)

However, in order to learn a pattern from the dataset that we’ve collected, we need to make an important assumption.

We don’t really need our function $h$ to make good predictions on the dataset that we’ve already collected. We know the actual commute times on day 1, day 2, ..., day $n$. In other words, we’re working with a labeled dataset, in which we’re given the values of $y_1, y_2, \ldots, y_n$.

What we do need is for our function $h$ to make good predictions on unseen data from the future, i.e. for future commutes, the ones we don’t know about yet. This is the only way our function $h$ will be practically useful.

But, if the future doesn’t resemble the past, the patterns we learn from the past will not be generalizable to the future. For example, if a new highway between Detroit and Ann Arbor gets built, the patterns previously learned won’t necessarily still exist. This idea of generalizability is key, so keep it in mind even if we’re not explicitly talking about it.

Models¶

I’ve used the word “model” loosely, but let’s give it a formal definition.

“All models are wrong, but some are useful.” - George Box

My interpretation of George Box’s famous quote is that no matter how complex a model is, it will never be 100% correct, so sometimes – especially when we’re starting our machine learning journey – it’s better to use a simpler model that is also wrong but interpretable.

We gain value from simple models all the time. In a physics class, you may have learned that acceleration due to gravity is $9.81 \text{ m/s}^2$ towards the center of the Earth. This is not fully accurate – think about how parachutes work, for example – but it’s still a useful approximation, and one that is relatively easy to understand. A related idea is Occam’s razor, which states that the simplest explanation of a phenomenon is often the best.

Above, you’ll see a degree-40 polynomial fit to our dataset. We’ll learn how to build such polynomials throughout the semester.

At first glance, it looks to be quite accurate, albeit complex. In fact, it’s a little too complex, and the phenomenon we see above is called overfitting. For $x_i$’s in the dataset that we collected, sure, the polynomial will make accurate predictions, but for any $x_i$’s that don’t match the exact pattern in the dataset, the predictions will be off. (For example, it’s unlikely that commutes will take 110 minutes around 10:15AM, but that’s what the model predicts.) This polynomial model wouldn’t generalize well to unseen data.

If we look at the scatter plot closely, it seems reasonable to start with a line of best fit, much like you may have seen in a statistics class. In fact, we’ll start with something even more simple than that. But first, some notation.

Hypothesis Functions¶

The hypothesis functions we’ll study have parameters, usually denoted by $w$, which describe the relationship between the input and output. The two hypothesis functions we’ll study are:

1. Constant model: $h(x_i) = w$
2. Simple linear regression model: $h(x_i) = w_0 + w_1 x_i$

We’ll study the constant model first, but it’s easier to understand the role of parameters in the simple linear regression model.

$h(x_i) = w_0 + w_1 x_i$ represents the equation of a line, where $w_0$ is the intercept and $w_1$ is the slope. Above, we see that different choices of parameters $w_0$ and $w_1$ result in different lines. The million dollar question is: among all of the infinitely many choices of $w_0$ and $w_1$, which one is the best?

To fully answer that, we’ll have to wait until Chapter 1.4. Surprisingly, that answer involves multivariable calculus.

For now, let’s return to the constant model, $h(x_i) = w$. The constant model predicts the same value for all $x_i$’s, and looks like a flat line.

We’ll use the constant model for the rest of this section to illustrate core ideas in machine learning, and will move to more sophisticated models in Chapter 1.4.

If we’re forced to use a constant model, it’s clear that some choices of $w$ (the height of the line) are better than others. $w=100$ yields a flat line that is far from most of the data. $w=60$ and $w=70$ seem like much more reasonable predictions, but how we can quantify which one is better, and which $w$ would be the best?

Since the constant model doesn’t depend on departure hours $x_i$, we can instead draw a histogram of just the true commute times.

An equivalent way of phrasing the problem is, which constant $w$ best summarizes the histogram above? Most commute times seem to be in the 60 to 80 range, so somewhere there makes sense. How can we be more precise?

Loss Functions¶

To illustrate, let’s consider a small dataset of only 5 commute times.

If asked to find the constant that best summarizes these 5 numbers, you might think of the mean or median, which are common summary statistics. There are other valid choices too, like the mode, or halfway between the minimum and maximum, or the most recent. What we need is a way to compare these choices.

A loss function quantifies how bad a prediction is for a single data point.

• If our prediction is close ✅ to the actual value, we should have low loss.
• If our prediction is far ❌ from the actual value, we should have high loss.

We’ll start by computing the error for a single data point, defined as the difference between an actual $y$-value and its corresponding predicted $y$-value.

where ${\color{3D81F6}y_i}$ is the actual value and ${\color{orange}h(x_i)}$ is the predicted value.

Could this be a loss function? Let’s think this through. Suppose we have the true commute time $y_i=80$.

• If I predict 75, $e_i=80-\textcolor{orange}{75}=5$.
• If I predict 72, $e_i=80-\textcolor{orange}{72}=8$.
• If I predict 100, $e_i=80-\textcolor{orange}{100}=-20$.

A lower error is better, so 75 (error of 5) is a better prediction than 72 (error of 8). 100 seems to be the worst of the three predictions, but technically has the smallest error (-20). The issue is that some errors are positive and some are negative, and so it’s hard to compare them directly.

So ideally, a loss function shouldn’t have negative outputs. How can we take these errors, in which some are positive and some are negative, and enforce that they’re all positive?

Squared Loss¶

The most common solution to the problem of negative errors is to square each error. This gives rise to the first loss function we’ll explore, and arguably the most common loss function in machine learning: squared loss.

The squared loss function, $L_\text{sq}$, computes $({\color{3D81F6}\text{actual}}-{\color{orange}\text{predicted}})^2$. That is:

Why did we square instead of take the absolute value? Absolute loss is a perfectly valid loss function – in fact, we’ll study it in Chapter 1.3 – and different loss functions have different pros and cons. That said, squared loss is a good first choice because:

• The resulting optimization problem is differentiable, as we’ll see in just a few moments.
• It has a nice relationship to the normal distribution in statistics, as we’ll see in Chapter 6, at the end of the course.

Let’s return to our small example dataset of 5 commute times.

How can we use squared loss to compare two choices of $w$, say $w=85$ (the median) and $w=80$ (the mean)? Let’s draw a picture (in which the $x$-axis positions of each point are irrelevant, since we’re not using departure hours).

Each output of $L_\text{sq}$, shown in pink, describes the quality of a prediction for a single data point. For example, in the left plot above, the annotated $(-13)^2$ came from an actual value of 72 and a predicted value of 85:

What we’d like is a single number which describes the quality of our predictions across the whole dataset, almost like a “score” for each choice of $w$. Then, we can compare scores to choose the best possible $w$. One way to construct such a score is to take the average of the squared losses.

• For the median, $w = 85$:

  $$\begin{aligned} \ &\frac{1}{5} \left( (72 - {\color{orange} 85})^2 + (90 - {\color{orange} 85})^2 + (61 - {\color{orange} 85})^2 + (85 - {\color{orange} 85})^2 + (92 - {\color{orange} 85})^2 \right) \\ &= 163.8 \end{aligned}$$

• For the mean, $w = 80$:

  $$\begin{aligned} \ &\frac{1}{5} \left( (72 - {\color{orange} 80})^2 + (90 - {\color{orange} 80})^2 + (61 - {\color{orange} 80})^2 + (85 - {\color{orange} 80})^2 + (92 - {\color{orange} 80})^2 \right) \\ &= 138.8 \end{aligned}$$

Losses are bad, so the better choice of $w$ has a lower average squared loss. Since $138.8 < 163.8$, the mean is a better prediction than the median.

Another term for average squared loss is mean squared error (MSE); this is the more common name for the technique we just defined.

Minimizing Mean Squared Error¶

Let’s start by generalizing mean squared error to any prediction $w$ for our small commute times dataset.

The function $R_\text{sq}$ takes in any prediction $w$ and outputs the mean squared error of that $w$. We’re searching for the value of $w$ that makes $R_\text{sq}(w)$ as small as possible, as that would correspond to the $w$ that makes the best possible predictions, for our humble constant model.

Where did the letter $R$ come from? It stands for risk, as in “empirical risk”. I’ll speak more on this soon. For now, remember that:

• $L$ always refers to loss for a single data point.
• $R$ always refers to average loss across an entire dataset.

What does $R_\text{sq}(w)$ actually look like, if we were to plot it? It is the sum of 5 quadratic functions – namely, $\frac{1}{5}(72 - w)^2$, $\frac{1}{5}(90 - w)^2$, and so on – and so it’s a quadratic function too, and looks like a parabola.

The question is, though, what is the $w$-value of the vertex of this parabola? That is, which $w$ minimizes $R_\text{sq}(w)$?

Before we find the answer, let’s cast our problem in more general terms, so that the answer is applicable to any dataset. Suppose we have a dataset of $n$ actual commute times, $y_1, y_2, \ldots, y_n$. Our goal is to find the $w$ that minimizes:

While it looks like there are many variables in this equation, we know the actual values in the dataset, so we can treat $y_1, y_2, \ldots, y_n$ as constants. The only true variable is $w$.

How do we minimize $R_\text{sq}(w)$? There are a few approaches. We’ll use a calculus-based approach here, though in Homework 1 you’ll look at an alternative approach. For a refresher on the relevant calculus ideas, see Chapter 0.2.

$R_\text{sq}(w)$ is a function of a single variable, $w$. To minimize a function of a single variable, we should:

1. Take the derivative of $R_\text{sq}(w)$ with respect to $w$.
2. Set the derivative equal to 0 and solve for $w$.
3. Verify that the second derivative at the critical point is positive.

Let’s go through these steps one by one.

Step 1: Take the derivative of $R_\text{sq}(w)$ with respect to $w$

Remember that constants can be pulled out of derivatives, e.g. the derivative of $2 f(x)$ is 2 times the derivative of $f(x)$.

From here, we’ll use the fact that the derivative of a sum is the sum of derivatives, to “push” the derivative operator inside the sum.

What is $\frac{\text{d}}{\text{d}w} (y_i - w)^2$? Try and work it out on your own, then check the solution below.

Using that result, we have:

Finally, we’ll pull the constant of -2 out of the sum.

We could simplify this further, but this form will do just fine. Don’t forget, though, that the expression on the right side is a function of $w$.

Step 2: Set the derivative equal to 0 and solve for $w$

First, we’ll multiply both sides by $-\frac{n}{2}$ to get rid of the fraction.

Separating the sum into two parts gives us:

$\displaystyle \sum_{i = 1}^n y_i$ can’t be broken down much further. But, $\displaystyle \sum_{i = 1}^n w$ is the sum of $n$ copies of $w$, i.e. $w + w + \ldots + w$. This is just $nw$!

Adding $nw$ to both sides, then dividing both sides by $n$, gives us:

The value of $w$ that minimizes $R_\text{sq}(w)$ is $w^* = \frac{1}{n} \sum_{i = 1}^n y_i$. Notice that I’ve called it $w^*$; think of “star” as meaning “best” or “optimal”.

The formula for $w^*$ should look very familar. It’s the mean of $y_1, y_2, \ldots, y_n$!

Step 3: Verify that the second derivative at the critical point is positive

We already know that $R_\text{sq}(w)$ is a parabola, which means that its only critical point is a global minimum. But, we’ll be thorough just to set a good example.

Here, we’ll need to find the second derivative of $R_\text{sq}(w)$ with respect to $w$.

The second derivative is 2 for all values of $w$, including at the $w^*$ we found. This tells us that $R_\text{sq}(w)$ is concave opening upwards across its entire domain, so the critical point we’ve found corresponds to a global minimum.

Conclusion¶

What was the point of all of that algebra? To recap:

• We decided to use the constant model, $h(x_i) = w$, to make predictions.
• To find the best value of $w$ – a model parameter • we decided to minimize mean squared error:
  $$R_\text{sq}(w) = \frac{1}{n} \sum_{i = 1}^n (y_i - w)^2$$
• Using calculus, we found that the value of $w$ that minimizes $R_\text{sq}(w)$ is
  $$w^* = \frac{1}{n} \sum_{i = 1}^n y_i = \text{Mean}(y_1, y_2, \ldots, y_n)$$

In other words, the mean minimizes mean squared error. This is a remarkable result. We use the mean all of the time in daily life, and now we’ve proven that it is optimal in some sense. It is the constant with the smallest mean squared error, no matter the dataset we’re working with.

Another name for $w^*$ is an optimal model parameter. In the context of our full commute times dataset, the optimal model parameter is the mean commute time. Visually, the value of $w^* \approx 73$ tells us the optimal “height” at which we should draw the constant model, $h(x_i) = w$.

Is this the best possible model? No, of course not – we’re not capturing the fact that later departure times are associated with shorter commute times. But as a first attempt at building a model, the constant model is valuable. If someone asked you how long your commutes are, saying something like “about 73 minutes” is reasonable.

What’s next?

• In Chapter 1.3, we’ll investigate other loss functions, like absolute loss.
• In Chapter 1.4, we’ll reintroduce the simple linear regression model, $h(x_i) = w_0 + w_1 x_i$, and see how to find the best values of $w_0$ and $w_1$.