Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

2.3. Projections, Part 1

The Approximation Problem

Suppose u⃗{\color{orange}\vec u} and v⃗{\color{#3d81f6}\vec v} are any two vectors in Rn\mathbb{R}^n. (When I say this, I just mean that they both have the same number of components.)

Let’s think about all possible vectors of the form kv⃗\color{#004d40} k\vec v, where kk can be any scalar. Any vector of the form kv⃗\color{#004d40} k\vec v is a scalar multiple of v⃗{\color{#3d81f6}\vec v}, and points in the same direction as v⃗{\color{#3d81f6}\vec v} (if k>0k > 0) or the opposite direction (if k<0k < 0). What is different about these scalar multiples is how long they are.

To get a sense of what I mean by this, play with the slider for kk below. There are three vectors being visualized: some u⃗\color{orange}\vec u, some v⃗\color{#3d81f6}\vec v, and kv⃗\color{#004d40} k\vec v, which depends on the value of kk you choose.

Loading...
Loading...
Loading...
Loading...

Notice that the set of vectors of the form kv⃗\color{#004d40} k\vec v fill out a line. So really, what we’re asking is which vector on this line is closest to u⃗\color{orange}\vec u.

In terms of angles, if θ\theta is the angle between u⃗\color{orange}\vec u and v⃗\color{#3d81f6}\vec v, then the angle between u⃗\color{orange}\vec u and kv⃗\color{#004d40} k\vec v is either θ\theta (if k>0k > 0) or 180∘−θ180^{\circ} - \theta (if k<0k < 0). So changing kk doesn’t change how “similar” u⃗\color{orange}\vec u and kv⃗\color{#004d40} k\vec v are in the cosine similarity sense.

But, some choices of kk will make kv⃗\color{#004d40} k\vec v closer to u⃗\color{orange}\vec u than others. I call this the approximation problem: how well can we recreate, or approximate, u⃗\color{orange}\vec u using a scalar multiple of v⃗\color{#3d81f6}\vec v? It turns out that linear regression is intimately related to this problem. Previously, we were trying to approximate commute times as best as we could using a linear function of departure times.

Let’s be more precise by what we mean by “closer”. For any value of kk, we can measure the error of our approximation by the length of the error vector, e⃗=u⃗−kv⃗{\color{#d81b60} \vec e } = {\color{orange}\vec u} - \color{#004d40} k\vec v.

Loading...
Loading...

Continue to play with the slider above for kk. How do you get the length of the error vector to be as small as possible?

Intuitively, it seems that to get the error vector e⃗\color{#d81b60} \vec e to be as short as possible, we should make it orthogonal to v⃗\color{#3d81f6}\vec v. Since we can control kk, we can control u⃗−kv⃗{\color{orange}\vec u} - \color{#004d40} k\vec v, so we can make the error vector orthogonal to v⃗\color{#3d81f6}\vec v by choosing the right kk.

Orthogonal Projections

Our goal is to minimize the length of the error vector ∥e⃗∥\lVert {\color{#d81b60}\vec e} \rVert.

∥e⃗∥\lVert {\color{#d81b60}\vec e} \rVert

This is the same as minimizing

∥u⃗−kv⃗∥\lVert {\color{orange}\vec u} - {\color{#004d40} k\vec v} \rVert

One way to approach this problem is to treat the above expression as a function of kk and find the value of kk that minimizes it through calculus. You’ll do this in Homework 3. I’ll show you a more geometric approach here.

We’ve guessed, but not yet shown that the shortest possible error vector is the one that is orthogonal to v⃗\color{#3d81f6}\vec v. Let kok_o be the value of kk that makes the error vector e⃗o=u⃗−kov⃗{\color{#d81b60}\vec e_o} = {\color{orange}\vec u} - \color{#004d40} k_o\vec v orthogonal to v⃗\color{#3d81f6}\vec v. Here, we’ll prove that kok_o is the “best” choice of kk by showing that any other choice of kk will result in an error vector that is longer than e⃗o\color{#d81b60}\vec e_o. Think of this as a proof by contradiction (if you’re familiar with that idea; no worries if not).

For comparison, let k′k' be some other value of kk, and let its error vector be e⃗′=u⃗−k′v⃗\vec e' = {\color{orange}\vec u} - k' \color{#3d81f6}\vec v.

Loading...
Loading...
Image produced in Jupyter

I’ve drawn k′v⃗k' \vec v in gray. Arbitrarily, I’ve shown it as being shorter than kov⃗\color{#004d40} k_o \vec v, but I could have drawn it as being longer and the argument would be the same. The prime has nothing to do with derivatives, by the way – it’s just a new variable.

The vectors k′v⃗k' \vec v and kov⃗\color{#004d40} k_o \vec v, along with their corresponding error vectors, create a right-angled triangle, shaded in gold above. This triangle has a hypotenuse of e⃗′\vec e' and legs of e⃗o\color{#d81b60} \vec e_o and k′v⃗−kov⃗k' \vec v - \color{#004d40} k_o \vec v.

Applying the Pythagorean theorem to this triangle gives us

∥e⃗′∥2=∥e⃗o∥2+∥k′v⃗−kov⃗∥2⏟>0\lVert \vec e' \rVert^2 = \lVert {\color{#d81b60}\vec e_o} \rVert^2 + \underbrace{\lVert k' \vec v - {\color{#004d40} k_o \vec v} \rVert^2}_{> 0}

∥k′v⃗−kov⃗∥2≥0\lVert k' \vec v - {\color{#004d40} k_o \vec v} \rVert^2 \geq 0, with equality only when we choose k′=kok' = k_o, but we’ve assumed that k′≠kok' \neq k_o.

This implies that

∥e⃗′∥2=∥e⃗o∥2+some positive number\lVert \vec e' \rVert^2 = \lVert {\color{#d81b60}\vec e_o} \rVert^2 + \text{some positive number}

which means that ∥e⃗′∥2>∥e⃗o∥2\lVert \vec e' \rVert^2 > \lVert {\color{#d81b60}\vec e_o} \rVert^2.

In other words, if k′≠kok' \neq k_o, then ∥e⃗′∥>∥e⃗o∥\lVert \vec e' \rVert > \lVert {\color{#d81b60}\vec e_o} \rVert. Thus, the error vector e⃗o\color{#d81b60}\vec e_o is shorter than any other error vector e⃗′\vec e', and the best choice of kk is kok_o!

What was the point of all this again?

We know that the answer is k∗v⃗\color{#004d40} k^* \vec v, where k∗k^* is the value of kk that makes the error vector e⃗=u⃗−k∗v⃗{\color{#d81b60}\vec e} = {\color{orange}\vec u} - \color{#004d40} k^* \vec v orthogonal to v⃗\color{#3d81f6}\vec v. (I’ve switched from calling this optimal scalar kok_o to k∗k^*; kok_o was a name I used in the proof above, but more generally, “optimal” values are starred for our purposes).

Let’s now find the value of k∗k^*, in terms of just u⃗\color{orange} \vec u and v⃗\color{#3d81f6} \vec v. If e⃗\color{#d81b60}\vec e is orthogonal to v⃗\color{#3d81f6}\vec v, then v⃗⋅e⃗=0{\color{#3d81f6}\vec v} \cdot {\color{#d81b60}\vec e} = 0.

v⃗⋅e⃗=0v⃗⋅(u⃗−k∗v⃗)=0v⃗⋅u⃗−v⃗⋅k∗v⃗=0v⃗⋅u⃗−k∗(v⃗⋅v⃗)=0v⃗⋅u⃗=k∗(v⃗⋅v⃗)k∗=u⃗⋅v⃗v⃗⋅v⃗\begin{align*} {\color{#3d81f6}\vec v} \cdot {\color{#d81b60}\vec e} &= 0 \\ {\color{#3d81f6}\vec v} \cdot ({\color{orange}\vec u} - {\color{#004d40} k^* \vec v}) &= 0 \\ {\color{#3d81f6}\vec v} \cdot {\color{orange}\vec u} - {\color{#3d81f6}\vec v} \cdot {\color{#004d40} k^* \vec v} &= 0 \\ {\color{#3d81f6}\vec v} \cdot {\color{orange}\vec u} - {\color{#004d40} k^*} ({\color{#3d81f6}\vec v} \cdot {\color{#3d81f6} \vec v}) &= 0 \\ {\color{#3d81f6}\vec v} \cdot {\color{orange}\vec u} &= {\color{#004d40} k^*} ({\color{#3d81f6}\vec v} \cdot {\color{#3d81f6} \vec v}) \\ {\color{#004d40} k^*} &= \boxed{\frac{{\color{orange}\vec u} \cdot {\color{#3d81f6}\vec v}}{{\color{#3d81f6}\vec v} \cdot {\color{#3d81f6}\vec v}}} \end{align*}

The boxed value above is a scalar. It tells us the optimal amount to multiply v⃗\color{#3d81f6}\vec v by to get the best approximation of u⃗\color{orange}\vec u. Once we multiply that boxed scalar, k∗k^*, by the vector v⃗\color{#3d81f6}\vec v, we get what’s called the orthogonal projection of u⃗\color{orange}\vec u onto v⃗\color{#3d81f6}\vec v.

Among all vectors of the form kv⃗k\color{#3d81f6}\vec v, p⃗\vec p above is the one that is closest to u⃗\color{orange}\vec u.

Why “orthogonal projection”? “Orthogonal” comes from the fact that p⃗\vec p’s error vector e⃗=u⃗−p⃗{\color{#d81b60}\vec e} = {\color{orange}\vec u} - \vec p is orthogonal to v⃗\color{#3d81f6}\vec v. “Projection” comes from the intuition you should have that p⃗\vec p is the shadow of u⃗\color{orange}\vec u onto v⃗\color{#3d81f6}\vec v.

Loading...
Loading...
Image produced in Jupyter

We’ve defined the error vector as e⃗=u⃗−p⃗{\color{#d81b60}\vec e} = {\color{orange}\vec u} - \vec p, and we know that e⃗{\color{#d81b60}\vec e} is orthogonal to v⃗\color{#3d81f6}\vec v. Rearranging the definition of the error vector gives us

u⃗=p⃗⏟parallel to v⃗+e⃗⏟orthogonal to v⃗{\color{orange} \vec u} = \underbrace{\vec p}_\text{parallel to $\color{#3d81f6}\vec v$} + \underbrace{{\color{#d81b60}\vec e}}_\text{orthogonal to $\color{#3d81f6}\vec v$}

All this says is that u⃗\color{orange}\vec u is the sum of:

  • p⃗\color{#004d40}\vec p, which is parallel to v⃗\color{#3d81f6}\vec v (by definition of orthogonal projection)

  • e⃗\color{#d81b60}\vec e, which is orthogonal to v⃗\color{#3d81f6}\vec v (by definition of error vector)

Sometimes, we call this the orthogonal decomposition of u⃗\color{orange}\vec u with respect to v⃗\color{#3d81f6}\vec v. I’ll speak more about decompositions later in this section.

Examples

Let’s make things concrete by working through several examples. Each one was carefully chosen to illustrate something in particular.

We will work through several of these examples in lecture; attempt the ones that we don’t on your own.

Example: Fundamentals

Let u⃗=[122]\vec u = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} and v⃗=[111]\vec v = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}.

  1. Find the orthogonal projection of u⃗\vec u onto v⃗\vec v.

  2. Find the error vector, i.e. the vector e⃗=u⃗−p⃗\vec e = \vec u - \vec p, and verify that it is orthogonal to v⃗\vec v.

  3. What is the length of the error vector (i.e. the projection error)?

Solution

Part 1

The orthogonal projection of u⃗\vec u onto v⃗\vec v is given by

p⃗=u⃗⋅v⃗v⃗⋅v⃗v⃗\vec p = \frac{\vec u \cdot \vec v}{\vec v \cdot \vec v} \vec v

Following the same steps as in the previous example, we have

u⃗⋅v⃗=[122]⋅[111]=1+2+2=5\vec u \cdot \vec v = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = 1 + 2 + 2 = 5
v⃗⋅v⃗=[111]⋅[111]=1+1+1=3\vec v \cdot \vec v = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = 1 + 1 + 1 = 3

So, the orthogonal projection of u⃗\vec u onto v⃗\vec v is

p⃗=53[111]=[5/35/35/3]\vec p = \frac{5}{3} \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 5 / 3 \\ 5 / 3 \\ 5 / 3 \end{bmatrix}

Part 2

The error vector is

e⃗=u⃗−p⃗=[122]−[5/35/35/3]=[1−5/32−5/32−5/3]=[−2/31/31/3]\vec e = \vec u - \vec p = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} - \begin{bmatrix} 5 / 3 \\ 5 / 3 \\ 5 / 3 \end{bmatrix} = \begin{bmatrix} 1 - 5 / 3 \\ 2 - 5 / 3 \\ 2 - 5 / 3 \end{bmatrix} = \begin{bmatrix} -2 / 3 \\ 1 / 3 \\ 1 / 3 \end{bmatrix}

To check if it’s orthogonal to v⃗\vec v, we compute their dot product; we’re hoping it’s 0.

e⃗⋅v⃗=[−2/31/31/3]⋅[111]=−2/3+1/3+1/3=0\vec e \cdot \vec v = \begin{bmatrix} -2 / 3 \\ 1 / 3 \\ 1 / 3 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = -2 / 3 + 1 / 3 + 1 / 3 = 0

So, the error vector e⃗\vec e is orthogonal to v⃗\vec v.

Part 3

The length of the error vector is

∥e⃗∥=(−2/3)2+(1/3)2+(1/3)2=4/9+1/9+1/9=6/9=2/3\begin{align*} \lVert \vec e \rVert &= \sqrt{(-2 / 3)^2 + (1 / 3)^2 + (1 / 3)^2} \\ &= \sqrt{4 / 9 + 1 / 9 + 1 / 9} \\ &= \sqrt{6 / 9} = \sqrt{2 / 3} \end{align*}

We might say 2/3\sqrt{2 / 3} is the projection error. Another way of thinking of it is as the shortest distance from the point (1,2,2)(1, 2, 2) to the line that passes through (0,0,0)(0, 0, 0) and (1,1,1)(1, 1, 1).


Example: The Line Perspective

Consider the points p1=(1,2,2)p_1 = (1, 2, 2) and p2=(1,1,1)p_2 = (1, 1, 1) in R3\mathbb{R}^3.

What is the shortest distance between p1p_1 and the line that passes through p2p_2 and the origin, (0,0,0)(0, 0, 0)?

Solution

The answer is 2/3\sqrt{2 / 3}. This example didn’t require any addition math beyond the previous example; it just serves to remind you of the geometry of the situation. The set of all possible scalar multiples of v⃗\vec v fill out a line in R3\mathbb{R}^3, and that line passes through (0,0,0)(0, 0, 0) and p2=(1,1,1)p_2 = (1, 1, 1).

Why does that line pass through (0,0,0)(0, 0, 0)? Consider the vector 0v⃗0\vec v – it’s the zero vector!


Example: Which Order?

Let u⃗=[122]\vec u = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} and v⃗=[111]\vec v = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}.

In the first example, we found the orthogonal projection of u⃗\vec u onto v⃗\vec v.

Now, do the opposite: find the orthogonal projection of v⃗\vec v onto u⃗\vec u.

Solution

Now, we’re projecting v⃗\vec v onto u⃗\vec u, which means our answer is going to be a multiple of u⃗\vec u, not v⃗\vec v as in the first part.

The orthogonal projection of v⃗\vec v onto u⃗\vec u is given by

p⃗=v⃗⋅u⃗u⃗⋅u⃗u⃗\vec p = \frac{\vec v \cdot \vec u}{\vec u \cdot \vec u} \vec u

The formula for the scalar in front of u⃗\vec u is the same as in Part 1, but with all v⃗\vec v’s replaced by u⃗\vec u’s and vice versa. The numerator is the same, since u⃗⋅v⃗=v⃗⋅u⃗\vec u \cdot \vec v = \vec v \cdot \vec u. The denominator is different; just remember that the denominator is the squared norm of the vector you’re projecting onto.

v⃗⋅u⃗=[111]⋅[122]=1+2+2=5\vec v \cdot \vec u = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} = 1 + 2 + 2 = 5
u⃗⋅u⃗=[122]⋅[122]=1+4+4=9\vec u \cdot \vec u = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} = 1 + 4 + 4 = 9

So, the orthogonal projection of v⃗\vec v onto u⃗\vec u is

p⃗=v⃗⋅u⃗u⃗⋅u⃗u⃗=59[122]=[5/910/910/9]\vec p = \frac{\vec v \cdot \vec u}{\vec u \cdot \vec u} \vec u = \frac{5}{9} \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} = \begin{bmatrix} 5 / 9 \\ 10 / 9 \\ 10 / 9 \end{bmatrix}

Note that the corresponding error vector, e⃗=v⃗−p⃗\vec e = \vec v - \vec p, is orthogonal to u⃗\vec u (not v⃗\vec v), since u⃗\vec u is the vector we projected onto.


Example: Unit Vectors

Let u⃗=[4−12]\vec u = \begin{bmatrix} 4 \\ -1 \\ 2 \end{bmatrix}, v⃗=[26−3]\vec v = \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} and V⃗=[2/76/7−3/7]\vec V = \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix}.

  1. Find the orthogonal projection of u⃗\vec u onto v⃗\vec v.

  2. Find the orthogonal projection of u⃗\vec u onto V⃗\vec V.

  3. What do you notice about your answers to the above two parts?

Solution

Part 1

We know that the orthogonal projection of u⃗\vec u onto v⃗\vec v is given by

p⃗=u⃗⋅v⃗v⃗⋅v⃗v⃗\vec p = \frac{\vec u \cdot \vec v}{\vec v \cdot \vec v} \vec v

Let’s compute the relevant dot products.

u⃗⋅v⃗=[4−12]⋅[26−3]=4⋅2+(−1)⋅6+2⋅(−3)=8−6−6=−4\vec u \cdot \vec v = \begin{bmatrix} 4 \\ -1 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} = 4 \cdot 2 + (-1) \cdot 6 + 2 \cdot (-3) = 8 - 6 - 6 = -4
v⃗⋅v⃗=[26−3]⋅[26−3]=2⋅2+6⋅6+(−3)⋅(−3)=4+36+9=49\vec v \cdot \vec v = \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} \cdot \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} = 2 \cdot 2 + 6 \cdot 6 + (-3) \cdot (-3) = 4 + 36 + 9 = 49

So, the orthogonal projection of u⃗\vec u onto v⃗\vec v is

p⃗=−449[26−3]=[−8/49−24/4912/49]\vec p = \frac{-4}{49} \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} = \begin{bmatrix} -8 / 49 \\ -24 / 49 \\ 12 / 49 \end{bmatrix}

Part 2

Now, we need to find u⃗⋅V⃗V⃗⋅V⃗V⃗\frac{\vec u \cdot \vec V}{\vec V \cdot \vec V} \vec V.

Let’s compute the relevant dot products.

u⃗⋅V⃗=[4−12]⋅[2/76/7−3/7]=87−67−67=−47\vec u \cdot \vec V = \begin{bmatrix} 4 \\ -1 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix} = \frac{8}{7} - \frac{6}{7} - \frac{6}{7} = -\frac{4}{7}
V⃗⋅V⃗=[2/76/7−3/7]⋅[2/76/7−3/7]=449+3649+949=4949=1\vec V \cdot \vec V = \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix} \cdot \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix} = \frac{4}{49} + \frac{36}{49} + \frac{9}{49} = \frac{49}{49} = 1

So, the orthogonal projection of u⃗\vec u onto V⃗\vec V is

p⃗=−471[2/76/7−3/7]=[−8/49−24/4912/49]\vec p = \frac{-\frac{4}{7}}{1} \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix} = \begin{bmatrix} -8 / 49 \\ -24 / 49 \\ 12 / 49 \end{bmatrix}

Part 3

Notice that in both parts, the orthogonal projection p⃗\vec p is the same! This is not a coincidence. Both vectors point in the same direction, meaning the set of possible vectors of the form kV⃗k \vec V is the same as the set of possible vectors of the form kv⃗k \vec v. Another way to think about this is that they both span the same line in R3\mathbb{R}^3 through the origin.

The difference between v⃗\vec v and V⃗\vec V is that V⃗\vec V is a unit vector in the direction of v⃗\vec v, meaning that it points in the same direction as v⃗\vec v but has ∥V⃗∥=1\lVert \vec V \rVert = 1 rather than ∥v⃗∥=7\lVert \vec v \rVert = 7.

What’s different is the scalar we need to multiply each vector by to get the orthogonal projection. In the case of the unit vector V⃗\vec V, the number in front of V⃗\vec V is u⃗⋅V⃗V⃗⋅V⃗\frac{\vec u \cdot \vec V}{\vec V \cdot \vec V}, but since V⃗⋅V⃗=∥V⃗∥2=1\vec V \cdot \vec V = \lVert \vec V \rVert^2 = 1, this simplifies to u⃗⋅V⃗\vec u \cdot \vec V.


Example: Unit Vectors, Continued

Suppose u⃗,v⃗∈Rn\vec u, \vec v \in \mathbb{R}^n. Let θ\theta be the angle between u⃗\vec u and v⃗\vec v.

Show that the orthogonal projection of u⃗\vec u onto v⃗\vec v is equal to

p⃗=(∥u⃗∥cos⁡θ)v⃗∥v⃗∥\vec p = \left( \lVert \vec u \rVert \cos \theta \right) \frac{\vec v}{\lVert \vec v \rVert}

This is not a formula we’d use to actually compute p⃗\vec p, since finding cos⁡θ\cos \theta is harder than using the dot product-based formula from above. But, what does this formula tell us about the relationship between u⃗\vec u and p⃗\vec p?

Solution

Let’s start with the original formula for the orthogonal projection of u⃗\vec u onto v⃗\vec v.

p⃗=u⃗⋅v⃗v⃗⋅v⃗v⃗\vec p = \frac{\vec u \cdot \vec v}{\vec v \cdot \vec v} \vec v

Using the fact that u⃗⋅v⃗=∥u⃗∥∥v⃗∥cos⁡θ\vec u \cdot \vec v = \lVert \vec u \rVert \lVert \vec v \rVert \cos \theta and v⃗⋅v⃗=∥v⃗∥2\vec v \cdot \vec v = \lVert \vec v \rVert^2, we can rewrite the formula as

p⃗=∥u⃗∥∥v⃗∥cos⁡θ∥v⃗∥2v⃗=∥u⃗∥cos⁡θ∥v⃗∥v⃗=(∥u⃗∥cos⁡θ)v⃗∥v⃗∥\vec p = \frac{\lVert \vec u \rVert \lVert \vec v \rVert \cos \theta}{\lVert \vec v \rVert^2} \vec v = \frac{\lVert \vec u \rVert \cos \theta}{\lVert \vec v \rVert} \vec v = (\lVert \vec u \rVert \cos \theta) \frac{\vec v}{\lVert \vec v \rVert}

The parentheses around ∥u⃗∥cos⁡θ\lVert \vec u \rVert \cos \theta don’t change the calculation, but they help with the interpretation. This shows us that we can think of the orthogonal projection of u⃗\vec u onto v⃗\vec v as a vector with:

  • a length of ∥u⃗∥cos⁡θ\lVert \vec u \rVert \cos \theta

  • in the direction of v⃗∥v⃗∥\frac{\vec v}{\lVert \vec v \rVert}, which is a unit vector in the direction of v⃗\vec v


Example: Projecting onto an Orthogonal Vector

Let u⃗=[4−12]\vec u = \begin{bmatrix} 4 \\ -1 \\ 2 \end{bmatrix} and v⃗=[−306]\vec v = \begin{bmatrix} -3 \\ 0 \\ 6 \end{bmatrix}.

u⃗\vec u and v⃗\vec v are orthogonal. What does this say about the orthogonal projection of u⃗\vec u onto v⃗\vec v?

Solution

Since u⃗⋅v⃗=0\vec u \cdot \vec v = 0, the orthogonal projection of u⃗\vec u onto v⃗\vec v is the zero vector, 0⃗\vec 0.

p=u⃗⋅v⃗v⃗⋅v⃗v⃗=036v⃗=0⃗p = \frac{\vec u \cdot \vec v}{\vec v \cdot \vec v} \vec v = \frac{0}{36} \vec v = \vec 0

Intuitively, u⃗\vec u and v⃗\vec v travel in totally different different directions. Travelling any amount of v⃗\vec v will take you further away from u⃗\vec u. So, it’s best to stick with the zero vector, 0⃗\vec 0.

Orthogonal Decomposition

To motivate the next section, let’s consider another example.

Let u⃗=[35]\color{orange} \vec u = \begin{bmatrix} 3 \\ 5 \end{bmatrix}, v⃗1=[6−2]\color{#3d81f6} \vec v_1 = \begin{bmatrix} 6 \\ -2 \end{bmatrix}, and v⃗2=[13]\color{#3d81f6} \vec v_2 = \begin{bmatrix} 1 \\ 3 \end{bmatrix}.

Image produced in Jupyter

Notice that v⃗1\color{#3d81f6}\vec v_1 and v⃗2\color{#3d81f6}\vec v_2 are orthogonal – that’s important to what we’re about to discover.

Let’s find the orthogonal projection of u⃗\color{orange}\vec u onto v⃗1\color{#3d81f6}\vec v_1 (called p⃗1\color{#004d40}\vec p_1) and the orthogonal projection of u⃗\color{orange}\vec u onto v⃗2\color{#3d81f6}\vec v_2 (called p⃗2\color{#004d40}\vec p_2).

p⃗1=u⃗⋅v⃗1v⃗1⋅v⃗1v⃗1=840v⃗1=15v⃗1=[6/5−2/5]{\color{#004d40}\vec p_1} = \frac{{\color{orange}\vec u} \cdot \color{#3d81f6}\vec v_1}{\color{#3d81f6}\vec v_1 \cdot \color{#3d81f6}\vec v_1} {\color{#3d81f6}\vec v_1} = \frac{8}{40} {\color{#3d81f6}\vec v_1} = \frac{1}{5} {\color{#3d81f6}\vec v_1} = {\color{#004d40}\begin{bmatrix} 6 / 5 \\ -2 / 5 \end{bmatrix}}
p⃗2=u⃗⋅v⃗2v⃗2⋅v⃗2v⃗2=1810v⃗2=95v⃗2=[9/527/5]{\color{#004d40}\vec p_2} = \frac{{\color{orange}\vec u} \cdot \color{#3d81f6}\vec v_2}{\color{#3d81f6}\vec v_2 \cdot \color{#3d81f6}\vec v_2} {\color{#3d81f6}\vec v_2} = \frac{18}{10} {\color{#3d81f6}\vec v_2} = \frac{9}{5} {\color{#3d81f6} \vec v_2} = {\color{#004d40}\begin{bmatrix} 9 / 5 \\ 27 / 5 \end{bmatrix}}

Notice that u⃗\color{orange}\vec u is the sum of p⃗1\color{#004d40}\vec p_1 and p⃗2\color{#004d40}\vec p_2!

u⃗=p⃗1+p⃗2=15v⃗1+95v⃗2⏟linear combination=[6/5−2/5]+[9/527/5]=[35]{\color{orange}\vec u} = {\color{#004d40}\vec p_1} + {\color{#004d40}\vec p_2} = \underbrace{\frac{1}{5} {\color{#3d81f6}\vec v_1} + \frac{9}{5} {\color{#3d81f6}\vec v_2}}_{\text{linear combination}} = \begin{bmatrix} 6 / 5 \\ -2 / 5 \end{bmatrix} + \begin{bmatrix} 9 / 5 \\ 27 / 5 \end{bmatrix} = \begin{bmatrix} 3 \\ 5 \end{bmatrix}
Image produced in Jupyter

Why is the sum of p⃗1\color{#004d40}\vec p_1 and p⃗2\color{#004d40}\vec p_2 equal to u⃗\color{orange}\vec u? Earlier, I mentioned that we can use orthogonal projections to decompose vectors. Here, when we project u⃗\color{orange}\vec u onto v⃗1\color{#3d81f6}\vec v_1, the corresponding error vector e⃗1\color{#d81b60}\vec e_1 is orthogonal to v⃗1\color{#3d81f6}\vec v_1.

u⃗=p⃗1⏟parallel to v⃗1+e⃗1⏟orthogonal to v⃗1{\color{orange} \vec u} = \underbrace{{\color{#004d40}\vec p_1}}_{\text{parallel to } \color{#3d81f6}\vec v_1} + \underbrace{{\color{#d81b60}\vec e_1}}_{\text{orthogonal to } \color{#3d81f6}\vec v_1}

By projecting u⃗\color{orange}\vec u onto v⃗2\color{#3d81f6}\vec v_2, we can recreate the error vector exactly, meaning e⃗1=p⃗2{\color{#d81b60}\vec e_1} = \color{#004d40}\vec p_2.

Taking a step back, the fact that v1\color{#3d81f6} v_1 and v2\color{#3d81f6} v_2 are orthogonal meant that writing u⃗\color{orange}\vec u as a linear combination of v1\color{#3d81f6} v_1 and v2\color{#3d81f6} v_2 was easy.

a1[6−2]⏟v⃗1+a2[13]⏟v⃗2=[35]⏟u⃗a_1 {\color{#3d81f6} \underbrace{\begin{bmatrix} 6 \\ -2 \end{bmatrix}}_{\vec v_1}} + a_2 {\color{#3d81f6} \underbrace{\begin{bmatrix} 1 \\ 3 \end{bmatrix}}_{\vec v_2}} = \color{orange} \underbrace{\begin{bmatrix} 3 \\ 5 \end{bmatrix}}_{\vec u}

If v1\color{#3d81f6} v_1 and v2\color{#3d81f6} v_2 were not orthogonal, then writing u⃗\color{orange}\vec u as a linear combination of v1\color{#3d81f6} v_1 and v2\color{#3d81f6} v_2 would have involved solving a system of 2 equations and 2 unknowns, as we’ve had to do in previous sections.

For instance, if we keep [35]\color{orange} \begin{bmatrix} 3 \\ 5 \end{bmatrix} and look at x⃗1=[12]\color{#3d81f6} \vec x_1 = \begin{bmatrix} 1 \\ 2 \end{bmatrix} and x⃗2=[21]\color{#3d81f6} \vec x_2 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}, we have that

  • the projection of u⃗\color{orange}\vec u onto x⃗1\color{#3d81f6} \vec x_1 is (u⃗⋅x⃗1x⃗1⋅x⃗1)x⃗1=135x⃗1 \left( \frac{{\color{orange}\vec u} \cdot \color{#3d81f6} \vec x_1}{{\color{#3d81f6} \vec x_1} \cdot \color{#3d81f6} \vec x_1} \right) {\color{#3d81f6} \vec x_1} = \frac{13}{5} \color{#3d81f6} \vec x_1

  • the projection of u⃗\color{orange}\vec u onto x⃗2\color{#3d81f6} \vec x_2 is (u⃗⋅x⃗2x⃗2⋅x⃗2)x⃗2=115x⃗2 \left( \frac{{\color{orange}\vec u} \cdot \color{#3d81f6} \vec x_2}{{\color{#3d81f6} \vec x_2} \cdot \color{#3d81f6} \vec x_2} \right) {\color{#3d81f6} \vec x_2} = \frac{11}{5} \color{#3d81f6} \vec x_2

but

135x⃗1+115x⃗2=135[12]+115[21]=[13/5+22/526/5+11/5]=[77.4]≠u⃗\frac{13}{5} {\color{#3d81f6} \vec x_1} + \frac{11}{5} {\color{#3d81f6} \vec x_2} = \frac{13}{5} \begin{bmatrix} 1 \\ 2 \end{bmatrix} + \frac{11}{5} \begin{bmatrix} 2 \\ 1 \end{bmatrix} = \begin{bmatrix} 13 / 5 + 22 / 5 \\ 26 / 5 + 11 / 5 \end{bmatrix} = \begin{bmatrix} 7 \\ 7.4 \end{bmatrix} \neq {\color{orange}\vec u}

I’d like to provide a more general “theorem”, on when you can use orthogonal projections to more easily write a vector u⃗\color{orange}\vec u as a linear combination of vectors v⃗1\color{#3d81f6} \vec v_1, v⃗2\color{#3d81f6} \vec v_2, ..., v⃗d\color{#3d81f6} \vec v_d, but we’ll need to first study the idea of a basis. That’s to come.

This section has been light on activities, since it provided many examples that we’ll work through in lecture. But, here’s one to tie this last point together.

What’s Next?

The motivating problem for this section was the approximation problem, which asked us to find the best approximation of a vector u⃗\color{orange}\vec u using only a scalar multiple of a vector v⃗\color{#3d81f6}\vec v.

The next natural step is to consider the case where we want to approximate u⃗\color{orange}\vec u using a linear combination of more than one vector, v⃗1\color{#3d81f6} \vec v_1, v⃗2\color{#3d81f6} \vec v_2, ..., v⃗d\color{#3d81f6} \vec v_d. Why? Remember, this all connects back to the problem of linear regression. The more vectors we have as “building blocks” in our linear combination, the more features our model will be able to use. (I haven’t made the connection from linear algebra to linear regression yet, but just know this is why we’re studying projections.)

For example, let’s consider u⃗=[136]\color{orange}\vec u = \begin{bmatrix} 1 \\ 3 \\ 6 \end{bmatrix}, v⃗1=[101]\color{#3d81f6} \vec v_1 = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}, and v⃗2=[221]\color{#3d81f6} \vec v_2 = \begin{bmatrix} 2 \\ 2 \\ 1 \end{bmatrix}. Among all linear combinations of v⃗1\color{#3d81f6} \vec v_1 and v⃗2\color{#3d81f6} \vec v_2, which one is closest to u⃗\color{orange}\vec u?

To answer this question, we’d need to find the scalars a1a_1 and a2a_2 such that the error vector

e⃗=u⃗−(a1v⃗1+a2v⃗2){\color{#d81b60}\vec e} = {\color{orange}\vec u} - {(a_1 {\color{#3d81f6} \vec v_1} + a_2 {\color{#3d81f6} \vec v_2})}

has minimal length, which presumably happens when e⃗\color{#d81b60}\vec e is orthogonal to both v⃗1\color{#3d81f6} \vec v_1 and v⃗2\color{#3d81f6} \vec v_2.

Travelling down this road, we might be able to find the values of a1a_1 and a2a_2 that minimize the length of e⃗\color{#d81b60}\vec e. But then we’ll want to ask how we can do this for any u⃗\color{orange}\vec u and any set of vectors v⃗1\color{#3d81f6} \vec v_1, v⃗2\color{#3d81f6} \vec v_2, ..., v⃗d\color{#3d81f6} \vec v_d, and it seems like we’ll need a more general solution. In general, to find the “best” approximation of u⃗\color{orange}\vec u using a linear combination of v⃗1\color{#3d81f6} \vec v_1, v⃗2\color{#3d81f6} \vec v_2, ..., v⃗d\color{#3d81f6} \vec v_d, we’ll need to know about matrices. We’ll introduce matrices in Chapter 2.5.

Instead, in Chapter 2.4, we will set aside the goal of projections temporarily, and instead focus on truly understanding the set of possible linear combinations of a given set of vectors. For example, the vectors v⃗1=[100]\color{#3d81f6} \vec v_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, v⃗2=[−518]\color{#3d81f6} \vec v_2 = \begin{bmatrix} -5 \\ 1 \\ 8 \end{bmatrix} from earlier define a plane. So, asking which linear combination of v⃗1\color{#3d81f6} \vec v_1 and v⃗2\color{#3d81f6} \vec v_2 is closest to u⃗\color{orange}\vec u is equivalent to asking which point on the plane is closest to u⃗\color{orange}\vec u.

Loading...

Chapter 2.4 will answer the questions, “why do v⃗1\color{#3d81f6} \vec v_1 and v⃗2\color{#3d81f6} \vec v_2 define a plane?”, “which plane do they define?”, and “in general, what do v⃗1\color{#3d81f6} \vec v_1, v⃗2\color{#3d81f6} \vec v_2, ..., v⃗d\color{#3d81f6} \vec v_d, all in Rn\mathbb{R}^n, define?”

EECS 245 Course Notes
2.2. The Dot Product
EECS 245 Course Notes
2.4. Span and Linear Independence