Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

3.4. Projecting onto a Single Vector

The Approximation ProblemΒΆ

Suppose u⃗{\color{orange}\vec u} and v⃗{\color{#3d81f6}\vec v} are any two vectors in Rn\mathbb{R}^n. (When I say this, I just mean that they both have the same number of components.)

Let’s think about all possible vectors of the form kvβƒ—\color{#004d40} k\vec v, where kk can be any scalar. Any vector of the form kvβƒ—\color{#004d40} k\vec v is a scalar multiple of vβƒ—{\color{#3d81f6}\vec v}, and points in the same direction as vβƒ—{\color{#3d81f6}\vec v} (if k>0k > 0) or the opposite direction (if k<0k < 0). What is different about these scalar multiples is how long they are.

To get a sense of what I mean by this, play with the slider for kk below. There are three vectors being visualized: some u⃗\color{orange}\vec u, some v⃗\color{#3d81f6}\vec v, and kv⃗\color{#004d40} k\vec v, which depends on the value of kk you choose.

Loading...
Loading...
Loading...
Loading...

Notice that the set of vectors of the form kvβƒ—\color{#004d40} k\vec v fill out a line. So really, what we’re asking is which vector on this line is closest to uβƒ—\color{orange}\vec u.

In terms of angles, if ΞΈ\theta is the angle between uβƒ—\color{orange}\vec u and vβƒ—\color{#3d81f6}\vec v, then the angle between uβƒ—\color{orange}\vec u and kvβƒ—\color{#004d40} k\vec v is either ΞΈ\theta (if k>0k > 0) or 180βˆ˜βˆ’ΞΈ180^{\circ} - \theta (if k<0k < 0). So changing kk doesn’t change how β€œsimilar” uβƒ—\color{orange}\vec u and kvβƒ—\color{#004d40} k\vec v are in the cosine similarity sense.

But, some choices of kk will make kv⃗\color{#004d40} k\vec v closer to u⃗\color{orange}\vec u than others. I call this the approximation problem: how well can we recreate, or approximate, u⃗\color{orange}\vec u using a scalar multiple of v⃗\color{#3d81f6}\vec v? It turns out that linear regression is intimately related to this problem. Previously, we were trying to approximate commute times as best as we could using a linear function of departure times.

Let’s be more precise by what we mean by β€œcloser”. For any value of kk, we can measure the error of our approximation by the length of the error vector, eβƒ—=uβƒ—βˆ’kvβƒ—{\color{#d81b60} \vec e } = {\color{orange}\vec u} - \color{#004d40} k\vec v.

Loading...
Loading...

Continue to play with the slider above for kk. How do you get the length of the error vector to be as small as possible?

Intuitively, it seems that to get the error vector eβƒ—\color{#d81b60} \vec e to be as short as possible, we should make it orthogonal to vβƒ—\color{#3d81f6}\vec v. Since we can control kk, we can control uβƒ—βˆ’kvβƒ—{\color{orange}\vec u} - \color{#004d40} k\vec v, so we can make the error vector orthogonal to vβƒ—\color{#3d81f6}\vec v by choosing the right kk.

Orthogonal ProjectionsΒΆ

Our goal is to minimize the length of the error vector βˆ₯eβƒ—βˆ₯\lVert {\color{#d81b60}\vec e} \rVert.

βˆ₯eβƒ—βˆ₯\lVert {\color{#d81b60}\vec e} \rVert

This is the same as minimizing

βˆ₯uβƒ—βˆ’kvβƒ—βˆ₯\lVert {\color{orange}\vec u} - {\color{#004d40} k\vec v} \rVert

One way to approach this problem is to treat the above expression as a function of kk and find the value of kk that minimizes it through calculus. You’ll do this in Homework 3. I’ll show you a more geometric approach here.

We’ve guessed, but not yet shown that the shortest possible error vector is the one that is orthogonal to vβƒ—\color{#3d81f6}\vec v. Let kok_o be the value of kk that makes the error vector eβƒ—o=uβƒ—βˆ’kovβƒ—{\color{#d81b60}\vec e_o} = {\color{orange}\vec u} - \color{#004d40} k_o\vec v orthogonal to vβƒ—\color{#3d81f6}\vec v. Here, we’ll prove that kok_o is the β€œbest” choice of kk by showing that any other choice of kk will result in an error vector that is longer than eβƒ—o\color{#d81b60}\vec e_o.

For comparison, let kβ€²k' be some other value of kk, and let its error vector be eβƒ—β€²=uβƒ—βˆ’kβ€²vβƒ—\vec e' = {\color{orange}\vec u} - k' \color{#3d81f6}\vec v.

Loading...
Loading...
Image produced in Jupyter

I’ve drawn kβ€²vβƒ—k' \vec v in gray. Arbitrarily, I’ve shown it as being shorter than kovβƒ—\color{#004d40} k_o \vec v, but I could have drawn it as being longer and the argument would be the same. The prime has nothing to do with derivatives, by the way – it’s just a new variable.

The vectors kβ€²vβƒ—k' \vec v and kovβƒ—\color{#004d40} k_o \vec v, along with their corresponding error vectors, create a right-angled triangle, shaded in gold above. This triangle has a hypotenuse of eβƒ—β€²\vec e' and legs of eβƒ—o\color{#d81b60} \vec e_o and kβ€²vβƒ—βˆ’kovβƒ—k' \vec v - \color{#004d40} k_o \vec v.

Applying the Pythagorean theorem to this triangle gives us

βˆ₯eβƒ—β€²βˆ₯2=βˆ₯eβƒ—oβˆ₯2+βˆ₯kβ€²vβƒ—βˆ’kovβƒ—βˆ₯2⏟>0\lVert \vec e' \rVert^2 = \lVert {\color{#d81b60}\vec e_o} \rVert^2 + \underbrace{\lVert k' \vec v - {\color{#004d40} k_o \vec v} \rVert^2}_{> 0}

βˆ₯kβ€²vβƒ—βˆ’kovβƒ—βˆ₯2β‰₯0\lVert k' \vec v - {\color{#004d40} k_o \vec v} \rVert^2 \geq 0, with equality only when we choose kβ€²=kok' = k_o, but we’ve assumed that kβ€²β‰ kok' \neq k_o.

This implies that

βˆ₯eβƒ—β€²βˆ₯2=βˆ₯eβƒ—oβˆ₯2+someΒ positiveΒ number\lVert \vec e' \rVert^2 = \lVert {\color{#d81b60}\vec e_o} \rVert^2 + \text{some positive number}

which means that βˆ₯eβƒ—β€²βˆ₯2>βˆ₯eβƒ—oβˆ₯2\lVert \vec e' \rVert^2 > \lVert {\color{#d81b60}\vec e_o} \rVert^2.

In other words, if kβ€²β‰ kok' \neq k_o, then βˆ₯eβƒ—β€²βˆ₯>βˆ₯eβƒ—oβˆ₯\lVert \vec e' \rVert > \lVert {\color{#d81b60}\vec e_o} \rVert. Thus, the error vector eβƒ—o\color{#d81b60}\vec e_o is shorter than any other error vector eβƒ—β€²\vec e', and the best choice of kk is kok_o!

What was the point of all this again?

We know that the answer is kβˆ—vβƒ—\color{#004d40} k^* \vec v, where kβˆ—k^* is the value of kk that makes the error vector eβƒ—=uβƒ—βˆ’kβˆ—vβƒ—{\color{#d81b60}\vec e} = {\color{orange}\vec u} - \color{#004d40} k^* \vec v orthogonal to vβƒ—\color{#3d81f6}\vec v. (I’ve switched from calling this optimal scalar kok_o to kβˆ—k^*; kok_o was a name I used in the proof above, but more generally, β€œoptimal” values are starred for our purposes).

Let’s now find the value of kβˆ—k^*, in terms of just uβƒ—\color{orange} \vec u and vβƒ—\color{#3d81f6} \vec v. If eβƒ—\color{#d81b60}\vec e is orthogonal to vβƒ—\color{#3d81f6}\vec v, then vβƒ—β‹…eβƒ—=0{\color{#3d81f6}\vec v} \cdot {\color{#d81b60}\vec e} = 0.

vβƒ—β‹…eβƒ—=0vβƒ—β‹…(uβƒ—βˆ’kβˆ—vβƒ—)=0vβƒ—β‹…uβƒ—βˆ’vβƒ—β‹…kβˆ—vβƒ—=0vβƒ—β‹…uβƒ—βˆ’kβˆ—(vβƒ—β‹…vβƒ—)=0vβƒ—β‹…uβƒ—=kβˆ—(vβƒ—β‹…vβƒ—)kβˆ—=uβƒ—β‹…vβƒ—vβƒ—β‹…vβƒ—\begin{align*} {\color{#3d81f6}\vec v} \cdot {\color{#d81b60}\vec e} &= 0 \\ {\color{#3d81f6}\vec v} \cdot ({\color{orange}\vec u} - {\color{#004d40} k^* \vec v}) &= 0 \\ {\color{#3d81f6}\vec v} \cdot {\color{orange}\vec u} - {\color{#3d81f6}\vec v} \cdot {\color{#004d40} k^* \vec v} &= 0 \\ {\color{#3d81f6}\vec v} \cdot {\color{orange}\vec u} - {\color{#004d40} k^*} ({\color{#3d81f6}\vec v} \cdot {\color{#3d81f6} \vec v}) &= 0 \\ {\color{#3d81f6}\vec v} \cdot {\color{orange}\vec u} &= {\color{#004d40} k^*} ({\color{#3d81f6}\vec v} \cdot {\color{#3d81f6} \vec v}) \\ {\color{#004d40} k^*} &= \boxed{\frac{{\color{orange}\vec u} \cdot {\color{#3d81f6}\vec v}}{{\color{#3d81f6}\vec v} \cdot {\color{#3d81f6}\vec v}}} \end{align*}

The boxed value above is a scalar. It tells us the optimal amount to multiply vβƒ—\color{#3d81f6}\vec v by to get the best approximation of uβƒ—\color{orange}\vec u. Once we multiply that boxed scalar, kβˆ—k^*, by the vector vβƒ—\color{#3d81f6}\vec v, we get what’s called the orthogonal projection of uβƒ—\color{orange}\vec u onto vβƒ—\color{#3d81f6}\vec v.

Among all vectors of the form kv⃗k\color{#3d81f6}\vec v, p⃗\vec p above is the one that is closest to u⃗\color{orange}\vec u.

Why β€œorthogonal projection”? β€œOrthogonal” comes from the fact that pβƒ—\vec p’s error vector eβƒ—=uβƒ—βˆ’pβƒ—{\color{#d81b60}\vec e} = {\color{orange}\vec u} - \vec p is orthogonal to vβƒ—\color{#3d81f6}\vec v. β€œProjection” comes from the intuition you should have that pβƒ—\vec p is the shadow of uβƒ—\color{orange}\vec u onto vβƒ—\color{#3d81f6}\vec v.

Loading...
Loading...
Image produced in Jupyter

We’ve defined the error vector as eβƒ—=uβƒ—βˆ’pβƒ—{\color{#d81b60}\vec e} = {\color{orange}\vec u} - \vec p, and we know that eβƒ—{\color{#d81b60}\vec e} is orthogonal to vβƒ—\color{#3d81f6}\vec v. Rearranging the definition of the error vector gives us

uβƒ—=pβƒ—βŸparallelΒ toΒ vβƒ—+eβƒ—βŸorthogonalΒ toΒ vβƒ—{\color{orange} \vec u} = \underbrace{\vec p}_\text{parallel to $\color{#3d81f6}\vec v$} + \underbrace{{\color{#d81b60}\vec e}}_\text{orthogonal to $\color{#3d81f6}\vec v$}

All this says is that u⃗\color{orange}\vec u is the sum of:

  • pβƒ—\color{#004d40}\vec p, which is parallel to vβƒ—\color{#3d81f6}\vec v (by definition of orthogonal projection)

  • eβƒ—\color{#d81b60}\vec e, which is orthogonal to vβƒ—\color{#3d81f6}\vec v (by definition of error vector)

Sometimes, we call this the orthogonal decomposition of uβƒ—\color{orange}\vec u with respect to vβƒ—\color{#3d81f6}\vec v. I’ll speak more about decompositions later in this section.

ExamplesΒΆ

Let’s make things concrete by working through several examples. Each one was carefully chosen to illustrate something in particular.

We will work through several of these examples in lecture; attempt the ones that we don’t on your own.

Example: FundamentalsΒΆ

Let u⃗=[122]\vec u = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} and v⃗=[111]\vec v = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}.

  1. Find the orthogonal projection of u⃗\vec u onto v⃗\vec v.

  2. Find the error vector, i.e. the vector eβƒ—=uβƒ—βˆ’pβƒ—\vec e = \vec u - \vec p, and verify that it is orthogonal to vβƒ—\vec v.

  3. What is the length of the error vector (i.e. the projection error)?

Solution

Part 1

The orthogonal projection of u⃗\vec u onto v⃗\vec v is given by

p⃗=u⃗⋅v⃗v⃗⋅v⃗v⃗\vec p = \frac{\vec u \cdot \vec v}{\vec v \cdot \vec v} \vec v

Following the same steps as in the previous example, we have

u⃗⋅v⃗=[122]⋅[111]=1+2+2=5\vec u \cdot \vec v = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = 1 + 2 + 2 = 5
v⃗⋅v⃗=[111]⋅[111]=1+1+1=3\vec v \cdot \vec v = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = 1 + 1 + 1 = 3

So, the orthogonal projection of u⃗\vec u onto v⃗\vec v is

p⃗=53[111]=[5/35/35/3]\vec p = \frac{5}{3} \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 5 / 3 \\ 5 / 3 \\ 5 / 3 \end{bmatrix}

Part 2

The error vector is

eβƒ—=uβƒ—βˆ’pβƒ—=[122]βˆ’[5/35/35/3]=[1βˆ’5/32βˆ’5/32βˆ’5/3]=[βˆ’2/31/31/3]\vec e = \vec u - \vec p = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} - \begin{bmatrix} 5 / 3 \\ 5 / 3 \\ 5 / 3 \end{bmatrix} = \begin{bmatrix} 1 - 5 / 3 \\ 2 - 5 / 3 \\ 2 - 5 / 3 \end{bmatrix} = \begin{bmatrix} -2 / 3 \\ 1 / 3 \\ 1 / 3 \end{bmatrix}

To check if it’s orthogonal to vβƒ—\vec v, we compute their dot product; we’re hoping it’s 0.

eβƒ—β‹…vβƒ—=[βˆ’2/31/31/3]β‹…[111]=βˆ’2/3+1/3+1/3=0\vec e \cdot \vec v = \begin{bmatrix} -2 / 3 \\ 1 / 3 \\ 1 / 3 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = -2 / 3 + 1 / 3 + 1 / 3 = 0

So, the error vector e⃗\vec e is orthogonal to v⃗\vec v.

Part 3

The length of the error vector is

βˆ₯eβƒ—βˆ₯=(βˆ’2/3)2+(1/3)2+(1/3)2=4/9+1/9+1/9=6/9=2/3\begin{align*} \lVert \vec e \rVert &= \sqrt{(-2 / 3)^2 + (1 / 3)^2 + (1 / 3)^2} \\ &= \sqrt{4 / 9 + 1 / 9 + 1 / 9} \\ &= \sqrt{6 / 9} = \sqrt{2 / 3} \end{align*}

We might say 2/3\sqrt{2 / 3} is the projection error. Another way of thinking of it is as the shortest distance from the point (1,2,2)(1, 2, 2) to the line that passes through (0,0,0)(0, 0, 0) and (1,1,1)(1, 1, 1).


Example: The Line PerspectiveΒΆ

Consider the points p1=(1,2,2)p_1 = (1, 2, 2) and p2=(1,1,1)p_2 = (1, 1, 1) in R3\mathbb{R}^3.

What is the shortest distance between p1p_1 and the line that passes through p2p_2 and the origin, (0,0,0)(0, 0, 0)?

Solution

The answer is 2/3\sqrt{2 / 3}. This example didn’t require any addition math beyond the previous example; it just serves to remind you of the geometry of the situation. The set of all possible scalar multiples of vβƒ—\vec v fill out a line in R3\mathbb{R}^3, and that line passes through (0,0,0)(0, 0, 0) and p2=(1,1,1)p_2 = (1, 1, 1).

Why does that line pass through (0,0,0)(0, 0, 0)? Consider the vector 0vβƒ—0\vec v – it’s the zero vector!


Example: Which Order?ΒΆ

Let u⃗=[122]\vec u = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} and v⃗=[111]\vec v = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}.

In the first example, we found the orthogonal projection of u⃗\vec u onto v⃗\vec v.

Now, do the opposite: find the orthogonal projection of v⃗\vec v onto u⃗\vec u.

Solution

Now, we’re projecting vβƒ—\vec v onto uβƒ—\vec u, which means our answer is going to be a multiple of uβƒ—\vec u, not vβƒ—\vec v as in the first part.

The orthogonal projection of v⃗\vec v onto u⃗\vec u is given by

p⃗=v⃗⋅u⃗u⃗⋅u⃗u⃗\vec p = \frac{\vec v \cdot \vec u}{\vec u \cdot \vec u} \vec u

The formula for the scalar in front of uβƒ—\vec u is the same as in Part 1, but with all vβƒ—\vec v’s replaced by uβƒ—\vec u’s and vice versa. The numerator is the same, since uβƒ—β‹…vβƒ—=vβƒ—β‹…uβƒ—\vec u \cdot \vec v = \vec v \cdot \vec u. The denominator is different; just remember that the denominator is the squared norm of the vector you’re projecting onto.

v⃗⋅u⃗=[111]⋅[122]=1+2+2=5\vec v \cdot \vec u = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} = 1 + 2 + 2 = 5
u⃗⋅u⃗=[122]⋅[122]=1+4+4=9\vec u \cdot \vec u = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} = 1 + 4 + 4 = 9

So, the orthogonal projection of v⃗\vec v onto u⃗\vec u is

p⃗=v⃗⋅u⃗u⃗⋅u⃗u⃗=59[122]=[5/910/910/9]\vec p = \frac{\vec v \cdot \vec u}{\vec u \cdot \vec u} \vec u = \frac{5}{9} \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} = \begin{bmatrix} 5 / 9 \\ 10 / 9 \\ 10 / 9 \end{bmatrix}

Note that the corresponding error vector, eβƒ—=vβƒ—βˆ’pβƒ—\vec e = \vec v - \vec p, is orthogonal to uβƒ—\vec u (not vβƒ—\vec v), since uβƒ—\vec u is the vector we projected onto.


Example: Unit VectorsΒΆ

Let uβƒ—=[4βˆ’12]\vec u = \begin{bmatrix} 4 \\ -1 \\ 2 \end{bmatrix}, vβƒ—=[26βˆ’3]\vec v = \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} and Vβƒ—=[2/76/7βˆ’3/7]\vec V = \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix}.

  1. Find the orthogonal projection of u⃗\vec u onto v⃗\vec v.

  2. Find the orthogonal projection of u⃗\vec u onto V⃗\vec V.

  3. What do you notice about your answers to the above two parts?

Solution

Part 1

We know that the orthogonal projection of u⃗\vec u onto v⃗\vec v is given by

p⃗=u⃗⋅v⃗v⃗⋅v⃗v⃗\vec p = \frac{\vec u \cdot \vec v}{\vec v \cdot \vec v} \vec v

Let’s compute the relevant dot products.

uβƒ—β‹…vβƒ—=[4βˆ’12]β‹…[26βˆ’3]=4β‹…2+(βˆ’1)β‹…6+2β‹…(βˆ’3)=8βˆ’6βˆ’6=βˆ’4\vec u \cdot \vec v = \begin{bmatrix} 4 \\ -1 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} = 4 \cdot 2 + (-1) \cdot 6 + 2 \cdot (-3) = 8 - 6 - 6 = -4
vβƒ—β‹…vβƒ—=[26βˆ’3]β‹…[26βˆ’3]=2β‹…2+6β‹…6+(βˆ’3)β‹…(βˆ’3)=4+36+9=49\vec v \cdot \vec v = \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} \cdot \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} = 2 \cdot 2 + 6 \cdot 6 + (-3) \cdot (-3) = 4 + 36 + 9 = 49

So, the orthogonal projection of u⃗\vec u onto v⃗\vec v is

pβƒ—=βˆ’449[26βˆ’3]=[βˆ’8/49βˆ’24/4912/49]\vec p = \frac{-4}{49} \begin{bmatrix} 2 \\ 6 \\ -3 \end{bmatrix} = \begin{bmatrix} -8 / 49 \\ -24 / 49 \\ 12 / 49 \end{bmatrix}

Part 2

Now, we need to find u⃗⋅V⃗V⃗⋅V⃗V⃗\frac{\vec u \cdot \vec V}{\vec V \cdot \vec V} \vec V.

Let’s compute the relevant dot products.

uβƒ—β‹…Vβƒ—=[4βˆ’12]β‹…[2/76/7βˆ’3/7]=87βˆ’67βˆ’67=βˆ’47\vec u \cdot \vec V = \begin{bmatrix} 4 \\ -1 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix} = \frac{8}{7} - \frac{6}{7} - \frac{6}{7} = -\frac{4}{7}
Vβƒ—β‹…Vβƒ—=[2/76/7βˆ’3/7]β‹…[2/76/7βˆ’3/7]=449+3649+949=4949=1\vec V \cdot \vec V = \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix} \cdot \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix} = \frac{4}{49} + \frac{36}{49} + \frac{9}{49} = \frac{49}{49} = 1

So, the orthogonal projection of u⃗\vec u onto V⃗\vec V is

pβƒ—=βˆ’471[2/76/7βˆ’3/7]=[βˆ’8/49βˆ’24/4912/49]\vec p = \frac{-\frac{4}{7}}{1} \begin{bmatrix} 2 / 7 \\ 6 / 7 \\ -3 / 7 \end{bmatrix} = \begin{bmatrix} -8 / 49 \\ -24 / 49 \\ 12 / 49 \end{bmatrix}

Part 3

Notice that in both parts, the orthogonal projection p⃗\vec p is the same! This is not a coincidence. Both vectors point in the same direction, meaning the set of possible vectors of the form kV⃗k \vec V is the same as the set of possible vectors of the form kv⃗k \vec v. Another way to think about this is that they both span the same line in R3\mathbb{R}^3 through the origin.

The difference between vβƒ—\vec v and Vβƒ—\vec V is that Vβƒ—\vec V is a unit vector in the direction of vβƒ—\vec v, meaning that it points in the same direction as vβƒ—\vec v but has βˆ₯Vβƒ—βˆ₯=1\lVert \vec V \rVert = 1 rather than βˆ₯vβƒ—βˆ₯=7\lVert \vec v \rVert = 7.

What’s different is the scalar we need to multiply each vector by to get the orthogonal projection. In the case of the unit vector Vβƒ—\vec V, the number in front of Vβƒ—\vec V is uβƒ—β‹…Vβƒ—Vβƒ—β‹…Vβƒ—\frac{\vec u \cdot \vec V}{\vec V \cdot \vec V}, but since Vβƒ—β‹…Vβƒ—=βˆ₯Vβƒ—βˆ₯2=1\vec V \cdot \vec V = \lVert \vec V \rVert^2 = 1, this simplifies to uβƒ—β‹…Vβƒ—\vec u \cdot \vec V.


Example: Unit Vectors, ContinuedΒΆ

Suppose uβƒ—,vβƒ—βˆˆRn\vec u, \vec v \in \mathbb{R}^n. Let ΞΈ\theta be the angle between uβƒ—\vec u and vβƒ—\vec v.

Show that the orthogonal projection of u⃗\vec u onto v⃗\vec v is equal to

pβƒ—=(βˆ₯uβƒ—βˆ₯cos⁑θ)vβƒ—βˆ₯vβƒ—βˆ₯\vec p = \left( \lVert \vec u \rVert \cos \theta \right) \frac{\vec v}{\lVert \vec v \rVert}

This is not a formula we’d use to actually compute pβƒ—\vec p, since finding cos⁑θ\cos \theta is harder than using the dot product-based formula from above. But, what does this formula tell us about the relationship between uβƒ—\vec u and pβƒ—\vec p?

Solution

Let’s start with the original formula for the orthogonal projection of uβƒ—\vec u onto vβƒ—\vec v.

p⃗=u⃗⋅v⃗v⃗⋅v⃗v⃗\vec p = \frac{\vec u \cdot \vec v}{\vec v \cdot \vec v} \vec v

Using the fact that uβƒ—β‹…vβƒ—=βˆ₯uβƒ—βˆ₯βˆ₯vβƒ—βˆ₯cos⁑θ\vec u \cdot \vec v = \lVert \vec u \rVert \lVert \vec v \rVert \cos \theta and vβƒ—β‹…vβƒ—=βˆ₯vβƒ—βˆ₯2\vec v \cdot \vec v = \lVert \vec v \rVert^2, we can rewrite the formula as

pβƒ—=βˆ₯uβƒ—βˆ₯βˆ₯vβƒ—βˆ₯cos⁑θβˆ₯vβƒ—βˆ₯2vβƒ—=βˆ₯uβƒ—βˆ₯cos⁑θβˆ₯vβƒ—βˆ₯vβƒ—=(βˆ₯uβƒ—βˆ₯cos⁑θ)vβƒ—βˆ₯vβƒ—βˆ₯\vec p = \frac{\lVert \vec u \rVert \lVert \vec v \rVert \cos \theta}{\lVert \vec v \rVert^2} \vec v = \frac{\lVert \vec u \rVert \cos \theta}{\lVert \vec v \rVert} \vec v = (\lVert \vec u \rVert \cos \theta) \frac{\vec v}{\lVert \vec v \rVert}

The parentheses around βˆ₯uβƒ—βˆ₯cos⁑θ\lVert \vec u \rVert \cos \theta don’t change the calculation, but they help with the interpretation. This shows us that we can think of the orthogonal projection of uβƒ—\vec u onto vβƒ—\vec v as a vector with:

  • a length of βˆ₯uβƒ—βˆ₯cos⁑θ\lVert \vec u \rVert \cos \theta

  • in the direction of vβƒ—βˆ₯vβƒ—βˆ₯\frac{\vec v}{\lVert \vec v \rVert}, which is a unit vector in the direction of vβƒ—\vec v


Example: Projecting onto an Orthogonal VectorΒΆ

Let uβƒ—=[4βˆ’12]\vec u = \begin{bmatrix} 4 \\ -1 \\ 2 \end{bmatrix} and vβƒ—=[βˆ’306]\vec v = \begin{bmatrix} -3 \\ 0 \\ 6 \end{bmatrix}.

u⃗\vec u and v⃗\vec v are orthogonal. What does this say about the orthogonal projection of u⃗\vec u onto v⃗\vec v?

Solution

Since u⃗⋅v⃗=0\vec u \cdot \vec v = 0, the orthogonal projection of u⃗\vec u onto v⃗\vec v is the zero vector, 0⃗\vec 0.

p=u⃗⋅v⃗v⃗⋅v⃗v⃗=036v⃗=0⃗p = \frac{\vec u \cdot \vec v}{\vec v \cdot \vec v} \vec v = \frac{0}{36} \vec v = \vec 0

Intuitively, uβƒ—\vec u and vβƒ—\vec v travel in totally different different directions. Travelling any amount of vβƒ—\vec v will take you further away from uβƒ—\vec u. So, it’s best to stick with the zero vector, 0βƒ—\vec 0.

Orthogonal DecompositionΒΆ

To motivate the next section, let’s consider another example.

Let uβƒ—=[35]\color{orange} \vec u = \begin{bmatrix} 3 \\ 5 \end{bmatrix}, vβƒ—1=[6βˆ’2]\color{#3d81f6} \vec v_1 = \begin{bmatrix} 6 \\ -2 \end{bmatrix}, and vβƒ—2=[13]\color{#3d81f6} \vec v_2 = \begin{bmatrix} 1 \\ 3 \end{bmatrix}.

Image produced in Jupyter

Notice that vβƒ—1\color{#3d81f6}\vec v_1 and vβƒ—2\color{#3d81f6}\vec v_2 are orthogonal – that’s important to what we’re about to discover.

Let’s find the orthogonal projection of uβƒ—\color{orange}\vec u onto vβƒ—1\color{#3d81f6}\vec v_1 (called pβƒ—1\color{#004d40}\vec p_1) and the orthogonal projection of uβƒ—\color{orange}\vec u onto vβƒ—2\color{#3d81f6}\vec v_2 (called pβƒ—2\color{#004d40}\vec p_2).

pβƒ—1=uβƒ—β‹…vβƒ—1vβƒ—1β‹…vβƒ—1vβƒ—1=840vβƒ—1=15vβƒ—1=[6/5βˆ’2/5]{\color{#004d40}\vec p_1} = \frac{{\color{orange}\vec u} \cdot \color{#3d81f6}\vec v_1}{\color{#3d81f6}\vec v_1 \cdot \color{#3d81f6}\vec v_1} {\color{#3d81f6}\vec v_1} = \frac{8}{40} {\color{#3d81f6}\vec v_1} = \frac{1}{5} {\color{#3d81f6}\vec v_1} = {\color{#004d40}\begin{bmatrix} 6 / 5 \\ -2 / 5 \end{bmatrix}}
p⃗2=u⃗⋅v⃗2v⃗2⋅v⃗2v⃗2=1810v⃗2=95v⃗2=[9/527/5]{\color{#004d40}\vec p_2} = \frac{{\color{orange}\vec u} \cdot \color{#3d81f6}\vec v_2}{\color{#3d81f6}\vec v_2 \cdot \color{#3d81f6}\vec v_2} {\color{#3d81f6}\vec v_2} = \frac{18}{10} {\color{#3d81f6}\vec v_2} = \frac{9}{5} {\color{#3d81f6} \vec v_2} = {\color{#004d40}\begin{bmatrix} 9 / 5 \\ 27 / 5 \end{bmatrix}}

Notice that u⃗\color{orange}\vec u is the sum of p⃗1\color{#004d40}\vec p_1 and p⃗2\color{#004d40}\vec p_2!

uβƒ—=pβƒ—1+pβƒ—2=15vβƒ—1+95vβƒ—2⏟linearΒ combination=[6/5βˆ’2/5]+[9/527/5]=[35]{\color{orange}\vec u} = {\color{#004d40}\vec p_1} + {\color{#004d40}\vec p_2} = \underbrace{\frac{1}{5} {\color{#3d81f6}\vec v_1} + \frac{9}{5} {\color{#3d81f6}\vec v_2}}_{\text{linear combination}} = \begin{bmatrix} 6 / 5 \\ -2 / 5 \end{bmatrix} + \begin{bmatrix} 9 / 5 \\ 27 / 5 \end{bmatrix} = \begin{bmatrix} 3 \\ 5 \end{bmatrix}
Image produced in Jupyter

Why is the sum of p⃗1\color{#004d40}\vec p_1 and p⃗2\color{#004d40}\vec p_2 equal to u⃗\color{orange}\vec u? Earlier, I mentioned that we can use orthogonal projections to decompose vectors. Here, when we project u⃗\color{orange}\vec u onto v⃗1\color{#3d81f6}\vec v_1, the corresponding error vector e⃗1\color{#d81b60}\vec e_1 is orthogonal to v⃗1\color{#3d81f6}\vec v_1.

uβƒ—=pβƒ—1⏟parallelΒ toΒ vβƒ—1+eβƒ—1⏟orthogonalΒ toΒ vβƒ—1{\color{orange} \vec u} = \underbrace{{\color{#004d40}\vec p_1}}_{\text{parallel to } \color{#3d81f6}\vec v_1} + \underbrace{{\color{#d81b60}\vec e_1}}_{\text{orthogonal to } \color{#3d81f6}\vec v_1}

By projecting u⃗\color{orange}\vec u onto v⃗2\color{#3d81f6}\vec v_2, we can recreate the error vector exactly, meaning e⃗1=p⃗2{\color{#d81b60}\vec e_1} = \color{#004d40}\vec p_2.

Taking a step back, the fact that v1\color{#3d81f6} v_1 and v2\color{#3d81f6} v_2 are orthogonal meant that writing u⃗\color{orange}\vec u as a linear combination of v1\color{#3d81f6} v_1 and v2\color{#3d81f6} v_2 was easy.

a1[6βˆ’2]⏟vβƒ—1+a2[13]⏟vβƒ—2=[35]⏟uβƒ—a_1 {\color{#3d81f6} \underbrace{\begin{bmatrix} 6 \\ -2 \end{bmatrix}}_{\vec v_1}} + a_2 {\color{#3d81f6} \underbrace{\begin{bmatrix} 1 \\ 3 \end{bmatrix}}_{\vec v_2}} = \color{orange} \underbrace{\begin{bmatrix} 3 \\ 5 \end{bmatrix}}_{\vec u}

If v1\color{#3d81f6} v_1 and v2\color{#3d81f6} v_2 were not orthogonal, then writing uβƒ—\color{orange}\vec u as a linear combination of v1\color{#3d81f6} v_1 and v2\color{#3d81f6} v_2 would have involved solving a system of 2 equations and 2 unknowns, as we’ve had to do in previous sections.

For instance, if we keep [35]\color{orange} \begin{bmatrix} 3 \\ 5 \end{bmatrix} and look at x⃗1=[12]\color{#3d81f6} \vec x_1 = \begin{bmatrix} 1 \\ 2 \end{bmatrix} and x⃗2=[21]\color{#3d81f6} \vec x_2 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}, we have that

  • the projection of uβƒ—\color{orange}\vec u onto xβƒ—1\color{#3d81f6} \vec x_1 is (uβƒ—β‹…xβƒ—1xβƒ—1β‹…xβƒ—1)xβƒ—1=135xβƒ—1 \left( \frac{{\color{orange}\vec u} \cdot \color{#3d81f6} \vec x_1}{{\color{#3d81f6} \vec x_1} \cdot \color{#3d81f6} \vec x_1} \right) {\color{#3d81f6} \vec x_1} = \frac{13}{5} \color{#3d81f6} \vec x_1

  • the projection of uβƒ—\color{orange}\vec u onto xβƒ—2\color{#3d81f6} \vec x_2 is (uβƒ—β‹…xβƒ—2xβƒ—2β‹…xβƒ—2)xβƒ—2=115xβƒ—2 \left( \frac{{\color{orange}\vec u} \cdot \color{#3d81f6} \vec x_2}{{\color{#3d81f6} \vec x_2} \cdot \color{#3d81f6} \vec x_2} \right) {\color{#3d81f6} \vec x_2} = \frac{11}{5} \color{#3d81f6} \vec x_2

but

135x⃗1+115x⃗2=135[12]+115[21]=[13/5+22/526/5+11/5]=[77.4]≠u⃗\frac{13}{5} {\color{#3d81f6} \vec x_1} + \frac{11}{5} {\color{#3d81f6} \vec x_2} = \frac{13}{5} \begin{bmatrix} 1 \\ 2 \end{bmatrix} + \frac{11}{5} \begin{bmatrix} 2 \\ 1 \end{bmatrix} = \begin{bmatrix} 13 / 5 + 22 / 5 \\ 26 / 5 + 11 / 5 \end{bmatrix} = \begin{bmatrix} 7 \\ 7.4 \end{bmatrix} \neq {\color{orange}\vec u}

I’d like to provide a more general β€œtheorem”, on when you can use orthogonal projections to more easily write a vector uβƒ—\color{orange}\vec u as a linear combination of vectors vβƒ—1\color{#3d81f6} \vec v_1, vβƒ—2\color{#3d81f6} \vec v_2, ..., vβƒ—d\color{#3d81f6} \vec v_d, but we’ll need to first study the idea of a basis. That’s to come.

This section has been light on activities, since it provided many examples that we’ll work through in lecture. But, here’s one to tie this last point together.

Activity 1ΒΆ

Activity 1

Let vβƒ—1=[βˆ’122]\vec v_1 = \begin{bmatrix} -1 \\ 2 \\ 2 \end{bmatrix}, vβƒ—2=[22βˆ’1]\vec v_2 = \begin{bmatrix} 2 \\ 2 \\ -1 \end{bmatrix}, and vβƒ—3=[2βˆ’12]\vec v_3 = \begin{bmatrix} 2 \\ -1 \\ 2 \end{bmatrix}.

Write u⃗=[111]\vec u = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} as a linear combination of v⃗1\vec v_1, v⃗2\vec v_2, and v⃗3\vec v_3, without solving a system of equations.

Solution

The answer is u⃗=13v⃗1+13v⃗2+13v⃗3\boxed{\vec u = \frac{1}{3}\vec v_1 + \frac{1}{3}\vec v_2 + \frac{1}{3}\vec v_3}.

The useful observation is that the three vectors are pairwise orthogonal:

vβƒ—1β‹…vβƒ—2=βˆ’2+4βˆ’2=0\vec v_1 \cdot \vec v_2 = -2 + 4 - 2 = 0
vβƒ—1β‹…vβƒ—3=βˆ’2βˆ’2+4=0\vec v_1 \cdot \vec v_3 = -2 - 2 + 4 = 0
vβƒ—2β‹…vβƒ—3=4βˆ’2βˆ’2=0\vec v_2 \cdot \vec v_3 = 4 - 2 - 2 = 0

This suggests projecting u⃗\vec u onto each direction separately. For every ii, u⃗⋅v⃗i=3\vec u \cdot \vec v_i = 3 and v⃗i⋅v⃗i=9\vec v_i \cdot \vec v_i = 9, so each projection is:

u⃗⋅v⃗iv⃗i⋅v⃗iv⃗i=39v⃗i=13v⃗i\frac{\vec u \cdot \vec v_i}{\vec v_i \cdot \vec v_i}\vec v_i = \frac{3}{9}\vec v_i = \frac{1}{3}\vec v_i

Let’s verify that adding the three projections gives all of uβƒ—\vec u:

13([βˆ’122]+[22βˆ’1]+[2βˆ’12])=13[333]=[111]\frac{1}{3}\left(\begin{bmatrix} -1 \\ 2 \\ 2 \end{bmatrix} + \begin{bmatrix} 2 \\ 2 \\ -1 \end{bmatrix} + \begin{bmatrix} 2 \\ -1 \\ 2 \end{bmatrix}\right) = \frac{1}{3}\begin{bmatrix} 3 \\ 3 \\ 3 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}

So, the projections give exactly the linear combination we need, without solving a system of equations.

What’s Next?ΒΆ

The motivating problem for this section was the approximation problem, which asked us to find the best approximation of a vector u⃗\color{orange}\vec u using only a scalar multiple of a vector v⃗\color{#3d81f6}\vec v.

The next natural step is to consider the case where we want to approximate uβƒ—\color{orange}\vec u using a linear combination of more than one vector, vβƒ—1\color{#3d81f6} \vec v_1, vβƒ—2\color{#3d81f6} \vec v_2, ..., vβƒ—d\color{#3d81f6} \vec v_d. Why? Remember, this all connects back to the problem of linear regression. The more vectors we have as β€œbuilding blocks” in our linear combination, the more features our model will be able to use. (I haven’t made the connection from linear algebra to linear regression yet, but just know this is why we’re studying projections.)

For example, let’s consider uβƒ—=[136]\color{orange}\vec u = \begin{bmatrix} 1 \\ 3 \\ 6 \end{bmatrix}, vβƒ—1=[101]\color{#3d81f6} \vec v_1 = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}, and vβƒ—2=[221]\color{#3d81f6} \vec v_2 = \begin{bmatrix} 2 \\ 2 \\ 1 \end{bmatrix}. Among all linear combinations of vβƒ—1\color{#3d81f6} \vec v_1 and vβƒ—2\color{#3d81f6} \vec v_2, which one is closest to uβƒ—\color{orange}\vec u?

To answer this question, we’d need to find the scalars a1a_1 and a2a_2 such that the error vector

eβƒ—=uβƒ—βˆ’(a1vβƒ—1+a2vβƒ—2){\color{#d81b60}\vec e} = {\color{orange}\vec u} - {(a_1 {\color{#3d81f6} \vec v_1} + a_2 {\color{#3d81f6} \vec v_2})}

has minimal length, which presumably happens when e⃗\color{#d81b60}\vec e is orthogonal to both v⃗1\color{#3d81f6} \vec v_1 and v⃗2\color{#3d81f6} \vec v_2.

Travelling down this road, we might be able to find the values of a1a_1 and a2a_2 that minimize the length of eβƒ—\color{#d81b60}\vec e. But then we’ll want to ask how we can do this for any uβƒ—\color{orange}\vec u and any set of vectors vβƒ—1\color{#3d81f6} \vec v_1, vβƒ—2\color{#3d81f6} \vec v_2, ..., vβƒ—d\color{#3d81f6} \vec v_d, and it seems like we’ll need a more general solution. In general, to find the β€œbest” approximation of uβƒ—\color{orange}\vec u using a linear combination of vβƒ—1\color{#3d81f6} \vec v_1, vβƒ—2\color{#3d81f6} \vec v_2, ..., vβƒ—d\color{#3d81f6} \vec v_d, we’ll need to know about matrices. We’ll introduce matrices in Chapter 5.1.

Instead, in Chapter 4.1, we will set aside the goal of projections temporarily, and instead focus on truly understanding the set of possible linear combinations of a given set of vectors. For example, the vectors vβƒ—1=[100]\color{#3d81f6} \vec v_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, vβƒ—2=[βˆ’518]\color{#3d81f6} \vec v_2 = \begin{bmatrix} -5 \\ 1 \\ 8 \end{bmatrix} from earlier define a plane. So, asking which linear combination of vβƒ—1\color{#3d81f6} \vec v_1 and vβƒ—2\color{#3d81f6} \vec v_2 is closest to uβƒ—\color{orange}\vec u is equivalent to asking which point on the plane is closest to uβƒ—\color{orange}\vec u.

Loading...

Chapter 4.1 will answer the questions, β€œwhy do vβƒ—1\color{#3d81f6} \vec v_1 and vβƒ—2\color{#3d81f6} \vec v_2 define a plane?”, β€œwhich plane do they define?”, and β€œin general, what do vβƒ—1\color{#3d81f6} \vec v_1, vβƒ—2\color{#3d81f6} \vec v_2, ..., vβƒ—d\color{#3d81f6} \vec v_d, all in Rn\mathbb{R}^n, define?”

EECS 245 Course Notes
3.3. The Dot Product
EECS 245 Course Notes
Chapter 4: Linear Independence