2.1. Vectors

Linear algebra can be thought of as the study of vectors, matrices, and linear transformations, all of which are ideas we’ll need to use in our journey to understand machine learning. We’ll start with vectors, which are the building blocks of linear algebra.

Definition¶

There are many ways to define vectors, but I’ll give you the most basic and practically relevant definition of a vector for now. I’ll introduce more abstract definitions later if we need them.

By ordered list, I mean that the order of the numbers in the vector matters.

For example, the vector $\vec v = \begin{bmatrix} 4 \\ -3 \\ 15 \end{bmatrix}$ is not the same as the vector $\vec w = \begin{bmatrix} 15 \\ -3 \\ 4 \end{bmatrix}$ , even though they have the same components.
$\vec v$ is also different from the vector $\vec u = \begin{bmatrix} 4 \\ -3 \\ 15 \\ 1 \end{bmatrix}$ , even though their first three components are the same.

In general, we’re mostly concerned with vectors in $\mathbb{R}^n$ , which is the set of all vectors with $n$ components or elements, each of which is a real number. It’s possible to consider vectors with complex components (the set of all vectors with complex components is denoted $\mathbb{C}^n$ ), but we’ll stick to real vectors for now.

The vector $\vec v$ defined in the box above is in $\mathbb{R}^3$ , which we can express as $\vec v \in \mathbb{R}^3$ . This is pronounced as “v is an element of R three”.

A general vector in $\mathbb{R}^n$ can be expressed in terms of its $n$ components:

\vec v = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}

Subscripts can be used for different, sometimes conflicting purposes:

In the definition of $\vec v$ above, the components of the vector are denoted $v_1, v_2, \ldots, v_n$ . Each of these individual components is a single real number – known as a scalar, not a vector.
But in the near future, we may want to consider multiple vectors at once, and may use subscripts to refer to them as well. For instance, I might have $d$ different vectors, $\vec v_1, \vec v_2, \ldots, \vec v_d$ , each corresponding to some feature I care about.

The meaning of the subscript depends on the context, so just be careful!

While we’ll use the definition of a vector as a list of numbers for now, I hope you’ll soon appreciate that vectors are more than just a list of numbers – they encode remarkable amounts of information and beauty.

Norm (i.e. Length or Magnitude)¶

In the context of physics, vectors are often described as creatures with “a magnitude and a direction”. While this is not a physics class – this is EECS 245, after all! – this interpretation has some value for us too.

To illustrate what we mean, let’s consider some concrete vectors in $\mathbb{R}^2$ , since it is easy to visualize vectors in 2 dimensions on a computer screen. Suppose:

{\color{orange}\vec u = \begin{bmatrix} 3 \\ 1 \end{bmatrix}}, \quad {\color{3d81f6}\vec v = \begin{bmatrix} 4 \\ -6 \end{bmatrix}}

Then, we can visualize $\color{orange}{\vec u}$ and $\color{#3d81f6}{\vec v}$ as arrows pointing from the origin $(0,0)$ to the points $(3, 1)$ and $(4, -6)$ in the 2D Cartesian plane, respectively.

# This chunk must be in the first plotting cell of each notebook in order to guarantee that the mathjax script is loaded.
import plotly
from IPython.display import display, HTML

# Set default renderer to high-DPI static image
plotly.io.renderers.default = "png"

display(HTML(
    '<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-MML-AM_SVG"></script>'
))

# ---

from utils import plot_vectors
import numpy as np

fig = plot_vectors([((4, -6), '#3d81f6', r'$\vec v$'), ((3, 1), 'orange', r'$\vec u$')], vdeltax=0.3, vdeltay=0.8)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-1, 6], tickvals=np.arange(-5, 15))
fig.update_yaxes(range=[-8, 4], tickvals=np.arange(-8, 4))
fig.show(scale=3)

The vector $\vec v = \begin{bmatrix} 4 \\ -6 \end{bmatrix}$ moves 4 units to the right and 6 units down, which we know by reading the components of the vector. In Chapter 2.2, we’ll see how to describe the direction of $\vec v$ in terms of the angle it makes with the $x$ -axis (and you may remember how to calculate that angle using trigonometry).

It’s worth noting that $\vec v$ isn’t “fixed” to start at the origin – vectors don’t have positions. All three vectors in the figure below are the same vector, $\vec v$ .

from utils import plot_vectors_non_origin
import numpy as np

vectors_non_origin_2d = [
    (((2, 1), (2 + 4, 1 - 6)), "#3d81f6", r"$\vec v$"),     
    (((-0.5, 1.5), (-0.5 + 4, 1.5 - 6)), "#3d81f6", r"$\vec v$"),
    (((-2, -1), (-2 + 4, -1 - 6)), "#3d81f6", r"$\vec v$")
]
fig = plot_vectors_non_origin(vectors_non_origin_2d, vdeltay=0.9)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-1, 6], tickvals=np.arange(-5, 15))
fig.update_yaxes(range=[-8, 2], tickvals=np.arange(-8, 2))
fig.show(config={'displayModeBar': False}, scale=3)

To compute the length of $\vec v$ – i.e. the distance between $(0, 0)$ and $(4, -6)$ – we should remember the Pythagorean theorem, which states that if we have a right triangle with legs of length $a$ and $b$ , then the length of the hypotenuse is $\sqrt{a^2 + b^2}$ . Here, that’s $\sqrt{4^2 + (-6)^2} = \sqrt{16 + 36} = \sqrt{52} = 2\sqrt{13}$ .

from utils import plot_vectors
import numpy as np

fig = plot_vectors([((4, -6), '#3d81f6', r'$\vec v$')], vdeltax=0.3, vdeltay=1)

# Add horizontal dotted line from (0,0) to (4,0)
fig.add_shape(
    type="line",
    x0=0, y0=0, x1=4, y1=0,
    line=dict(color="blue", width=3, dash="dot")
)

# Add vertical dotted line from (4,0) to (4,-6)
fig.add_shape(
    type="line",
    x0=4, y0=0, x1=4, y1=-6,
    line=dict(color="blue", width=3, dash="dot")
)

# Add label "4" above the horizontal line
fig.add_annotation(
    x=2, y=0.3,
    text="$4$",
    showarrow=False,
    font=dict(size=14, color="blue")
)

# Add label "6" to the right of the vertical line
fig.add_annotation(
    x=4.3, y=-3,
    text="$6$",
    showarrow=False,
    font=dict(size=14, color="blue")
)

# Add diagonal dotted line from (0,0) to (4,-6) - the hypotenuse
fig.add_shape(
    type="line",
    x0=0, y0=0, x1=4, y1=-6,
    line=dict(color="blue", width=3, dash="dot")
)

# Add calculation annotation for the hypotenuse
fig.add_annotation(
    x=1.4, y=-3.5,
    text=r"$$\sqrt{4^2 + (-6)^2} = 2\sqrt{13}$$",
    showarrow=False,
    font=dict(size=12, color="blue"),
    bgcolor="rgba(255,255,255,0.8)",
    bordercolor="blue",
    borderwidth=1,
    textangle=np.arctan(6/4) * 180 / np.pi
)

fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-1, 6], tickvals=np.arange(-5, 15))
fig.update_yaxes(range=[-8, 2], tickvals=np.arange(-8, 2))
fig.show(config={'displayModeBar': False}, scale=3)

Note that the norm involves a sum of squares, much like mean squared error 🤯. This connection will be made more explicit in Chapter 3, when we return to studying linear regression.

Activity 1

As we’ll see later in this section, a unit vector is a vector with norm 1. It’s common to use unit vectors to describe directions. For instance, there are infinitely many vectors in $\mathbb{R}^2$ that point in the same direction as $\vec v = \begin{bmatrix} 4 \\ -6 \end{bmatrix}$ from above, like $\begin{bmatrix} 2 \\ -3 \end{bmatrix}$ and $\begin{bmatrix} 40 \\ -60 \end{bmatrix}$ . (If you don’t believe me, draw it out!)

Find a unit vector that points in the same direction as the vector $\vec x = \begin{bmatrix} 12 \\ 5 \end{bmatrix}$ , and verify that it has norm 1. Technically, to answer this, you’ll need to use the fact that vectors can be multiplied by a scalar, which we haven’t yet discussed, but see how far your intuition takes you!

What may not be immediately obvious is why the Pythagorean theorem seems to extend to higher dimensions. The 2D case seems reasonable, but why is the length of the vector $\color{#d81b60}\vec w = \begin{bmatrix} 6 \\ -2 \\ 3 \end{bmatrix}$ in $\mathbb{R}^3$ equal to $\sqrt{6^2 + (-2)^2 + 3^2}$ ?

from utils import plot_vectors
import numpy as np
import plotly.graph_objects as go

fig = plot_vectors([((6, -2, 3), '#d81b60', '<b>w</b>')], vdeltax=-0.6)

# Add light blue triangle connecting (0, 0, 0), (0, -2, 0), and (6, -2, 0)
fig.add_trace(go.Mesh3d(
    x=[0, 0, 6],
    y=[0, -2, -2],
    z=[0, 0, 0],
    i=[0],
    j=[1],
    k=[2],
    color='lightblue',
    opacity=0.7,
    showscale=False,
    name='Triangle 1'
))

# Add red triangle connecting (0, 0, 0), (6, -2, 0) and (6, -2, 3)
fig.add_trace(go.Mesh3d(
    x=[0, 6, 6],
    y=[0, -2, -2],
    z=[0, 0, 3],
    i=[0],
    j=[1],
    k=[2],
    color='#d81b60',
    opacity=0.5,
    showscale=False,
    name='Triangle 2'
))

# Add dotted blue lines for the edges
edges = [
    # Edge "2": (0,0,0) to (0,-2,0)
    ([0, 0], [0, -2], [0, 0]),
    # Edge "6": (0,-2,0) to (6,-2,0)
    ([0, 6], [-2, -2], [0, 0]),
    # Edge "h": (0,0,0) to (6,-2,0)
    ([0, 6], [0, -2], [0, 0]),
    # Edge "3": (6,-2,0) to (6,-2,3)
    ([6, 6], [-2, -2], [0, 3])
]

for x_coords, y_coords, z_coords in edges:
    fig.add_trace(go.Scatter3d(
        x=x_coords, y=y_coords, z=z_coords,
        mode='lines',
        line=dict(color='blue', width=3, dash='dash'),
        showlegend=False
    ))

# Add vertex labels with better positioning
vertices = {
    '(0,0,0)': (-0.3, 0.3, 0.2),
    '(0,-2,0)': (-0.3, -2.3, 0.2),
    '(6,-2,0)': (6.3, -2.3, 0.2),
    '(6,-2,3)': (6.3, -2.3, 3.3)
}

for label, (x, y, z) in vertices.items():
    fig.add_trace(go.Scatter3d(
        x=[x], y=[y], z=[z],
        mode='text',
        text=[label],
        textfont=dict(size=12, color='black', family='Palatino'),
        textposition='middle center',
        showlegend=False
    ))

# Add edge labels at adjusted positions to avoid triangular areas
edge_labels = {
    '2': (-0.5, -1, 0),     # shifted left from midpoint of (0,0,0) to (0,-2,0)
    '6': (2.2, -2.2, 0),     # shifted up and back from midpoint of (0,0,0) to (6,-2,0)
    '3': (6.5, -2.1, 1.5),       # shifted right from midpoint of (6,-2,0) to (6,-2,3)
    'h': (4, -1, -0)     # shifted up and back from midpoint of (0,0,0) to (6,-2,0)
}

for label, (x, y, z) in edge_labels.items():
    fig.add_trace(go.Scatter3d(
        x=[x], y=[y], z=[z],
        mode='text',
        text=[label],
        textfont=dict(size=14, color='blue', family='Palatino'),
        textposition='middle center',
        showlegend=False
    ))

fig.update_layout(
    scene=dict(
        camera=dict(
            eye=dict(x=-1.5, y=0, z=0.5)
        )
    )
)
fig.show(renderer="notebook")

There are two right angle triangles in the picture above:

One triangle has legs of length 6 and 2, with a hypotenuse of $h$ ; this triangle is shaded $\color{lightblue} \text{light blue}$ above.
Another triangle has legs of length 3 and $h$ , with a hypotenuse of $\left\| \vec w \right\|$ ; this triangle is shaded $\color{#d81b60} \text{dark pink}$ above.

To find $\left\| \vec w \right\|$ , we can use the Pythagorean theorem twice:

h^2 = 6^2 + (-2)^2 = 36 + 4 = 40 \implies h = \sqrt{40}

Then, we can use the Pythagorean theorem again to find $\left\| \vec w \right\|$ :

\left\| \vec w \right\| = \sqrt{h^2 + 3^2} = \sqrt{40 + 9} = 7 = \sqrt{6^2 + (-2)^2 + 3^2}

So, to find $\left\| \vec w \right\|$ , we used the Pythagorean theorem twice, and ended up computing the square root of the sum of the squares of the components of the vector, which is what the definition above states. This argument naturally extends to higher dimensions. We will do this often: build intuition in the dimensions we can visualize (2D, and with the help of interactive graphics, 3D), and then rely on the power of abstraction to extend our understanding to higher dimensions, even when we can’t visualize.

Vector norms satisfy several interesting properties, which we will introduce shortly once we have more context.

Addition and Scalar Multiplication¶

Vectors support two core operations: addition and scalar multiplication. These two operations are core to the study of linear algebra – so much so, that sometimes vectors are defined abstractly as “things that can be added and multiplied by scalars”.

Addition¶

Definition: Vector Addition

Suppose ${\color{orange} \vec u}, {\color{#3d81f6} \vec v} \in \mathbb{R}^n$ , i.e. both are vectors with $n$ components.

Then, the sum of ${\color{orange} \vec u}$ and ${\color{#3d81f6} \vec v}$ is defined as follows:

{\color{orange} \vec u} + {\color{#3d81f6} \vec v} = {\color{orange} \begin{bmatrix} u_1 \\ u_2 \\ \vdots \\ u_n \end{bmatrix}} + {\color{#3d81f6} \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}} = \begin{bmatrix} {\color{orange} u_1} + {\color{#3d81f6} v_1} \\ {\color{orange} u_2} + {\color{#3d81f6} v_2} \\ \vdots \\ {\color{orange} u_n} + {\color{#3d81f6} v_n} \end{bmatrix}

This tells us that vector addition is performed element-wise. This is a term that you’ll encounter quite a bit in the context of writing numpy code, as you’ll see in lab.

Using our examples from earlier, $\color{orange}\vec u = \begin{bmatrix} 3 \\ 1 \end{bmatrix}$ and $\color{#3d81f6}\vec v = \begin{bmatrix} 4 \\ -6 \end{bmatrix}$ , we have that ${\color{orange}\vec u} + {\color{#3d81f6}\vec v} = \begin{bmatrix} 7 \\ -5 \end{bmatrix}$ .

Geometrically, we can arrive at the vector $\begin{bmatrix} 7 \\ -5 \end{bmatrix}$ by drawing ${\color{orange}\vec u}$ at the origin, then placing ${\color{#3d81f6}\vec v}$ at the tip of ${\color{orange}\vec u}$ .

from utils import plot_vectors_non_origin
import numpy as np

fig = plot_vectors_non_origin([(((0, 0), (3, 1)), 'orange', r'$\vec u$'),
                               (((3, 1), (3 + 4, 1 - 6)), '#3d81f6', r'$\vec v$'),
                               (((0, 0), (7, -5)), 'black', r'$\vec u + \vec v$')
                               ]
                               , vdeltax=0.3, vdeltay=0.9)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-1, 6], tickvals=np.arange(-5, 15))
fig.update_yaxes(range=[-8, 4], tickvals=np.arange(-8, 4))
fig.show(scale=3)

Vector addition is commutative, i.e. ${\color{orange}\vec u} + {\color{#3d81f6}\vec v} = {\color{#3d81f6}\vec v} + {\color{orange}\vec u}$ , for any two vectors ${\color{orange}\vec u}, {\color{#3d81f6}\vec v} \in \mathbb{R}^n$ . Algebraically, this should not be a surprise, since ${\color{orange}u_i} + {\color{#3d81f6}v_i} = {\color{#3d81f6}v_i} + {\color{orange}u_i}$ for all $i$ .

Visually, this means that we can instead start with ${\color{#3d81f6}\vec v}$ at the origin and then draw ${\color{orange}\vec u}$ starting from the tip of ${\color{#3d81f6}\vec v}$ , and we should land in the same place.

from utils import plot_vectors
import numpy as np

fig = plot_vectors_non_origin([(((0, 0), (4, -6)), '#3d81f6', r'$\vec v$'),
                               (((4, -6), (4 + 3, -6 + 1)), 'orange', r'$\vec u$'),
                               (((0, 0), (7, -5)), 'black', r'$\vec u + \vec v$')
                               ]
                               , vdeltax=0.3, vdeltay=0.7)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-1, 6], tickvals=np.arange(-5, 15))
fig.update_yaxes(range=[-8, 4], tickvals=np.arange(-8, 4))
fig.show(scale=3)

We cannot, however, add $\vec w = \begin{bmatrix} 6 \\ -2 \\ 3 \end{bmatrix}$ to $\vec u$ , since $\vec u$ and $\vec w$ have different numbers of components.

In Python, we define vectors using numpy arrays, and addition occurs element-wise by default.

Activity 2

In the cell above, try and define $\vec w = \begin{bmatrix} 6 \\ -2 \\ 3 \end{bmatrix}$ as an array and add it to u. What error do you see?

Scalar Multiplication¶

Using our examples from earlier, $\color{#3d81f6}\vec v = \begin{bmatrix} 4 \\ -6 \end{bmatrix}$ as an example, $3 {\color{#3d81f6}\vec v} = \begin{bmatrix} 12 \\ -18 \end{bmatrix}$ . Note that I’ve deliberately defined this operation as scalar multiplication, not just “multiplication” in general, as there’s more nuance to the definition of multiplication in linear algebra.

Visually, a scalar multiple is equivalent to stretching or compressing a vector by a factor of the scalar. If the scalar is negative, the direction of the vector is reversed. Below, $-\frac{2}{3} \color{#3d81f6}\vec v$ points opposite to $\color{#3d81f6}\vec v$ and $3 \color{#3d81f6}\vec v$ .

from utils import plot_vectors
import numpy as np

fig = plot_vectors([((4 * 3, -6 * 3), 'black', r'$3 \vec v$'),
                    ((4 * (-2 / 3), -6 * (-2 / 3)), 'gray', r'$\frac{-2}{3} \vec v$'),
                    ((4, -6), '#3d81f6', r'$\vec v$')], vdeltax=0.3, vdeltay=5)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x", xaxis_range=[-3, 15])
fig.show(scale=3)

An important observation is that $\color{#3d81f6}\vec v$ , $3 \color{#3d81f6}\vec v$ , and $-\frac{2}{3} \color{#3d81f6}\vec v$ all lie on the same line.

Activity 3

Addition and scalar multiplication can be used at the same time, as we’re about to see in the next subsection.

Find the vector $\vec x$ such that:

3\begin{bmatrix} 7 \\ 3 \\ -2 \end{bmatrix} + 4 \vec x = \begin{bmatrix} 9 \\ 4 \\ 2 \end{bmatrix}

Linear Combinations¶

Motivation and Definition¶

The two operations we’ve defined – vector addition and scalar multiplication – are the building blocks of linear algebra, and are often used in conjunction. For example, if we stick with the same vectors $\color{orange} \vec u$ and $\color{#3d81f6} \vec v$ from earlier, what might the vector $3 {\color{orange} \vec u} - \frac{1}{2} {\color{#3d81f6} \vec v}$ look like?

3 {\color{orange} \vec u} - \frac{1}{2} {\color{#3d81f6} \vec v} = 3 {\color{orange} \begin{bmatrix} 3 \\ 1 \end{bmatrix}} - \frac{1}{2} {\color{#3d81f6} \begin{bmatrix} 4 \\ -6 \end{bmatrix}} = \begin{bmatrix} 9 \\ 3 \end{bmatrix} - \begin{bmatrix} 2 \\ -3 \end{bmatrix} = \begin{bmatrix} 7 \\ 6 \end{bmatrix}

from utils import plot_vectors_non_origin
import numpy as np

# Define coefficients (easily changeable)
c1 = 3
c2 = -1/2

# Define original vectors
u = np.array([3, 1])
v = np.array([4, -6])

# Calculate scaled vectors
scaled_u = c1 * u  # -2u
scaled_v = c2 * v  # 1.5v
result = scaled_u + scaled_v  # -2u + 1.5v

# Calculate positions for vector addition visualization
origin = np.array([0, 0])
end_of_scaled_u = origin + scaled_u
end_of_result = origin + result

# Create vector list for plotting
vectors = [
    # Step 1: -2u starting at origin (gray)
    ((tuple(origin), tuple(end_of_scaled_u)), 'gray', f'${c1} \\vec u$'),
    
    # Step 2: 1.5v starting from end of -2u (gray)
    ((tuple(end_of_scaled_u), tuple(end_of_result)), 'gray', f'$-\\frac{1}{2} \\vec v$'),
    
    # Final result: -2u + 1.5v (black)
    ((tuple(origin), tuple(end_of_result)), 'black', f'$3 \\vec u -\\frac{1}{2} \\vec v$'),
    
    # Reference vectors (original u and v)
    ((tuple(origin), tuple(u)), 'orange', r'$\vec u$'),
    ((tuple(origin), tuple(v)), '#3d81f6', r'$\vec v$')
]

fig = plot_vectors_non_origin(vectors, vdeltax=0.3, vdeltay=0.5)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x",
                  xaxis_range=[-5, 15])
fig.show(scale=3)

The vector $\begin{bmatrix} 7 \\ 6 \end{bmatrix}$ , drawn in black above, is a linear combination of $\color{orange}\vec u$ and $\color{#3d81f6}\vec v$ , since it can be written in the form $3{\color{orange}\vec u} - \frac{1}{2}{\color{#3d81f6}\vec v}$ . 3 and $-\frac{1}{2}$ are the scalars that the definition above refers to as $a_1$ and $a_2$ , and we’ve used $\color{orange}\vec u$ and $\color{#3d81f6}\vec v$ in place of $\color{#d81b60}\vec v_1$ and $\color{#d81b60}\vec v_2$ . (I’ve tried to make the definition a bit more general – here, we’re just working with $d = 2$ vectors in $n = 2$ dimensions, but in practice $d$ and $n$ could both be much larger.)

Example in 2D¶

Here’s another linear combination of $\color{orange}\vec u$ and $\color{#3d81f6}\vec v$ , namely $6{\color{orange}\vec u} + 5{\color{#3d81f6}\vec v}$ . Algebraically, this is:

6{\color{orange}\vec u} + 5{\color{#3d81f6}\vec v} = 6{\color{orange}\begin{bmatrix} 3 \\ 1 \end{bmatrix}} + 5{\color{#3d81f6}\begin{bmatrix} 4 \\ -6 \end{bmatrix}} = \begin{bmatrix} 38 \\ -24 \end{bmatrix}

Visually:

from utils import plot_vectors_non_origin
import numpy as np

# Define coefficients (easily changeable)
c1 = 6
c2 = 5

# Define original vectors
u = np.array([3, 1])
v = np.array([4, -6])

# Calculate scaled vectors
scaled_u = c1 * u  # -2u
scaled_v = c2 * v  # 1.5v
result = scaled_u + scaled_v  # -2u + 1.5v

# Calculate positions for vector addition visualization
origin = np.array([0, 0])
end_of_scaled_u = origin + scaled_u
end_of_result = origin + result

# Create vector list for plotting
vectors = [
    # Step 1: -2u starting at origin (gray)
    ((tuple(origin), tuple(end_of_scaled_u)), 'gray', f'${c1} \\vec u$'),
    
    # Step 2: 1.5v starting from end of -2u (gray)
    ((tuple(end_of_scaled_u), tuple(end_of_result)), 'gray', f'${c2} \\vec v$'),
    
    # Final result: -2u + 1.5v (black)
    ((tuple(origin), tuple(end_of_result)), 'black', f'${c1} \\vec u {"+" if c2 >= 0 else ""} {c2} \\vec v$'),
    
    # Reference vectors (original u and v)
    ((tuple(origin), tuple(u)), 'orange', r'$\vec u$'),
    ((tuple(origin), tuple(v)), '#3d81f6', r'$\vec v$')
]

fig = plot_vectors_non_origin(vectors, vdeltax=0.3, vdeltay=0.5)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_layout(xaxis_range=[-1, 40], yaxis_range=[-30, 10])
fig.show(scale=3)

I like thinking of a linear combination as taking “a little bit of the first vector, a little bit of the second vector, etc.” and then adding them all together. (By “little bit”, I mean some amount of, e.g. $6 {\color{orange}\vec u}$ is a little bit of $\color{orange}\vec u$ .) Another useful analogy is to think of the original vectors as “building blocks” that we can use to create new vectors through addition and scalar multiplication.

This idea, of creating new vectors by scaling and adding existing vectors, is so important that it’s essentially what our regression problem boils down to. In the context of our commute times example, imagine $\vec{\text{dt}}$ contains the departure time for each row in our dataset (i.e. the time left in the morning), and $\vec{\text{nc}}$ contains the average number of cars on the road on a particular day. If we want to use these two features in a linear model to predict commute time, our problem boils down to finding the optimal coefficients $w_1$ and $w_2$ in a linear combination of $\vec{\text{dt}}$ and $\vec{\text{nc}}$ that best predicts commute times.

\text{vector of predicted commute times} = w_1 \vec{\text{dt}} + w_2 \vec{\text{nc}}

The Three Questions¶

We’re going to spend a lot of time thinking about linear combinations. Specifically:

Again, just as an example, suppose the two vectors we’re dealing with are our familiar friends:

{\color{orange}\vec u = \begin{bmatrix} 3 \\ 1 \end{bmatrix}}, \quad {\color{#3d81f6}\vec v = \begin{bmatrix} 4 \\ -6 \end{bmatrix}}

These are $d = 2$ vectors in $n = 2$ dimensions. With regards to the Three Questions:

Can we write $\vec b$ as a linear combination of ${\color{orange}\vec u}$ and ${\color{#3d81f6}\vec v}$ ?
If $\vec b = \begin{bmatrix} 7 \\ 6 \end{bmatrix}$ , then the answer to the first question is yes, because we’ve shown that:
$3 {\color{orange}\vec u} - \frac{1}{2} {\color{#3d81f6}\vec v} = \begin{bmatrix} 7 \\ 6 \end{bmatrix}$
Similarly, if $\vec b = \begin{bmatrix} 38 \\ -24 \end{bmatrix}$ , then the answer to the first question is also yes, because we’ve shown that:
$6 {\color{orange}\vec u} + 5 {\color{#3d81f6}\vec v} = \begin{bmatrix} 38 \\ -24 \end{bmatrix}$
If $\vec b$ is some other vector, the answer may be yes or no, for all we know right now.
If so, are the values of the scalars on ${\color{orange}\vec u}$ and ${\color{#3d81f6}\vec v}$ unique?
Not sure! It’s true that $\begin{bmatrix} 7 \\ 6 \end{bmatrix} = 3 {\color{orange}\vec u} - \frac{1}{2} {\color{#3d81f6}\vec v}$ , but for all I know at this point, there could be other scalars $a_1 \neq 3$ and $a_2 \neq -\frac{1}{2}$ such that:
$a_1 {\color{orange}\vec u} + a_2 {\color{#3d81f6}\vec v} = \begin{bmatrix} 7 \\ 6 \end{bmatrix}$
(As it turns out, the answer is that the values 3 and $-\frac{1}{2}$ are unique – you’ll show why this is the case in a following activity.)
What is the shape of the set of all possible linear combinations of ${\color{orange}\vec u}$ and ${\color{#3d81f6}\vec v}$ ?
Also not sure! I know that $\begin{bmatrix} 7 \\ 6 \end{bmatrix}$ and $\begin{bmatrix} 38 \\ -24 \end{bmatrix}$ are both linear combinations of $\color{orange}\vec u$ and $\color{#3d81f6}\vec v$ , and presumably there are many more, but I don’t know what they are.
(It turns out that any vector in $\mathbb{R}^2$ can be written as a linear combination of $\color{orange}\vec u$ and $\color{#3d81f6}\vec v$ ! Again, you’ll show this in an activity.)

We’ll more comprehensively study the “Three Questions” in Chapter 2.4. I just wanted to call them out for you here so that you know where we’re heading.

Activity 4

Let $\vec x = \begin{bmatrix} 3 \\ -1 \\ 2 \end{bmatrix}$ and $\vec y = \begin{bmatrix} 1 \\ 4 \\ 3 \end{bmatrix}$ .

Find scalars $c$ and $d$ such that:

c \vec x + d \vec y = \begin{bmatrix} 9 \\ -16 \\ -1 \end{bmatrix}

Example in 3D¶

As a final example, let’s consider the vectors:

{\color{#d81b60}\vec w = \begin{bmatrix} 12 \\ -4 \\ 6 \end{bmatrix}}, \quad {\color{#004d40}\vec r = \begin{bmatrix} 7 \\ 1 \\ 10 \end{bmatrix}}

These are $d = 2$ vectors, as before, but now in $n = 3$ dimensions. What do some of their linear combinations look like?

from utils import plot_vectors
import numpy as np

w = np.array([12, -4, 6])
r = np.array([7, 1, 10])

# Create vector list for plotting
vectors = [
    (tuple(w), '#d81b60', r'<b>w</b>'),
    (tuple(r), '#004d40', r'<b>r</b>'),
    (tuple(w + 2 * r), 'gray', r'<b>w + 2r</b>'),
    (tuple(2 * w + r), 'gray', r'<b>2w + r</b>'),
    (tuple(- 0.75 * w - 3 * r), 'gray', r'<b>-0.75w - 3r</b>'),
]

fig = plot_vectors(vectors, vdeltax=30)
# fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")

fig.update_layout(
    scene=dict(
        camera=dict(
            eye=dict(x=-2, y=1, z=0.8)
        )
    ),
)

fig.show(config={'displayModeBar': False}, renderer='notebook');

Drag the plot above to look at all five vectors from different angles. You should notice that all of the linear combinations of $\color{#d81b60}\vec w$ and $\color{#004d40}\vec r$ lie on the same plane! We’ll develop more precise terminology to describe this idea, once again, in Chapter 2.4. For now, think of a plane as a flat sheet of paper (more formally, a surface) that extends infinitely in all directions.

Norms, Revisited¶

Earlier in this section, we defined the norm of a vector $\vec v$ as:

\lVert \vec v \rVert = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2} = \sqrt{\sum_{i=1}^n v_i^2}

Now that we know how to add and scale vectors, we should think about how the norm behaves under these operations.

Properties of the Norm¶

The Three Properties

Recall, for $\vec v \in \mathbb{R}^n$ :

\lVert \vec v \rVert = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2} = \sqrt{\sum_{i=1}^n v_i^2}

Then, $\lVert \vec v \rVert$ satisfies:

$\lVert \vec v \rVert \geq 0$ , and $\lVert \vec v \rVert = 0$ if and only if $\vec v$ is the zero vector, $\vec 0 = \begin{bmatrix} 0 \\ 0 \\ \vdots \\ 0 \end{bmatrix}$ .
$\lVert c \vec v \rVert = |c| \lVert \vec v \rVert$ for all $c \in \mathbb{R}$ .
$\lVert \vec u + \vec v \rVert \leq \lVert \vec u \rVert + \lVert \vec v \rVert$ .

Properties 1 and 2 are intuitive enough:

Property 1 states that it’s impossible for a vector to have a negative norm. To calculate the norm a vector, we sum the squares of each of the vector’s components. As long as each component $v_i$ is a real number, then $v_i^2 \geq 0$ , and so $\sum_{i=1}^n v_i^2 \geq 0$ . The square root of a non-negative number is always non-negative, so the norm of a vector is always non-negative. The only case in which $\sum_{i=1}^n v_i^2 = 0$ is when each $v_i = 0$ , so the only vector with a norm of 0 is the zero vector.
Property 2 states that scaling a vector by a scalar scales its norm by the absolute value of the scalar. For instance, it’s saying that both $2 \vec v$ and $-2 \vec v$ should be double the length of $\vec v$ . See, at this point, if you can prove why this is the case.

Activity 5

Prove Property 2 of vector norms, i.e. that:

\lVert c \vec v \rVert = |c| \lVert \vec v \rVert

Start by using the definition of the norm of the vector $c \vec v$ .

Once you’ve given it a shot, read the solution below.

Solution

As I suggested, let’s start by expanding the definition of $\lVert c \vec v \rVert$ .

\begin{aligned} \lVert c \vec v \rVert &= \sqrt{(c v_1)^2 + (c v_2)^2 + \cdots + (c v_n)^2} \\ &= \sqrt{c^2 v_1^2 + c^2 v_2^2 + \cdots + c^2 v_n^2} \\ &= \sqrt{c^2 (v_1^2 + v_2^2 + \cdots + v_n^2)} \\ \end{aligned}

At this point, we’d like to remove the $c^2$ from the square root. The issue is that $c^2$ is a non-negative number, but $c$ could either have been positive or negative. So, we need to take the absolute value of $c$ . (Remember, $\sqrt{x^2} = |x|$ , not $\sqrt{x} = x$ .)

Then:

\begin{aligned} \lVert c \vec v \rVert &= \sqrt{c^2 (v_1^2 + v_2^2 + \cdots + v_n^2)} \\ &= |c| \underbrace{\sqrt{v_1^2 + v_2^2 + \cdots + v_n^2}}_{\text{This is just $\lVert \vec v \rVert$!}} \\ &= |c| \lVert \vec v \rVert \end{aligned}

So, we’ve shown that $\lVert c \vec v \rVert = |c| \lVert \vec v \rVert$ . Even though this fact may seem like it “just should be true”, we need to be able to rigorously justify it, and we’ve now done just that.

The Triangle Inequality¶

Property 3 is a bit more interesting. As a reminder, it states that:

\lVert \vec u + \vec v \rVert \leq \lVert \vec u \rVert + \lVert \vec v \rVert

This is a famous inequality, generally known as the triangle inequality, and it comes up all the time in proofs. Intuitively, it says that the length of a sum of vectors cannot be greater than the sum of the lengths of the individual vectors – or, more philosophically, a sum cannot be more than its parts. It’s called the triangle inequality because it’s a generalization of the fact that in a triangle, the sum of the lengths of any two sides is greater than the length of the third side.

from utils import plot_vectors_non_origin
import numpy as np

fig = plot_vectors_non_origin([(((0, 0), (3, 1)), 'orange', r'$\vec u$'),
                               (((3, 1), (3 + 4, 1 - 6)), '#3d81f6', r'$\vec v$'),
                               (((0, 0), (7, -5)), 'black', r'$\vec u + \vec v$')
                               ]
                               , vdeltax=0.3, vdeltay=0.9)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-1, 6], tickvals=np.arange(-5, 15))
fig.update_yaxes(range=[-8, 4], tickvals=np.arange(-8, 4))
fig.show(config={'displayModeBar': False})

Above:

\lVert {\color{orange}\vec u} \rVert = \sqrt{3^2 + 1^2} = \sqrt{10}

\lVert {\color{#3d81f6}\vec v} \rVert = \sqrt{4^2 + (-6)^2} = \sqrt{52}

\lVert {\color{orange}\vec u} + {\color{#3d81f6}\vec v} \rVert = \sqrt{7^2 + (-5)^2} = \sqrt{74}

And indeed, $\sqrt{74} \approx 8.6$ is less than $\sqrt{10} + \sqrt{52} \approx 10.4$ .

To prove that the triangle inequality holds in general, for any two vectors $\vec u, \vec v \in \mathbb{R}^n$ , we’ll need to wait until Chapter 2.2. We don’t currently have any way to expand the norm $\lVert \vec u + \vec v \rVert$ – but we’ll develop the tools to do so soon. Just keep it in mind for now.

Activity 6

The triangle inequality says that:

\lVert \vec u + \vec v \rVert \leq \lVert \vec u \rVert + \lVert \vec v \rVert

In the example above, we had $\sqrt{74} < \sqrt{10} + \sqrt{52}$ . In other words, there was strict inequality.

Find a pair of vectors $\vec u, \vec v$ (say, in $\mathbb{R}^2$ ) such that the triangle inequality achieves equality, i.e. $\lVert \vec u + \vec v \rVert = \lVert \vec u \rVert + \lVert \vec v \rVert$ .

Unit Vectors and the Norm Ball¶

It’s common to use unit vectors to describe directions. I’ll use the same example as in Activity 1, when this idea was first introduced. Consider the vector $\vec x = \begin{bmatrix} 12 \\ 5 \end{bmatrix}$ . Its norm is $\lVert \vec x \rVert = \sqrt{12^2 + 5^2} = \sqrt{169} = \boxed{13}$ . (You might remember the $(5, 12, 13)$ Pythagorean triple from high school algebra– but that’s not important.)

There are plenty of vectors that point in the same direction as $\vec x$ – any vector $c \vec x$ for $c > 0$ does. (If $c < 0$ , then the vector $c \vec x$ points in the opposite direction of $\vec x$ .)

But among all those, the only one with a norm of 1 is $\frac{1}{\boxed{13}} \vec x$ . Property 2 of the norm tells us this.

from utils import plot_vectors
import numpy as np

x = np.array([12, 5])

fig = plot_vectors([(tuple(x), '#d81b60', r'$\vec x$'),
                    (tuple(x / 13), 'black', r'$\frac{1}{13} \vec x$')], vdeltax=0.6, vdeltay=1)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-3, 15], tickvals=np.arange(-5, 15))
fig.update_yaxes(range=[-2, 6], tickvals=np.arange(-5, 9))
fig.show(scale=3)

In general, if $\vec v$ is any vector, then:

\frac{\vec v}{\lVert \vec v \rVert}

is a unit vector in the same direction as $\vec v$ . Sometimes, we say that $\frac{\vec v}{\lVert \vec v \rVert}$ is a normalized version of $\vec v$ .

Here’s where things get interesting. Let’s visualize a few vectors and their normalized versions:

\begin{aligned} \vec u &= \begin{bmatrix} 3 \\ 1 \end{bmatrix} \implies \frac{\vec u}{\lVert \vec u \rVert} = \begin{bmatrix} \frac{3}{\sqrt{10}} \\ \frac{1}{\sqrt{10}} \end{bmatrix} \\ \vec v &= \begin{bmatrix} -6 \\ -6 \end{bmatrix} \implies \frac{\vec v}{\lVert \vec v \rVert} = \begin{bmatrix} \frac{-1}{\sqrt{2}} \\ \frac{-1}{\sqrt{2}} \end{bmatrix} \\ \vec w &= \begin{bmatrix} 7 \\ -5 \end{bmatrix} \implies \frac{\vec w}{\lVert \vec w \rVert} = \begin{bmatrix} \frac{7}{\sqrt{74}} \\ \frac{-5}{\sqrt{74}} \end{bmatrix} \\ \vec x &= \begin{bmatrix} -12 \\ 5 \end{bmatrix} \implies \frac{\vec x}{\lVert \vec x \rVert} = \begin{bmatrix} \frac{-12}{13} \\ \frac{5}{13} \end{bmatrix} \\ \vec y &= \begin{bmatrix} -1 \\ -6 \end{bmatrix} \implies \frac{\vec y}{\lVert \vec y \rVert} = \begin{bmatrix} \frac{-1}{\sqrt{37}} \\ \frac{-6}{\sqrt{37}} \end{bmatrix} \\ \end{aligned}

from utils import plot_vectors
import numpy as np

fig = plot_vectors([((3 / np.sqrt(10), 1 / np.sqrt(10)), 'black', r'$\frac{\vec u}{\lVert \vec u \rVert}$'),
                    ((-1 / np.sqrt(2), -1 / np.sqrt(2)), 'black', r'$\frac{\vec v}{\lVert \vec v \rVert}$'),
                    ((7 / np.sqrt(74), -5 / np.sqrt(74)), 'black', r'$\frac{\vec w}{\lVert \vec w \rVert}$'),
                    ((-12 / 13, 5 / 13), 'black', r'$\frac{\vec x}{\lVert \vec x \rVert}$'),
                    ((-1 / np.sqrt(37), -6 / np.sqrt(37)), 'black', r'$\frac{\vec y}{\lVert \vec y \rVert}$')
                    ], vdeltax=0, vdeltay=0)
fig.update_layout(width=400, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-1.5, 1.5], tickvals=np.arange(-1.5, 1.5, 0.25))
fig.update_yaxes(range=[-1.5, 1.5], tickvals=np.arange(-1.5, 1.5, 0.25))
fig.show(scale=3)

What do these vectors all have in common, other than being unit vectors? They all lie on a circle of radius 1, centered at $(0, 0)$ !

from utils import plot_vectors
import numpy as np
import plotly.graph_objects as go

# Draw the shaded unit circle
theta = np.linspace(0, 2 * np.pi, 200)
x_circle = np.cos(theta)
y_circle = np.sin(theta)

# Add the vectors on top
fig = plot_vectors([((3 / np.sqrt(10), 1 / np.sqrt(10)), 'black', r'$\frac{\vec u}{\lVert \vec u \rVert}$'),
                    ((-1 / np.sqrt(2), -1 / np.sqrt(2)), 'black', r'$\frac{\vec v}{\lVert \vec v \rVert}$'),
                    ((7 / np.sqrt(74), -5 / np.sqrt(74)), 'black', r'$\frac{\vec w}{\lVert \vec w \rVert}$'),
                    ((-12 / 13, 5 / 13), 'black', r'$\frac{\vec x}{\lVert \vec x \rVert}$'),
                    ((-1 / np.sqrt(37), -6 / np.sqrt(37)), 'black', r'$\frac{\vec y}{\lVert \vec y \rVert}$')
                    ], vdeltax=0, vdeltay=0)

# Add dotted circle outline
fig.add_trace(go.Scatter(
    x=x_circle,
    y=y_circle,
    mode="lines",
    line=dict(color="gray", dash="dash", width=3),
    hoverinfo='skip',
    showlegend=False
))

fig.update_layout(width=400, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-1.5, 1.5], tickvals=np.arange(-1.5, 1.5, 0.5))
fig.update_yaxes(range=[-1.5, 1.5], tickvals=np.arange(-1.5, 1.5, 0.5))
fig.show(scale=3)

The circle shown above is called the norm ball of radius 1 in $\mathbb{R}^2$ . It shows the set of all vectors $\vec v \in \mathbb{R}^2$ such that $\lVert \vec v \rVert = 1$ . Using set notation, we might say:

\{\vec v : \lVert \vec v \rVert = 1, \vec v \in \mathbb{R}^2\}

That this looks like a circle is no coincidence. The condition $\lVert \vec v \rVert = 1$ is equivalent to $\sqrt{v_1^2 + v_2^2} = 1$ . Squaring both sides, we get $v_1^2 + v_2^2 = 1$ . This is the equation of a circle with radius 1 centered at the origin.

In $\mathbb{R}^3$ , the norm ball of radius 1 is a sphere, and in general, in $\mathbb{R}^n$ , the norm ball of radius 1 is an $n$ -dimensional sphere.

Activity 7

Find the unit vector that points in the same direction as $\vec z = \begin{bmatrix} 4 \\ 0 \\ -3 \\ -3 \end{bmatrix}$ .

Clearly write out all four of its components.

Other Norms¶

So far, we’ve only discussed one “norm” of a vector, sometimes called the $L_{\color{orange}2}$ norm or Euclidean norm. In general, if $\vec v \in \mathbb{R}^n$ is one vector, $\vec v = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}$ , then its norm is:

\lVert \vec v \rVert = \sqrt{v_1^{\color{orange}2} + v_2^{\color{orange}2} + \cdots + v_n^{\color{orange}2}} = \sqrt{\sum_{i=1}^n v_i^{\color{orange}2}}

This is, by far, the most common and most relevant norm, and in many linear algebra classes, it’s the only norm you’ll see. But in machine learning, a few other norms are relevant, too, so I’ll briefly discuss them here.

The $L_1$ or Manhattan norm of $\vec v$ is:
$\lVert \vec v \rVert_1 = |v_1| + |v_2| + \cdots + |v_n| = \sum_{i=1}^n |v_i|$
It’s called the Manhattan norm because it’s the distance you would travel if you walked from the origin to $\vec v$ in a grid of streets, where you can only move horizontally or vertically.
The $L_\infty$ or maximum norm of $\vec v$ is:
$\lVert \vec v \rVert_\infty = \max_{i} |v_i|$
This is largest absolute value of any component of $\vec v$ .
For any $p \geq 1$ , the $L_p$ norm of $\vec v$ is:
$\lVert \vec v \rVert_p = \left( \sum_{i=1}^n |v_i|^p \right)^{\frac{1}{p}}$
Note that when $p = 2$ , this is the same as the $L_2$ norm. For other values of $p$ , this is a generalization. Something to think about: why is there an absolute value in the definition?

All of these norms measure the length of a vector, but in different ways. This might ring a bell: we saw very similar tradeoffs between squared and absolute losses in Chapter 1.

Believe it or not, all three of these norms satisfy the same “Three Properties” we discussed earlier.

Back to $\vec x = \begin{bmatrix} 12 \\ 5 \end{bmatrix}$ . What are the $L_2$ , $L_1$ , and $L_\infty$ norms of $\vec x$ ?

from utils import plot_vectors
import numpy as np

x = np.array([12, 5])

fig = plot_vectors([(tuple(x), '#d81b60', r'$\vec x$')], vdeltax=0.6, vdeltay=1)
fig.update_layout(width=500, height=400, yaxis_scaleanchor="x")
fig.update_xaxes(range=[-3, 15], tickvals=np.arange(-5, 15))
fig.update_yaxes(range=[-2, 6], tickvals=np.arange(-5, 9))
fig.show(scale=3)

Here:

$\lVert \vec x \rVert_2 = \sqrt{12^2 + 5^2} = \sqrt{144 + 25} = \sqrt{169} = 13$
$\lVert \vec x \rVert_1 = |12| + |5| = 12 + 5 = 17$
$\lVert \vec x \rVert_\infty = \max(|12|, |5|) = 12$

Let’s revisit the idea of a norm ball. Using the standard $L_2$ norm, the norm ball in $\mathbb{R}^2$ is a circle. What does the norm ball look like for the $L_1$ and $L_\infty$ norms? Or $L_p$ with an arbitrary $p$ ?

import numpy as np
import plotly.graph_objects as go
from plotly.subplots import make_subplots

theta = np.linspace(0, 2 * np.pi, 400)

# L1 norm ball (diamond)
x_l1 = np.cos(theta) / (np.abs(np.cos(theta)) + np.abs(np.sin(theta)))
y_l1 = np.sin(theta) / (np.abs(np.cos(theta)) + np.abs(np.sin(theta)))

# L1.3 norm ball
p13 = 1.3
x_l13 = np.cos(theta)
y_l13 = np.sin(theta)
norm_l13 = (np.abs(x_l13) ** p13 + np.abs(y_l13) ** p13) ** (1/p13)
x_l13 = x_l13 / norm_l13
y_l13 = y_l13 / norm_l13

# L2 norm ball (unit circle)
x_l2 = np.cos(theta)
y_l2 = np.sin(theta)

# L-infinity norm ball (square)
x_linf = np.empty_like(theta)
y_linf = np.empty_like(theta)
for i, t in enumerate(theta):
    c, s = np.cos(t), np.sin(t)
    if np.abs(c) > np.abs(s):
        x_linf[i] = np.sign(c)
        y_linf[i] = np.tan(t) * np.sign(c)
    else:
        y_linf[i] = np.sign(s)
        x_linf[i] = 1/np.tan(t) * np.sign(s)
    norm_inf = max(abs(x_linf[i]), abs(y_linf[i]))
    x_linf[i] /= norm_inf
    y_linf[i] /= norm_inf

fig = make_subplots(
    rows=2, cols=2,
    subplot_titles=[r"$L_1 \text{ norm ball}$", r"$L_{1.3} \text{ norm ball}$", r"$L_2 \text{ norm ball}$", r"$L_\infty \text{ norm ball}$"],
    horizontal_spacing=0.08, vertical_spacing=0.13
)

line_style = dict(color="#888", dash="dash", width=3)

fig.add_trace(go.Scatter(
    x=x_l1, y=y_l1, mode="lines",
    line=line_style, hoverinfo='skip', showlegend=False
), row=1, col=1)

fig.add_trace(go.Scatter(
    x=x_l13, y=y_l13, mode="lines",
    line=line_style, hoverinfo='skip', showlegend=False
), row=1, col=2)

fig.add_trace(go.Scatter(
    x=x_l2, y=y_l2, mode="lines",
    line=line_style, hoverinfo='skip', showlegend=False
), row=2, col=1)

fig.add_trace(go.Scatter(
    x=x_linf, y=y_linf, mode="lines",
    line=line_style, hoverinfo='skip', showlegend=False
), row=2, col=2)

for r in range(1, 3):
    for c in range(1, 3):
        fig.update_xaxes(
            range=[-1.5, 1.5], tickvals=np.arange(-1.5, 1.6, 0.5),
            row=r, col=c, showgrid=True, zeroline=True, gridcolor="#f0f0f0", zerolinecolor="#808080",
            showticklabels=(False if r * c > 1 else True), title=None
        )
        fig.update_yaxes(
            range=[-1.5, 1.5], tickvals=np.arange(-1.5, 1.6, 0.5),
            row=r, col=c, showgrid=True, zeroline=True, gridcolor="#f0f0f0", zerolinecolor="#808080",
            scaleanchor=f"x{2*(r-1)+c}", showticklabels=(False if r * c > 1 else True), title=None
        )

fig.update_layout(
    width=700, height=600,
    plot_bgcolor="white",
    paper_bgcolor="white",
    font=dict(family="Palatino", size=16, color="#222"),
    margin=dict(l=10, r=10, t=40, b=10),
)

fig.show(scale=3)

Most notably, the $L_1$ norm ball looks like a diamond, and the $L_\infty$ norm ball looks like a square. The $L_{1.3}$ norm ball looks like a diamond with rounded corners. This is not the last you’ll see of these norm balls – in particular, in future machine learning courses, you’ll see them again in the context of regularization, which is a technique for preventing overfitting in our models.

Activity 8

In the case of $\vec x = \begin{bmatrix} 12 \\ 5 \end{bmatrix}$ , the $L_1$ norm ( $12 + 5 = 17$ ) is greater than the $L_2$ norm ( $\sqrt{12^2 + 5^2} = \sqrt{169} = 13$ ), i.e. $\lVert \vec x \rVert_1 > \lVert \vec x \rVert_2$ .

Find a vector $\vec y \in \mathbb{R}^2$ such that $\lVert \vec y \rVert_1 = \lVert \vec y \rVert_2$ .
Try and find a vector $\vec z \in \mathbb{R}^2$ such that $\lVert \vec z \rVert_1 < \lVert \vec z \rVert_2$ . What do you encounter?
Prove that $\lVert \vec x \rVert_2 \leq \sqrt{n} \lVert \vec x \rVert_\infty$ for any vector $\vec x \in \mathbb{R}^n$ . (Hint: Start with the definition of the $L_2$ norm of $\vec x$ , square it, and try and compare each element in the sum to the largest element in the vector.)

`np.linalg.norm` and Vectorization¶

It’s been a while since we’ve experimented with numpy. A few things:

As we’ve seen, arrays can be added element-wise by default.
Arrays can also be multiplied by scalars out-of-the-box, meaning that linear combinations of arrays (vectors) are easy to compute. The above two facts mean that array operations are vectorized: they are applied to each element of the array in parallel, without needing to use a for-loop.
To compute the ( $L_2$ ) norm of an array (vector), we can use np.linalg.norm.

Suppose you didn’t know about np.linalg.norm. There’s another way to compute the norm of an array (vector), that doesn’t involve a for-loop. Follow the activity to discover it.

Activity 9

In the cell above:

Write u ** 2. Squaring a vector is not an operation we’ve discussed (and isn’t an operation that exists in math), but numpy gives you back another array. What does this array contain?
Using np.sum and the new array you just created, find the norm of u without using np.linalg.norm.
Find the norm of 3 * u - 0.5 * v using the same technique, and make sure you get the same result as was already displayed for you.

In general, we’ll want to avoid Python for-loops in our code when there are numpy-native alternatives, as these numpy functions are optimized to use C (the programming language) under the hood for speed and memory efficiency.

Math for ML Course Notes

Chapter 2: Vectors and Matrices