5.2. Special Matrices - EECS 245 Course Notes

There’s an important operation on matrices that we haven’t discussed yet.

Definition¶

To illustrate, let’s start with our familiar matrix $A$ :

A = \begin{bmatrix} 3 & {\color{#3d81f6} \mathbf{1}} & 4 \\ 2 & {\color{#3d81f6} \mathbf{1}} & 9 \\ 0 & {\color{#3d81f6} \mathbf{-1}} & 0 \\ 2 & {\color{#3d81f6} \mathbf{-2}} & 0 \end{bmatrix}

The transpose of $A$ is:

A^T = \begin{bmatrix} 3 & 2 & 0 & 2 \\ {\color{#3d81f6} \mathbf{1}} & {\color{#3d81f6} \mathbf{1}} & {\color{#3d81f6} \mathbf{-1}} & {\color{#3d81f6} \mathbf{-2}} \\ 4 & 9 & 0 & 0 \end{bmatrix}

Note that $A \in \mathbb{R}^{4 \times 3}$ and $A^T \in \mathbb{R}^{3 \times 4}$ .

Why would we ever need to do this? To illustrate, suppose $\vec u = \begin{bmatrix} u_1 \\ u_2 \\ u_3 \\ u_4 \end{bmatrix}$ , and that we’d like to compute the product $A^T \vec u$ . (Note that $\vec u$ must be in $\mathbb{R}^4$ in order for $A^T \vec u$ to be defined, unlike $\vec x \in \mathbb{R}^3$ in the product $A \vec x$ ). Then:

\begin{align*}A^T \vec u &= \begin{bmatrix} 3 & 2 & 0 & 2 \\ 1 & 1 & -1 & -2 \\ 4 & 9 & 0 & 0 \end{bmatrix} \begin{bmatrix} {\color{orange} \mathbf{u_1}} \\ {\color{orange} \mathbf{u_2}} \\ {\color{orange} \mathbf{u_3}} \\ {\color{orange} \mathbf{u_4}} \end{bmatrix} \\ &= {\color{orange} \mathbf{u_1}} \begin{bmatrix} 3 \\ 1 \\ 4 \end{bmatrix} + {\color{orange} \mathbf{u_2}} \begin{bmatrix} 2 \\ 1 \\ 9 \end{bmatrix} + {\color{orange} \mathbf{u_3}} \begin{bmatrix} 0 \\ -1 \\ 0 \end{bmatrix} + {\color{orange} \mathbf{u_4}} \begin{bmatrix} 2 \\ -2 \\ 0 \end{bmatrix} \end{align*}

This is a linear combination of the rows of $A$ , where the weights are the components of $\vec u$ . Remember, the standard product $A \vec x$ is a linear combination of the columns of $A$ , so the transpose helps us if we want to compute a linear combination of the rows of $A$ . (Equivalently, it helps us if we want to compute the dot product of the columns of $A$ with $\vec u$ – see the “Two Pictures” note from earlier in this chapter.)

The transpose also gives us another way of expressing the dot product of two vectors. If $\color{orange} \vec u$ and $\color{#3d81f6} \vec v$ are two vectors in $\mathbb{R}^n$ , then $\color{orange} \vec u^T$ is a row vector with 1 row and $n$ columns. Multiplying $\color{orange} \vec u^T$ by $\color{#3d81f6} \vec v$ results in a $1 \times 1$ matrix, which is just the scalar ${\color{orange} \vec u} \cdot {\color{#3d81f6} \vec v}$ .

\vec {\color{orange}u}^T \vec{\color{#3d81f6}v} = \begin{bmatrix} {\color{orange}u_1} & {\color{orange}u_2} & \ldots & {\color{orange}u_n} \end{bmatrix} \begin{bmatrix}{\color{#3d81f6}v_1} \\{\color{#3d81f6}v_2} \\ \vdots \\{\color{#3d81f6}v_n} \end{bmatrix} = {\color{orange}u_1}{\color{#3d81f6}v_1} + {\color{orange}u_2}{\color{#3d81f6}v_2} + \ldots + {\color{orange}u_n}{\color{#3d81f6}v_n} = \vec {\color{orange}u} \cdot \vec{\color{#3d81f6}v} = \vec{\color{#3d81f6}v} \cdot \vec {\color{orange}u} = \vec{\color{#3d81f6}v}^T \vec {\color{orange}u}

The benefit of using the transpose to express the dot product is that it allows us to write the dot product of two vectors in terms of matrix multiplication, rather than being an entirely different type of operation. (In fact, as we’ve seen here, matrix multiplication is just a generalization of the dot product.)

There are other uses for the transpose, too, so it’s a useful tool to have in your toolbox.

Properties¶

The first three properties are relatively straightforward. The last property is a bit more subtle. Try and reason as to why it’s true on your own, then peek into the box below to verify your reasoning and to see an example.

Why is

(AB)^T = B^T A^T

Let’s start with just $(AB)^T$ and reason our way from there. Define $C = (AB)^T$ . $C$ is presumably the product of two matrices $X$ and $Y$ , we just don’t know what $X$ and $Y$ are. By the definition of matrix multiplication, we know that $C_{ij}$ is the dot product of the $i$ th row of $X$ and the $j$ th column of $Y$ . How can we express the product $C = XY$ in terms of $A$ and $B$ ?

Let’s work backwards. Since $C_{ij} = (AB)^T_{ij} = (AB)_{ji}$ by the definition of the transpose, we know that:

C_{ij} = (AB)_{ji} = \left( \text{row } j \text{ of } A \right) \cdot \left( \text{column } i \text{ of } B \right) = \left( \text{column } i \text{ of } B \right) \cdot \left( \text{row } j \text{ of } A \right)

This is a little backwards relative to the definition of matrix multiplication, which says that:

C_{ij} = (XY)_{ij} = \left( \text{row } i \text{ of } X \right) \cdot \left( \text{column } j \text{ of } Y \right)

In order for the two definitions of $C_{ij}$ to be consistent, we must have:

\left( \text{column } i \text{ of } B \right) \cdot \left( \text{row } j \text{ of } A \right) = \left( \text{row } i \text{ of } X \right) \cdot \left( \text{column } j \text{ of } Y \right)

Row $i$ of $X$ is the same as column $i$ of $B$ , if $X = B^T$ .
Column $j$ of $Y$ is the same as row $j$ of $A$ , if $Y = A^T$ .

Putting this together, we have:

C = (AB)^T = B^T A^T

as we hoped!

To make things concrete, let’s consider two new matrices $A$ and $B$ :

A = \begin{bmatrix} 0 & -1 \\ 4 & 2 \\ 3 & 9 \\ 0 & 1 \end{bmatrix} \quad B = \begin{bmatrix} 1 & 2 & 3 \\ -1 & -2 & 4 \end{bmatrix}

Then,

AB = \begin{bmatrix} 0 & -1 \\ 4 & 2 \\ {\color{#3d81f6} \mathbf{3}} & {\color{#3d81f6} \mathbf{9}} \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 2 & {\color{#3d81f6} \mathbf{3}} \\ -1 & -2 & {\color{#3d81f6} \mathbf{4}} \end{bmatrix} = \begin{bmatrix} 1 & 2 & -4 \\ 2 & 4 & 20 \\ -6 & -12 & \boxed{{\color{#3d81f6} \mathbf{45}}} \\ -1 & -2 & 4 \end{bmatrix}

And:

B^T A^T = \begin{bmatrix} 1 & -1 \\ 2 & -2 \\ {\color{#3d81f6} \mathbf{3}} & {\color{#3d81f6} \mathbf{4}} \end{bmatrix} \begin{bmatrix} 0 & 4 & {\color{#3d81f6} \mathbf{3}} & 0 \\ -1 & 2 & {\color{#3d81f6} \mathbf{9}} & 1 \end{bmatrix} = \begin{bmatrix} 1 & 2 & -6 & -1 \\ 2 & 4 & -12 & -2 \\ -4 & 20 & \boxed{{\color{#3d81f6} \mathbf{45}}} & 4 \end{bmatrix}

$B^T A^T$ is the transpose of $AB$ . Both $AB$ and $B^T A^T$ have 12 elements, both computed using the same 12 dot products.

The fact that $(AB)^T = B^T A^T$ comes in handy when finding the norm of a matrix-vector product. If $A$ is an $n \times d$ matrix and $\vec x \in \mathbb{R}^d$ , then:

\lVert A \vec x \rVert^2 = (A \vec x)^T (A \vec x) = \vec x^T A^T A \vec x

As we’ll soon see, some matrices $A$ have special properties that make this computation particularly easy.

TODO: Add discussion of symmetric, triangular, diagonal, orthogonal, and other special matrices.

We now introduce the transpose and other special matrix types that will keep appearing.

In numpy, the T attribute is used to compute the transpose of a 2D array.

Activity 1

Activity 1.1

In the cell above:

Define x to be an array corresponding to the vector $\vec x = \begin{bmatrix} 1 \\ 0 \\ 3 \end{bmatrix}$ .
Find the norm of the product $A \vec x$ using np.linalg.norm.
Find the norm of the product $A \vec x$ using the fact that $\lVert A \vec x \rVert^2 = \vec x^T A^T A \vec x$ , and verify that you get the same answer.

Activity 1.2

Suppose $M \in \mathbb{R}^{n \times d}$ is a matrix, $\vec{v} \in \mathbb{R}^d$ is a vector, and $s \in \mathbb{R}$ is a scalar.

Determine whether each of the following quantities is a matrix, vector, scalar, or undefined. If the result is a matrix or vector, determine its dimensions.

$M\vec{v}$
$\vec{v} M$
$\vec{v}^2$
$M^TM$
$MM^T$
$\vec{v}^T M \vec{v}$
$(sM\vec{v}) \cdot (sM\vec{v})$
$(s \vec{v}^T M^T)^T$
$\vec{v}^T M^T M \vec{v}$
$\vec{v}\vec{v}^T + M^TM$
$\frac{M \vec{v}}{\lVert \vec{v} \rVert} + (\vec{v}^T M^T M \vec{v}) M \vec{v}$

Activity 1.3

Let $A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \\ -1 & 1 \end{bmatrix}$ , $B = \begin{bmatrix} 1 & 0 & 2 \\ 2 & 1 & 3 \end{bmatrix}$ , and $C = \begin{bmatrix} 1 & 0 & 2 & -1 \\ 0 & 1 & 1 & 1 \\ 1 & 1 & 0 & -1 \end{bmatrix}$ .

Compute $AB$ , then multiply the result by $C$ .
Compute $A$ , then multiply the result by $BC$ . Do you get the same result as above? If so, what property of matrix multiplication guarantees this?
Determine a formula for $(ABC)^T$ , and verify that your result works. (Hint: Start with the fact that $(AB)^T = B^T A^T$ .)

The Identity Matrix¶

Saying “the identity matrix” is a bit ambiguous, as there are infinitely many identity matrices – there’s a $1 \times 1$ identity matrix, a $2 \times 2$ identity matrix, a $3 \times 3$ identity matrix, and so on. Often, the dimension of the identity matrix is implied by context, and if not, we might provide it as a subscript, e.g. $I_n$ for the $n \times n$ identity matrix.

Why is the identity matrix defined this way? It’s the matrix equivalent of the number 1 in scalar multiplication, also known as the multiplicative identity. If $c$ is a scalar, then $c \cdot 1 = c$ and $1 \cdot c = c$ . (0 is known as the additive identity in scalar multiplication.)

Similarly, if $A$ is square $n \times n$ matrix and $\vec x \in \mathbb{R}^n$ is a vector, then the $n \times n$ identity matrix $I$ is the unique matrix that satisfies:

$I \vec x = \vec x$ for all $\vec x \in \mathbb{R}^n$ .
$I A = A I = A$ for all $A \in \mathbb{R}^{n \times n}$ .

A good exercise is to verify that the identity matrix satisfies these properties.

Activity 2

Let $X = \begin{bmatrix} 1 & -2 \\ -1 & 3 \\ 2 & 0 \\ 0 & -1 \\ 3 & 2 \end{bmatrix}$ .

Compute $X^TX$ .
Then, compute the transpose of $X^TX$ . What do you notice? ( $X^TX$ is called a symmetric matrix.)
Compute $X^TX + \frac{1}{2} I$ . We’ll use matrices of the form $X^TX + \lambda I$ in Chapter 5.

Preview: Transformations¶

This section was relatively mechanical, and I didn’t spend much time explaining why we’d multiply two matrices (or a matrix and a vector) together. More context for this operation will come throughout the rest of the chapter.

To conclude, I want to show you some of the magic behind matrix multiplication.

Consider the relatively innocent looking $2 \times 2$ matrix

A = \begin{bmatrix} \frac{\sqrt{3}}{2} & -\frac{1}{2} \\[6pt] \frac{1}{2} & \frac{\sqrt{3}}{2} \end{bmatrix}

Below, you’ll see that I’ve drawn out six vectors in $\mathbb{R}^2$ .

$\color{orange} \vec u = \begin{bmatrix} 3 \\ 2 \end{bmatrix}$ and $A \color{orange} \vec u$
$\color{#3d81f6} \vec v = \begin{bmatrix} 2 \\ -2 \end{bmatrix}$ and $A \color{#3d81f6} \vec v$
$\color{#d81a60} \vec w = \begin{bmatrix} -5 \\ -4 \end{bmatrix}$ and $A \color{#d81a60} \vec w$

What do you notice about the vectors $A \color{orange} \vec u$ , $A \color{#3d81f6} \vec v$ , and $A \color{#d81a60} \vec w$ ? and how they relate to $\color{orange} \vec u$ , $\color{#3d81f6} \vec v$ , and $\color{#d81a60} \vec w$ ?

from utils import plot_vectors
import numpy as np

A = np.array([[np.cos(np.pi/6), -np.sin(np.pi/6)], [np.sin(np.pi/6), np.cos(np.pi/6)]])
u = np.array([3, 2])
v = np.array([2, -2])
w = np.array([-5, -4])

Au = A @ u
Av = A @ v
Aw = A @ w

fig = plot_vectors([(tuple(u), 'orange', r'$\vec u$'),
              (tuple(v), '#3d81f6', r'$\vec v$'),
              (tuple(w), '#d81a60', r'$\vec w$'),
              (tuple(Au), 'orange', r'$A \vec u$'),
              (tuple(Av), '#3d81f6', r'$A \vec v$'),
              (tuple(Aw), '#d81a60', r'$A \vec w$')], vdeltay=0.3)

fig.show(scale=3, renderer='png')

$A$ is called a rotation matrix, since it rotates vectors by a certain angle (in this case, $\frac{\pi}{6}$ radians, or $30^\circ$ ). Rotations are a type of linear transformation.

Not all matrices are rotation matrices; there exist plenty of different types of linear transformations, like reflections, sheers, and projections (which sound familiar). These will all become familiar in Chapter 6.1. All I wanted to show you for now is that matrix multiplication may look like a bunch of random number crunching, but there’s a lot of meaning baked in.