Matrix Calculus but with Tensors

Multilinear Derivatives

Intro

Occasionally we need to do some calculus on vectors. The way these derivatives shake out often resemble usual 1-D rules, but the precise definition can be hard to get right.

We reframe vector and matrix calculus problems with the language of tensors, which we argue makes these types of problems easy to solve.

Recommended Background

Familiarity with vectors and partial derivatives is recommended. Previous exposure to tensors is not required.

We will use bra-ket notation. You can take a minute to become familiar, but it’s just a way of writing vectors.

Motivation

Let’s look at a derivative that we may want to know.

Projection of a vector, y, onto a unit vector, x, is given by

\(|x\rangle\langle x|y\rangle\)

We may want to know what happens when we change the vector, x, slightly. Well derivatives give you approximate solutions for small changes.

\(\left(|x\rangle+\varepsilon|t\rangle\right)\left(\langle x|+\varepsilon\langle t|\right)|y\rangle \approx |x\rangle\langle x|y\rangle + \varepsilon D|t\rangle\)

Where D is given by

\(D = \frac{\partial |x\rangle\langle x|y\rangle}{\partial \langle x|} = \langle x|y\rangle I + |x\rangle\langle y|\)

Which means when you move x in the direction of t, you will move the projection of y to x in the direction of t, by only by the magnitude of the projection. But you will also move in the direction of x with the magnitude of t projected onto y.

This is what’s beautiful about calculus; it can give you surprising answers, but if you follow the math, you can figure out exactly where the answer came from.

The only problem is: How do we do the math?? If you’re like me before I learned about tensors, you probably see the derivative problem and do one of the following:

1. Panic

2. Look up the answer in a table

3. Write out all the terms and do way too much math to try to figure out what the book just flew right over.

We will learn that with the language of tensors, derivatives on vectors becomes as simple as derivatives on scalars.

Way Too Much Math

Let’s see how the third approach would work out.

The projection vector has i scalars in a column, so let’s work out what the i-th term looks like.

\(\left( |x\rangle\langle x| y\rangle\right)_i = x_i\sum_{k}x_ky_k\)

Now the (i, j)-th entry in the matrix, D, should be the i-th term with derivative taken with respect to j.

\(\frac{x_i\sum_{k}x_ky_k}{\partial x_j} = \left\{\begin{array}{lcr} \sum_kx_ky_k + x_iy_i & ; &\text{if }i=j\\ x_iy_i &;&\text{if }i\ne j \end{array}\right. = x_iy_i + \left\{\begin{array}{lcr} \sum_kx_ky_k & ; &\text{if }i=j\\ 0 &;&\text{if }i\ne j \end{array}\right.\)

If you squint, you can see that these two terms are the same two terms of:

\(\left(x_iy_i\right)_{i,j} + \left(\left\{\begin{array}{lcr} \sum_kx_ky_k & ; &\text{if }i=j\\ 0 &;&\text{if }i\ne j \end{array}\right.\right)_{i,j} = |x\rangle\langle y| + I \sum_kx_ky_k = |x\rangle\langle y| + \langle x|y\rangle I\)

So we get the right answer, but calculus shouldn’t be this hard.

Shape Matters

You sorta just wanna use product rule. You can even see that the derivative we got is the result of product rule. But the application of product rule gets real sketchy. The convention¹ usually used for matrices is that:

\(\frac{\partial |x\rangle}{\partial\langle x|} = \frac{\partial \langle x|}{\partial\langle x|} = I\)

So now if you use product rule, you get

\(\frac{\partial |x\rangle\langle x|y\rangle}{\partial \langle x|} = I\langle x|y\rangle+|x\rangle |y\rangle\)

And you sorta know that two column vectors can’t be multiplied so something has to flip. And if you don’t know what happened, you might end up with the answer as:

\(\frac{\partial |x\rangle\langle x|y\rangle}{\partial \langle x|}=2I\langle x|y\rangle\)

The convention for vector derivatives isn’t productive. In fact we know that derivatives with respect to scalars has different shapes for different inputs:

\(\frac{\partial |x(t)\rangle}{\partial t} = \left|x'(t)\right\rangle\quad\quad \frac{\partial \langle x(t)|}{\partial t} = \left\langle x'(t)\right|\)

Why should shape not matter when the partial derivate contains vectors for both parts? The convention gets even more strange when the vectors are of different lengths. Is the derivative of a m-by-1 with respect to an n-by-1 an m-by-n matrix or an n-by-m matrix??

Let’s establish some rules we can all agree on:

• Derivative of a vector by a scalar should be a vector

  • mx1 by 1x1 = mx1

• A covector by a scalar should be a covector

  • 1xn by 1x1 = 1xn

• A scalar by a vector/covector should be a vector/covector

  • 1x1 by mx1 = mx1

  • 1x1 by 1xn = 1xn

• A vector by a covector should be a matrix, and vise versa

  • mx1 by 1xn = mxn

  • 1xn by mx1 = mxn

But let’s do pattern matching. What should be the dimension of the derivative of a vector by a vector (mx1 by mx1)? It should have 1 column, but for rows it should have something like mxm rows. Let’s write this as (m,m)x1. This is allowable, and it’s called a tensor.²³

The reason that the convention avoids talking about tensors is that they’re abstract and complicated. The math they usually show up in is advanced. And the standard notation can be sometimes confusing. But it doesn’t have to be complicated. We know that for the problem above, the solution is a matrix, so the tensors will go away. We need to know some basic rules about these to get them to work for us.

Defining the Tensor

Tensors are more about notation than new insights, and notation can quickly become cumbersome. To avoid getting to abstract, we’re going to talk about exactly one tensor and how to use it.

Before that though, let’s revisit some notation for the identity matrix.

\(\frac{\partial |x\rangle}{\partial\langle x|} = I = \sum_i|e_i\rangle\langle e_i| = |e\rangle\langle e|\)

The first equality, we’ll take as a rule of vector derivatives. The second equality uses the convention that e_i is a vector with 0 at all indices except i which has value 1. (You can check that if you multiply and add, this equals the identity matrix.) The last equality makes no sense, but it’s a notation that we adopt for short-hand.

Now then, we’ll take as a second rule of vector derivatives:

\(\frac{\partial |x\rangle}{\partial |x\rangle}= \sum_i|e_i\rangle\otimes|e_i\rangle = |e\rangle\otimes|e\rangle=|ee\rangle\)

The first two equalities are reminiscent of the previous derivative shown. The ⊗ symbol is called the “tensor” product and it just generalizes⁴ matrix/vector multiplication for the cases where the shapes don’t line up. The last equality is again just a different syntax.

The last syntax emphasizes that this is just a thing, a symbol, like i for the imaginary number. And once we’ve named it, we can study its properties. But since it’s the identity tensor, it’s properties are pretty simple.

To study it, let’s use the constant rule (you can pull constants out of derivatives).

\(\frac{\partial \langle a|x\rangle}{\partial |x\rangle}=\langle a|\frac{\partial |x\rangle}{\partial |x\rangle} = \langle a|ee\rangle\)

However the numerator is a scalar, and we know that the derivative of a scalar with respect to a vector equals the derivative of the scalar with respect to each term in the vector; the result should be the shape as the vector.

\(\frac{\partial \langle a|x\rangle}{\partial |x\rangle}=\frac{\partial \sum_i a_ix_i}{\partial |x\rangle}=\left(\begin{array}{c}a_1\\ \vdots\\ a_m\end{array}\right)=|a\rangle\)

Putting these together we get:

\(\langle a|ee\rangle = |a\rangle\)

Contrast that with how the identity matrix applies to a:

\(\langle a|e\rangle\langle e| = \langle a|\)

The identity matrix let’s a vector keep it’s shape, but the identity tensor changes the shape. I think of this like: The first e reduces a to a scalar, and the second gives it shape.⁵

We can now see how the earlier product rule gets fixed to give a correct answer.

\(\frac{\partial |x\rangle\langle x|y\rangle}{\partial \langle x|} = I\langle x|y\rangle + |x\rangle\langle ee|y\rangle = I\langle x|y\rangle + |x\rangle\langle y|\)

While tensor may generally be complicated and scary, this one identity tensor and property can get you most of what you need.

Solving Another Problem

Let’s say you encounter and apply a derivative:

\(\frac{\partial \langle x|A|x\rangle}{\partial |x\rangle} = 2A|x\rangle\)

You feel proud of yourself that you remembered a derivative rule. But oops; you forgot that this rule only works when A is symmetric. Let’s do the math.

\(\frac{\partial \langle x|A|x\rangle}{\partial |x\rangle} = IA|x\rangle + \left(\langle x|A\right)|ee\rangle = A|x\rangle + \left(\langle x|A\right)^\top=A|x\rangle+A^\top|x\rangle\)

Easy right?

Conclusions and Other Things to Think About

We saw how the introduction of an identity tensor can make vector derivatives easier by making sense of product rule. Here are some others you can try:

\(\frac{\partial\langle x|x\rangle}{\partial |x\rangle}\)

\( \frac{\partial\langle a|x\rangle\langle x|b\rangle}{\partial |x\rangle}\)

\(\frac{\partial (A|x\rangle+ |b\rangle)^\top C (D|x\rangle + |e\rangle)}{\partial|x\rangle}\)

And here’s one that’s a little bit different:

\(\frac{\partial \langle x|A|x\rangle}{\partial A}=|x\rangle\langle x|\)

We can think about also problems where the answer still has a tensor in it.

Going back to the original motivation, but let’s remove the y.

\(\frac{\partial |x\rangle\langle x|}{\partial \langle x|} = I\langle x| + |x\rangle\langle ee|\)

This is a tensor now that when you apply to a vector, you’ll get a matrix. How should we think about such an object? Is there a similar approximate linear equation that we can use to describe perturbations to the projection matrix onto x?

The result above can be described by what it produces when it’s applied to vectors. If we take derivatives of matrices with respect to vectors or derivatives of vectors with respect to matrices, we will end up with various tensors. How should we think of these? How does this generalize to derivatives of tensors?

1. See for example, this wiki: https://en.wikipedia.org/wiki/Matrix_calculus

2. See the wiki for a bigger conversation on tensors.

3. Generally the derivative of an (m1, m2, …)x(n1, n2, …) tensor by a (a1, a2, …)x(b1, b2, …) tensor has the shape (m1, m2, …, a1, a2, …)x(n1, n2, …, b1, b2, …). Very simple shape-wise.

4. Okay, if you must know. This is a bilinear operator, meaning that, like with matrix/vector multiplication, you can factor scalars out of either or both terms. Furthermore, it satisfies some universal property.

5. This is almost correct, but you have to remember that we’re not writing a summation. Only with the summation can you correctly recover all the entries of a.