Matrix Calculus

Matrix Calculus #

Revised

17 Jun 2023

.	scalar	vector	matrix
scalar	$\begin{array}{r} \frac{\partial y}{\partial x} \end{array}$	$\begin{array}{r} \frac{\partial y}{\partial x} \end{array}$	$\begin{array}{r} \frac{\partial Y}{\partial x} \end{array}$
vector	$\begin{array}{r} \frac{\partial y}{\partial x} \end{array}$	$\begin{array}{r} \frac{\partial y}{\partial x} \end{array}$
matrix	$\begin{array}{r} \frac{\partial y}{\partial X} \end{array}$

scalar-by-scalar #

derivative of a scalar (function) wrt a scalar

[example]

$\begin{aligned} f (x) & = k x \\ \frac{d f}{d x} & = k \end{aligned}$

scalar-by-vector #

derivative of a scalar (function) wrt a vector

$\begin{aligned} x & = {[\begin{array}{c} x_{1} & \dots & x_{n} \end{array}]}^{⊤} \\ \frac{\partial y}{\partial x} & = [\begin{array}{c} \frac{\partial y}{\partial x_{1}} & \dots & \frac{\partial y}{\partial x_{n}} \end{array}] \end{aligned}$

An example from the vector calculus is the gradient vector.

$\begin{array}{r} \nabla f (x) = \frac{\partial f}{\partial x} = [\begin{array}{c} \frac{\partial f}{\partial x_{1}} & \dots & \frac{\partial f}{\partial x_{n}} \end{array}] \end{array}$

vector-by-scalar #

derivative of a vector (function) wrt a scalar

$\begin{aligned} y & = {[\begin{array}{c} y_{1} & \dots & y_{m} \end{array}]}^{⊤} \\ \frac{\partial y}{\partial x} & = [\begin{array}{c} \frac{\partial y_{1}}{\partial x} \\ \dots \\ \frac{\partial y_{m}}{\partial x} \end{array}] \end{aligned}$

An example in the vector calculus is the tangent vector.

vector-by-vector #

derivative of a vector (function) wrt a vector

taking the derivative of a linear transformation (not “taking the derivative of a matrix”)

$\begin{aligned} \frac{\partial y}{\partial x} & = [\begin{array}{c} \frac{\partial y_{1}}{\partial x_{1}} & \dots & \frac{\partial y_{1}}{\partial x_{n}} \\ ⋮ & ⋱ & ⋮ \\ \frac{\partial y_{m}}{\partial x_{1}} & \dots & \frac{\partial y_{m}}{\partial x_{n}} \end{array}] \end{aligned}$

Jacobian #

An example from the vector calculus is the Jacobian.

$\begin{array}{r} \frac{\partial}{\partial x} Ax = A \end{array}$ #

[example]

$\begin{aligned} f (x) = Ax & = [\begin{array}{c} 1 & 2 \\ 3 & 4 \end{array}] [\begin{array}{c} x_{1} \\ x_{2} \end{array}] = [\begin{array}{c} x_{1} + 2 x_{2} = f_{1} (x) \\ 3 x_{1} + 4 x_{2} = f_{2} (x) \end{array}] \\ \frac{\partial f}{\partial x} & = [\begin{array}{c} \frac{\partial f_{1}}{\partial x_{1}} & \frac{\partial f_{1}}{\partial x_{2}} \\ \frac{\partial f_{2}}{\partial x_{1}} & \frac{\partial f_{2}}{\partial x_{2}} \end{array}] = [\begin{array}{c} 1 & 2 \\ 3 & 4 \end{array}] = A \end{aligned}$

$\begin{array}{r} \frac{\partial}{\partial x} x^{⊤} Ax = 2 x^{⊤} A^{⊤} \end{array}$ #

[example]

$\begin{aligned} f (x) = x^{⊤} Ax & = [\begin{array}{c} x_{1} & x_{2} \end{array}] [\begin{array}{c} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array}] [\begin{array}{c} x_{1} \\ x_{2} \end{array}] \\ = [\begin{array}{c} x_{1} & x_{2} \end{array}] [\begin{array}{c} a_{11} x_{1} + a_{12} x_{2} \\ a_{21} x_{1} + a_{22} x_{2} \end{array}] \\ = [\begin{array}{c} x_{1} (a_{11} x_{1} + a_{12} x_{2}) + x_{2} (a_{21} x_{1} + a_{22} x_{2}) \end{array}] \\ = [\begin{array}{c} a_{11} x_{1}^{2} + a_{12} x_{1} x_{2} + a_{21} x_{1} x_{2} + a_{22} x_{2}^{2} \end{array}] \\ = [\begin{array}{c} a_{11} x_{1}^{2} + 2 a x_{1} x_{2} + a_{22} x_{2}^{2} = f_{1} (x) \end{array}] & a_{12} = a_{21} for a symmetric matrix \\ \frac{\partial f}{\partial x} & = [\begin{array}{c} \frac{\partial f_{1}}{\partial x_{1}} & \frac{\partial f_{1}}{\partial x_{2}} \end{array}] \\ = [\begin{array}{c} 2 a_{11} x_{1} + 2 a x_{2} & 2 a x_{1} + 2 a_{22} x_{2} \end{array}] \\ = 2 [\begin{array}{c} a_{11} x_{1} + a x_{2} & a x_{1} + a_{22} x_{2} \end{array}] \\ = 2 (Ax)^{⊤} \\ = 2 x^{⊤} A^{⊤} \end{aligned}$

matrix-by-scalar #

derivative of a matrix (function) wrt a scalar

this is called the tangent matrix

$\begin{aligned} \frac{\partial Y}{\partial x} & = [\begin{array}{c} \frac{\partial y_{11}}{\partial x} & \dots & \frac{\partial y_{1 n}}{\partial x} \\ ⋮ & ⋱ & ⋮ \\ \frac{\partial y_{m 1}}{\partial x} & \dots & \frac{\partial y_{m n}}{\partial x} \end{array}] \end{aligned}$

scalar-by-matrix #

derivative of a scalar (function) wrt a matrix

this is called the gradient matrix

important examples of scalar functions of matrices include the trace of a matrix and the determinant

$\begin{aligned} \frac{\partial y}{\partial X} & = [\begin{array}{c} \frac{\partial y}{\partial x_{11}} & \dots & \frac{\partial y}{\partial x_{p 1}} \\ ⋮ & ⋱ & ⋮ \\ \frac{\partial y}{\partial x_{1 q}} & \dots & \frac{\partial y}{\partial x_{p q}} \end{array}] \end{aligned}$

Resources #

[Y] ritvikmath. (09 Sep 2019). “Derivative of a Matrix : Data Science Basics”. YouTube.

Terms #

[W] Jacobian
[W] Matrix Calculus
[W] Tangent Vector

Matrix Calculus

Contents