Exercise 0 - Matrix Calculus

TTK4135 - Optimalisering og regulering

Problem 1 (25%) Definitions

a)

Gradient

It points in the direction of the greatest rate of increase of the function, and its magnitude gives the rate of increase in that direction.

The gradient is a vector composed of the partial derivatives of the function with respect to each of its variables, where $f:{\mathbb{R}}^n\to \mathbb{R}$.

$$\nabla f(x)=[\frac{\partial f}{\partial x}{]}^T=\left[\begin{array}{ccc}\frac{\partial f}{\partial x_1}& \dots & \frac{\partial f}{\partial x_n}\end{array}\right]$$

---

Link to original

b)

Jacobian

The Jacobian is a generalization of the Gradient where we look at a continuously differentiable function $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$.

$$\frac{\partial f}{\partial x}=\left[\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& \cdots & \frac{\partial f_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial f_m}{\partial x_1}& \cdots & \frac{\partial f_m}{\partial x_n}\end{array}\right]$$

---

Link to original

c) The resulting gradient will be $1\times n$

$$\nabla f(x)=\frac{\partial f}{\partial x}=\left[\begin{array}{ccc}\frac{\partial f}{\partial x_1}& \dots & \frac{\partial f}{\partial x_n}\end{array}\right]$$

d) The resulting jacobian will be $m\times n$

$$\frac{\partial f}{\partial x}=\left[\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& \cdots & \frac{\partial f_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial f_m}{\partial x_1}& \cdots & \frac{\partial f_m}{\partial x_n}\end{array}\right]$$

Problem 2 (25%) Linear

Let $f(x)=Ax$, where

$$A=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right],x=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$$

a)

$$\begin{array}{rl}\frac{\partial f(x)}{\partial x}& =\frac{\partial }{\partial x}Ax\\ \frac{\partial f(x)}{\partial x}& =\frac{\partial }{\partial x}\left[\begin{array}{c}{a}_{11}{x}_1+a_{12}{x}_2\\ a_{21}{x}_1+a_{22}{x}_2\end{array}\right]=\left[\begin{array}{c}{f}_1\\ f_2\end{array}\right]\\ \frac{\partial f(x)}{\partial x}& =\left[\begin{array}{cc}\frac{\partial }{\partial x_1}{f}_1& \frac{\partial }{\partial x_2}{f}_2\\ \frac{\partial }{\partial x_1}{f}_2& \frac{\partial }{\partial x_2}{f}_2\end{array}\right]\\ \frac{\partial f(x)}{\partial x}& =\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]\\ \frac{\partial f(x)}{\partial x}& =A\end{array}$$

This is the jacobian of $x$, since $f(x)$ is a vector with two elements.

b)

$$\frac{\partial Ax}{\partial x}=A$$

When $x:n\times 1$ and $f:m\times n$.

Problem 3 (25%) Nonlinear

Let $f(x,y)=x^{T}Gy$, where

$$x=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right],G=\left[\begin{array}{ccc}{g}_{11}& g_{12}& g_{13}\\ g_{21}& g_{22}& g_{23}\end{array}\right],y=\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right]$$

a) Gradient With Respect to a Variable The dimensions of $f(x,y)$ is $1\times 2\ast 2\times 3\ast 3\times 1=1\times 1$

$\frac{\partial f(x,y)}{\partial x}$ will have dimensions $m\times n=1\times 2$, while ${\nabla }_{x}f(x,y)$ will have dimensions $2\times 1$. They are therefore a transpose different.

b) Method 1

$$\begin{array}{rl}{\nabla }_{x}f(x,y)& =Gy\\ {\nabla }_{x}f(x,y)& =\left[\begin{array}{ccc}{g}_{11}& g_{12}& g_{13}\\ g_{21}& g_{22}& g_{23}\end{array}\right]\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right]\\ {\nabla }_{x}f(x,y)& =\left[\begin{array}{c}{g}_{11}{y}_1+g_{12}{y}_2+g_{13}{y}_3\\ g_{21}{y}_1+g_{22}{y}_2+g_{23}{y}_3\end{array}\right]\end{array}$$

Method 2

$$\begin{array}{rl}{\nabla }_{x}f(x,y)& ={\nabla }_x(x^{T}Gy)\\ {\nabla }_{x}f(x,y)& ={\nabla }_x(x_1{y}_1{g}_{11}+x_2{y}_1{g}_{21}+x_1{y}_2{g}_{12}+x_2{y}_2{g}_{22}+x_1{y}_3{g}_{13}+x_2{y}_3{g}_{23})\\ {\nabla }_{x}f(x,y)& =\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \frac{\partial f}{x_2}\end{array}\right]\\ {\nabla }_{x}f(x,y)& =\left[\begin{array}{c}{g}_{11}{y}_1+g_{12}{y}_2+g_{13}{y}_3\\ g_{21}{y}_1+g_{22}{y}_2+g_{23}{y}_3\end{array}\right]\end{array}$$

c) We have $f(x,y)=x_1{y}_1{g}_{11}+x_2{y}_1{g}_{21}+x_1{y}_2{g}_{12}+x_2{y}_2{g}_{22}+x_1{y}_3{g}_{13}+x_2{y}_3{g}_{23}$, and calculate

$$\begin{array}{rl}{\nabla }_{y}f(x,y)& =\left[\begin{array}{c}\frac{\partial f}{\partial y_1}\\ \frac{\partial f}{\partial y_2}\\ \frac{\partial f}{\partial y_3}\end{array}\right]\\ {\nabla }_{y}f(x,y)& =\left[\begin{array}{c}{x}_1{g}_{11}+x_2{g}_{21}\\ x_1{g}_{12}+x_2{g}_{22}\\ x_1{g}_{13}+x_2{g}_{23}\end{array}\right]\end{array}$$

d) Using a variant of the product rule, we first differentiate with respect to the “first x”, and then with respect to the “second x” similar to how we did in a) and b)

$$\nabla f(x)=Gx+G^{T}x$$

Since $G$ is symmetric, $G=G^T$, we get

$$\nabla f(x)=2Gx$$

Problem 4 (25%) Common case

Given this scalar operator

$$\mathcal{L}(x,\lambda ,\mu )=x^{T}Gx+{\lambda }^T(Cx-d)+{\mu }^T(Ex-h)$$

Find

a) ${\nabla }_x\mathcal{L}(x,\lambda ,\mu )$

$$\begin{array}{rl}{\nabla }_x\mathcal{L}(x,\lambda ,\mu )& =(Gx+G^{T}x)+C^T\lambda +E^T\mu \end{array}$$

b) ${\nabla }_{\lambda }\mathcal{L}(x,\lambda ,\mu )$

$${\nabla }_{\lambda }\mathcal{L}(x,\lambda ,\mu )=Cx-d$$

c) ${\nabla }_{\mu }\mathcal{L}(x,\lambda ,\mu )$

$${\nabla }_{\mu }\mathcal{L}(x,\lambda ,\mu )=Ex-h$$

---