derivative with respect to an index permutation

1000 Views Asked by At

Let $x$ be a function expressed as a weighted sum of $K$ basis functions $u_1(t),..,u_K(t)$.

For the sake of simplicity, we will consider orthonormal basis functions that take discrete values. Thus, we can express these basis functions as vectors:

$$\mathbf{x} = \sum_{k=1}^K z_k\mathbf{u_k} = Uz$$

let $\sigma$ be a permutation of $\{1,..,K\}$, $\tilde{K}<K$.

approximating $\mathbf{x}$ by $$\mathbf{\hat{x}_\sigma} = \sum_{k=1}^\tilde{K} z_{\sigma(k)}\mathbf{u_{\sigma(k)}}$$

find $$\sigma^{min} = argmin_\sigma \Vert x-\hat{x}_\sigma\Vert_2^2$$

"Hint": $\Vert a-b\Vert_2^2=\langle a-b,a-b\rangle$

"Also keep in mind, that" $\langle u_k, u_\ell \rangle = \begin{cases}0 & ,k\neq \ell\\1 & , else\end{cases}$

Right ...

$\Vert x-\hat{x}_\sigma\Vert_2^2 = \langle x-\hat{x}_\sigma, x-\hat{x}_\sigma\rangle = ... = \langle x,x\rangle + \langle \hat{x}_\sigma,\hat{x}_\sigma \rangle -2 \langle x,\hat{x}_\sigma\rangle$

Now what? I suppose there'll be a derivative somewhere down the line, since this is an optimisation question, but I have zero ideas how to rewrite

$$\frac{\partial}{\partial \sigma}\left(\langle x,x\rangle + \langle \hat{x}_\sigma,\hat{x}_\sigma \rangle -2 \langle x,\hat{x}_\sigma\rangle\right)$$

I do suppose $\frac{\partial}{\partial \sigma}x = 0$ because $x$ has absolutely nothing to do with the permutation.

If so, $\frac{\partial}{\partial \sigma}\langle x,x\rangle=0$

and $\frac{\partial}{\partial \sigma}-2 \langle x,\hat{x}_\sigma\rangle= -2 \langle x,\frac{\partial}{\partial \sigma}\hat{x}_\sigma\rangle$

Fine, that leaves me with

$$\frac{\partial}{\partial \sigma}\Vert x-\hat{x}_\sigma\Vert_2^2 = 2\left(\langle\hat{x}_\sigma, \frac{\partial}{\partial \sigma}\hat{x}_\sigma \rangle - \langle x,\frac{\partial}{\partial \sigma}\hat{x}_\sigma \rangle\right) = 2\langle \hat{x}_\sigma -x , \frac{\partial}{\partial \sigma}\hat{x}_\sigma\rangle$$

and still no idea where to go from here.

How the heck do you calculate a derivative with respect to a permutation on an index?

1

There are 1 best solutions below

0
On BEST ANSWER

The derivative with respect to a permutation makes no sense, of course. You cannot differentiate with respect to a (vector) variable that only takes discrete-values.

Replacing $\hat{\mathbf{x}}_\sigma$ and $\mathbf{x}$ with their basis representations, it is easy to show that

$$ \begin{align} \|\hat{\mathbf{x}}_\sigma-\mathbf{x}\|_2^2 &= \sum_{k=1}^{K}|z_k|^2 - \sum_{k=1}^{\tilde{K}}|z_{\sigma(k)}|^2\\ &=\sum_{k=\tilde{K}+1}^{K}|z_{\sigma(k)}|^2 \end{align} $$

Therefore any permutation that minimizes the last sum is optimal.