Showing the Hessian is positive definite

61 Views Asked by At

I have a Hessian $H$ which is

$$ H=\sum_{i=1}^{n} \frac{d\left(\sigma\left(\vec{a}_{i}^{T} \vec{x}\right)-b_{i}\right) \vec{a}_{i}^{T}}{d \vec{x}}=\sum_{i=1}^{n} \overrightarrow{a_{i}}{\overrightarrow{a_{i}}}^{T} \sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\left(1-\sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\right)$$

where $\vec{a}_{i}, \vec{x} \in R^{d}$ (column vectors) and $\sigma(x)=\frac{1}{1+e^{-x}}$.

I need to show this Hessian $H$ is positive definite.

Then, the solution provided by the instructors says as follows.

" ... It's sufficient to prove that $v^{T} H v>0$ for any $v$. $$ v^{T} H v=\sum_{i=1}^{n}\left(\vec{a}_{i} \vec{v}^{T}\right)^{2} \sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\left(1-\sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\right) $$ ..."

I can't understand how $\vec{v}^TH\vec{v} = \sum_{i=1}^{n} \vec{v}^T\overrightarrow{a_{i}}{\overrightarrow{a_{i}}}^{T} \sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\left(1-\sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\right)\vec{v}$ becomes to have a form starting with $\left(\vec{a}_{i} \vec{v}^{T}\right)^{2}$.

Anyone can help me how $\vec{v}^T\overrightarrow{a_{i}}{\overrightarrow{a_{i}}}^{T}\vec{v}$ is cleared as $\left(\vec{a}_{i} \vec{v}^{T}\right)^{2}$?

1

There are 1 best solutions below

0
On BEST ANSWER

As @KBS suggested, basically it's the general rule of inner product, it's symmetric to its arguments. $a_i^Tv=v^Ta_i$ Then in your fomula, we could write

$$ \vec{v}^TH\vec{v} = \sum_{i=1}^{n} (\vec{v}^T\overrightarrow{a_{i}})({\overrightarrow{a_{i}}}^{T}\vec{v}) \sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\left(1-\sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\right)\\ =\sum_{i=1}^{n} ({\overrightarrow{a_{i}}}^{T}\vec{v})^2\sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\left(1-\sigma\left({\overrightarrow{a_{i}}}^{T} \vec{x}\right)\right) $$