Quadratic cost function solution

Question

Why the solution to the following cost function:

$$\frac{1}{2}\|Lm-d\|^2 + \frac{1}{2} \mu \|W_m m\|^2_2$$

the below equation:

$$(L^Td+\mu W^T_m W_m)^{-1} L^Td $$

Answer 1

Suppose the problem was discretized. I.e. $L$ and $W$ are matrices, $\vec{m}$ and $\vec{d}$ are vectors. Thus the problem is to find the minimum of the following cost function \begin{equation} f(\vec{m}) = \frac{1}{2}(L_{ij}m_j - d_i)(L_{ik}m_k - d_i) + \frac{1}{2}\mu W_{ij}m_jW_{ik}m_k \end{equation} by arguments $m_p$. To find the extremum of the function, we have to solve the following equation \begin{equation} \frac{\partial f}{\partial m_p} = 0, \end{equation} which brings us to the following system of equations \begin{equation} (L_{ip}L_{ik} + \mu W_{ip}W_{ik})m_k = L_{ip}d_i. \end{equation} The last equation can be rewritten in the matrix form as follows \begin{equation} (L^{\intercal}L+\mu W^{\intercal}W)\vec{m}=L^{\intercal}\vec{d}. \end{equation} The solution of this equation is \begin{equation} \vec{m} = (L^{\intercal}L + \mu W^{\intercal}W)^{-1}L^{\intercal}\vec{d}. \end{equation}

Answer 2

You can write this as

$\min \frac{1}{2} \| Am-b \|_{2}^{2}$

where

$A=\left[ \begin{array}{c} L \\ \sqrt{\mu} W_{m} \end{array} \right]$

and

$b=\left[ \begin{array}{c} d \\ 0 \end{array} \right]$.

Applying the normal equations to this linear least squares problem gives (assuming that the inverse exists)

$m=(A^{T}A)^{-1}A^{T}b.$

This simplifies to

$m=\left( \left[ L^{T} \sqrt{\mu}W_{m}^{T} \right] \left[ \begin{array}{c} L \\ \sqrt{\mu}W_{m} \end{array}\right]\right)^{-1} \left[ L^{T} \sqrt{\mu} W_{m}^{T} \right] \left[ \begin{array}{c} d \\ 0 \end{array} \right]$

and finally

$m=\left(L^{T}L+\mu W_{m}^{T}W_{m} \right)^{-1}L^{T}d$.