Following a probabilistic calculation of posterior probability distribution of hyperparameters that have to be integrated or maximized to get de MAP (Maximum a Posteriori) of that hyperparameters, I fall on :
$\begin{array}{l}P({\xi _D},{\xi _T},\sigma |Y){\rm{ }} \propto {\rm{ }}{\left| {\xi _T^2R + I} \right|^{ - \frac{1}{2}}}{\left( {\frac{1}{{{\sigma ^2}}}} \right)^{N + 1}}{\left( {{\xi _T}^2} \right)^{\frac{N}{2}}}*\\{\rm{ }}\exp \left[ {{\rm{ - }}{\xi _T}^2{{\left( {{\xi _D}} \right)}^T}K\left( {{\xi _D}} \right) + \frac{1}{2}{{\left( {{\xi _D}} \right)}^T}{B^T}A{{\left[ {{\xi _T}^2R + I} \right]}^{ - 1}}{A^T}B\left( {{\xi _D}} \right){\rm{ - }}\frac{1}{{2{\sigma ^2}}}{Y^T}\left[ {I - {{\left[ {{\xi _T}^2R + I} \right]}^{ - 1}}} \right]Y + \frac{1}{\sigma }{Y^T}{{\left[ {{\xi _T}^2R + I} \right]}^{ - 1}}{A^T}B{\xi _D}} \right]\end{array} $
where : ${\rm{}}{{\rm{\xi }}_{\rm{T}}}^{\rm{2}}{\rm{ = }}\frac{{\rm{1}}}{{{{\rm{\xi }}_{\rm{L}}}^{\rm{2}}{\rm{ - }}{{\left( {{{\rm{\xi }}_{\rm{D}}}} \right)}^{\rm{T}}}{\rm{K}}\left( {{{\rm{\xi }}_{\rm{D}}}} \right)}}$.
${{\rm{\xi }}_{\rm{D}}}{\rm}$ : a positive vector of N location hyperparameters (slope parameters ).
${{\rm{\xi }}_{\rm{L}}}$ : is a scalar positive location hyperparameter (regularizatin parameter).
${\rm{Y:}}$ is N by 1 vector of data.
${\rm{R}}$ : N by N matrix , is SDP that could be decomposed into ${{\rm{W}}^{\rm{T}}}{\rm{DW}}$ where ${{\rm{W}}^{\rm{T}}}{\rm{W = I}}$ and ${\rm{R = }}{{\rm{W}}^{\rm{T}}}{\rm{DW}}$.
${\rm{K}}$, ${\rm{A}}$, ${\rm{B}}$ are N by N matrix.
$\sigma$ : scale positive (nuisance) parameter.
$||$ determinant of a matrix.
$T$ : transpose of a vector or a matrix.
I have two questions that arise:
First, Is it possible to integrate over the nuisance parameter $\sigma$ to get a close form of $P({\xi _D},{\xi _T} |Y)$ and then found the MAP of ${{\rm{\xi }}_{\rm{D}}}{\rm}$ and ${{\rm{\xi }}_{\rm{L}}}{\rm}$. Or it's not possible to integrate over $\sigma$.
Second : In that case, I have to maximze $P({\xi _D},{\xi _T},\sigma |Y) $ find the MAP of $\sigma$ ,${{\rm{\xi }}_{\rm{D}}}{\rm}$ , ${{\rm{\xi }}_{\rm{L}}}{\rm}$.