Bounds on least squares and weighted least squares estimator

Question

I was wondering if I can get some help in getting bounds on the parameters estimated by least squares (LS) and weighted least squares (WLS) methods.

Suppose our observation model is:

$\mathbf{y} = \Phi \mathbf{a} + \boldsymbol{\eta}$

The LS estimator is then given by:

$\hat{\mathbf{a}} = (\Phi^{T}\Phi)^{-1}\Phi^{T}y$

And the WLS estimator is given by:

$\tilde{\mathbf{a}} = (\Phi^{T}W\Phi)^{-1}\Phi^{T}Wy$,

where $W$ is a weight matrix.

We further assume that $\Phi$ is a full rank matrix and $W$ is a positive seimidefinite matrix.

Is there a bound on $||\hat{\mathbf{a}}||$ and $||\tilde{\mathbf{a}}||$ in terms of $||\mathbf{y}||$ and some singular values or norms of $\Phi$ and $W$. What if $W$ is diagonal and positive definite? What if $W$ is diagonal and positive semidefinite definite?

Thanks!

Answer 1

In the OLS case, $\| \hat{\mathbf{a}} \| \leq \| (\Phi^{T}\Phi)^{-1}\| \cdot \| \Phi \| \cdot \|y\|$. The operator norm $\| (\Phi^{T}\Phi)^{-1}\|$ is directly related to the singular values of $\Phi$. Same goes for $\|\Phi \|$ (it is the square root of the largest singular value of $\Phi$).

In the WOLS case, replace $\Phi$ by $W^{\frac{1}{2}} \Phi$.