Let $\theta_*$ be the d-dimensional hidden true parameter,
$y_t = x_t^\top \theta_* + \eta_t$ where $\eta_t$ is a standard gaussian noise.
It is well known that regularized least square estimator is $\hat{\theta}_t = V_t^{-1} X_t y$,
where $V_t = \lambda I + \sum_{i=1}^T x_i x_i^\top$, $X_t = [x_1 | x_2 | \cdots | x_t]$, $y=(y_1, y_2, \cdots , y_t)^\top$.
Suppose that $\|\theta_*\|\leq 1 $. Then
- Is there any high-probability bound for $\|\hat{\theta}_t\|$?
- Is there any condition that gives bound for $\|\hat{\theta}_t\|$?
For me, 'any' bound will be great for me.
It doesn't have to be close to 1.