Given a convex set $S\subset \mathbb{R}^n$ and some $\theta\in S$, consider the observation $y=\theta+\epsilon$ where $\epsilon\sim \mathcal{N}(0,I)$, the ERM estimator is $$\hat{\theta}=\arg \min_{x\in S}\|x-y\|_2.$$ Consider the usual bias-variance decomposition $$\mathbb{E}\|\hat\theta-\theta\|^2=\|\mathbb{E}\hat\theta-\theta\|^2+var(\hat\theta)=bias^2+variance.$$ Does there exists some $S$ and $\theta$, such that the bias term is much larger than the variance term?
For example, for 1D linear regression on $x_1,...,x_n\in\mathbb{R}$, the $S$ here is $S=\{(y_1,...,y_n):y_i=kx_i+b, k,b\in\mathbb{R}\}$ is a hyperplane in $\mathbb{R}^n$, so the estimator is unbiased.