The problem:
Given a known quantity $x$, distributed with known distribution $π(x)$ ~ $N(0,σ^2)$, I'm looking for the distribution of the estimator of $x$, $\hat{x}$ distributed with $p(\hat{x}\mid x)$ by minimizing
$F = \langle (x-\hat{x})^2\rangle _{x,\hat{x}}+βI(x,\hat{x})$ with respect to $p(\hat{x}\mid x)$.
where $I(x,\hat{x})$ is the mutual information between the two quantities.
My attempt:
$\langle (x-\hat{x})^2\rangle _{x,\hat{x}} = \mathbb E_x[\mathbb E_{\hat{x}|x}[(x-\hat{x})^2\mid x]] = \int_{-\infty}^\infty (x-\hat{x})^2p(\hat{x}\mid x) \pi(x)dx$
$I(x,\hat{x}) = \mathbb E[\log \frac{p(\hat{x}|x)}{p(\hat{x})}] = \int p(\hat{x}\mid x)\log p(\hat{x}\mid x)-(\int p(\hat{x}\mid x')\pi(x')dx')\log(\int p(\hat{x}\mid x'')\pi(x'')dx'')dx$
I think I made an error in $I(x,\hat{x})$...
But taking a functional derivative of $F$ w.r.t. $p(\hat{x}\mid x)$ gives me:
$\int_{-\infty}^\infty (x-\hat{x})^2\pi(x)dx + β\int \{\log p(\hat{x}\mid x)-\log[\int p(\hat{x}\mid x')\pi(x')dx']\} $
I would set this to 0 to find $p(\hat{x}\mid x)$, but I'm not certain this is right.
Your problem seems unconventional and of no clear utility. Are you sure you have posed it correctly? Assuming you have, how about this approach:
It is well-known (you can show it) that the optimal $\hat{x}$ that minimizes the mean squared error (MSE) $\mathbb{E}((x-\hat{x})^2) $ is $\hat{x}=\mathbb{E}(x)$. Since, by default, $\mathbb{E}(x)=0$, it follows that the MSE-minimizing (conditional) distribution for $\hat{x}$ is the degenerate distribution $p(\hat{x}|x)=\delta(\hat{x})$, where $\delta(\cdot)$ is the Dirac delta function. Note that the MSE-minimizing $\hat{x}$ is independent of $x$.
Now, consider the mutual information term. It is well-known (you can show it) that $I(x;\hat{x})\geq 0$ with the lower bound achieved for any $\hat{x}$ that is indepedent of $x$. Clearly, one such choice that achieves the bound is a $\hat{x}$ disributed as the MSE-minimizing distribution $p(\hat{x}|x)=\delta(\hat{x})$.
Since $p(\hat{x}|x)=\delta(\hat{x})$ is a minimizer for both factors of your cost function, it is also a minimizer for the cost function as well.