BACKGROUND: Suppose the likelihood follows a Gaussian distribution, that is $X | \mu, \sigma \sim N(\mu, \sigma)$. Depending on which parameters are known (i.e. only $\mu$ is unknown; only $\sigma$ is unknown; and both $\mu, \sigma$ are unknown), we have appropriate conjugate priors such that the posterior distribution is also nicely behaved. Please see https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conjugate_distributions for details.
QUESTION: But what if we have a curved Gaussian distribution (or more generally, a curved exponential family)? That is, what if $X | \mu \sim N(\mu, \mu)$ (i.e. $E[X|\mu] = \mu$, and $Var(X|\mu) = \mu^2$ )? What prior distribution for $\mu$ would lead to a nice distribution for the posterior $\mu | X$?
In general I think this is going to be hard to say much in the general case, but when looking for conjugate priors lets first indicate the strategy with a popular example and then try your case
Finding a Conjugate Prior; the Gaussian precision
So in this instance the likelihood given a collection of observations is $$ p(\mathbf{x}|\lambda) = \prod_{i=1}^{n}\mathcal{N}(x_n | \mu, \lambda^{-1} ) \propto \lambda^{N/2} \exp\left\{ -\frac{\lambda}{2}\sum_{n=1}^{\infty}(x_n-\mu)^2 \right\} $$ Now we take that final term and write it in the form $f(\lambda)g(\lambda)$ where \begin{align} f(\lambda) &= \lambda^{N/2} \\ g(\lambda) &= \exp\left\{-\frac{\lambda}{2}\sum(x_n - \mu)^2 \right\}. \end{align} Now the step for constructing a conjugate prior $\pi(\lambda)$ is to chose its functional form such that $$ p(\mathbf{x}|\lambda)\pi(\lambda) \propto\tilde{f}(\lambda) \tilde{g}(\lambda) $$ where $\tilde{f}$ and $\tilde{g}$ are of the same function class as $f$ and $g$ respectively. This leads us to to consider a prior of the form $$ \pi (\lambda) \propto \lambda^c e^{a\lambda + b} $$ which after some normalisation we recognise as implying a Gamma distribution.
Curved Normal $\mathcal{N}\left(x | \mu, \mu\right)$
Using the same kind of strategy as above we have $$ p(\mathbf{x} | \mu) \propto \frac{1}{\mu^{\frac{1}{2}}}\exp\left\{-\frac{1}{2}\left(\mu + \frac{\sum_i x_i^2}{\mu} - 2\sum_i x_i\right) \right\} $$ this suggests a good prior may be of the form $$ \pi(\mu) \propto \frac{1}{\mu^{r+1}}\exp\left\{ -\frac{1}{2}\left(a_0\mu + \frac{b_0}{\mu} + 2c_0\right)\right\} $$ the $c_0$ term is just going to get rolled into the normalising constant, indeed we find $$ Z_{\pi} := e^{-c_0}\int_{0}^{\infty} \frac{1}{\mu^{r_0+1}}e^{-\frac{1}{2}\left(a_0\mu + \frac{b_0}{\mu} \right)}d\mu = 2e^{-c_0} \frac{\sqrt{a_0b_0}}{b_0}^{r_0} K_{r_0}(\sqrt{a_0 b_0}) $$ where $K_r(\cdot)$ is a modified Bessel function of the second kind. Since we are getting a clean closed form expression it would not be surprising if this example has been covered somewhere in the literature!
Finally then you get a posterior of the form $$ p(\mu | x ) \propto \frac{1}{\mu^{r + 1}}\exp\left\{-\frac{1}{2}\left( a\mu + \frac{b}{\mu} - 2c \right) \right\}, \qquad \mu > 0, $$ where the posterior parameters are \begin{align*} r &= r_0 + \frac{1}{2} \\ a &= a_0 + 1 \\ b &= b_0 + \sum_i x_i^2 \\ c &= c_0 + \sum_i x_i \end{align*} and the normalising constant is $$ Z_{\mu|x} = 2e^{-\sum_i x_i} \cdot \left( \frac{\sqrt{ab}}{b} \right)^{r}K_{r}(\sqrt{ab}). $$
For general curved exponential distributions I don't know how much can be said beyond examining specific cases. Having thought about it for a bit one would think it should be possible to take the conjugate prior for the full exponential family of interest and then apply the same parameter restriction - where I can see this getting harder is if enforcing these parameter constraints lead to awkward expressions for the hyper parameters of the prior.
I guess a reasonable place to start would be to see if the Gaussian-Gamma distribution which is the conjugate prior to the $\mathcal{N}(\mu,\sigma^2)$ distribution contains this distribution as a special subclass, in particular does the same parameter constraint $\mu = \sigma^2$ lead to the prior found above.