Suppose $\theta \sim N(\mu, \sigma)$ and $Y = f(\theta) = \theta^2 + w$ with $w \sim N(0, \sigma_w)$.
Obviously $p(Y)$ is not Gaussian, as $f$ is nonlinear in $\theta$.
My question is: Is $p(Y|\theta)$ Gaussian distributed? I assume it is, because due to $\theta$ being given (probability of $Y$ given $\theta$) its distribution doesn't matter, so we have $p(Y|\theta) \sim N(\theta^2, \sigma_w)$. Is this correct?
You are indeed correct, although correctly $Y|\theta\sim N(\theta^{2},\sigma_{w})$ - i.e. the random variable $Y$ given $\theta$ is normally distributed and not $P(Y|\theta)$