Let $(Xt)$ be a solution to the equation
$dX_t = aX_t dt + \sqrt{(1+X_t^2)} dW_t$ where $W_t$ is a Brownian motion process at time t
Let $Y = F(X_t)$ for a certain function $F$. Find $F$ for which $(Y_t) solves the equation
$dY_t = f(Y_t) dt + dW_t$
where the drift term f should be explicitly determined. You may use the fact that
$$ \int \frac{1}{\sqrt{1+x^2}}dx\ = ln (x + \sqrt{1+x^2}). $$
My attempt
$dY_t = dF(X_t)$
Applying ito's formula we obtain.
$dF(X_t) = F'(X_t) dX_t + 1/2 F''(X_t)(dX_t)^2$
Substituting $dX_t$ and $(dX_t)^2$
$dF(X_t) = F'(X_t) (aX_tdt + \sqrt{1+X^2_t} dW_t) + \frac{1}{2} F''(X_t)(1+X^2_t)dt$ $dF(X_t) = F'(X_t)(aX_t + \frac{1}{2} F''(X_t)(1+X^2_t)) dt + F'(X_t)\sqrt{1+X^2_t} dW_t$
Equating coefficients for $dW_t$
$F'(X_t)\sqrt{1+X^2_t} = 1$
$F(X_t) = \int \frac{1}{\sqrt{1+X_t^2}}dx$
$F(X_t) = ln (X_t + \sqrt{1+X_t^2}) + C$, where C is a constant
Now my question is; Can we find this constant? If not, is the answer above correct for $F(x) $ now after equating the coefficients of $dt$, the drift term f is
$f(Y_t) = f(F(X_t)) = aX_t + \frac{1}{2} F''(X_t)(1+X^2_t)$
Question; How can I find the function f?
Your solution is correct. You find $$ F(X_t) = \ln \left(X_t + \sqrt{1+X_t^2}\right) + C, $$ for any constant $C$. In fact, for all such constants, $(Y_t)$ will satisfy the expected stochastic differential equation, with initial value $Y_0=\ln \left(X_0 + \sqrt{1+X_0^2}\right) + C$.
Afterwards, you made a typo when writing $f(Y_t)$, it should be $$ f(Y_t) = f(F(X_t)) = aX_t\mathbf{F'(X_t)} + \frac{1}{2} F''(X_t)(1+X^2_t). $$ Simple computations show that $$ F'(X_t)=\frac1{\sqrt{1+X_t^2}},\text{ and }F''(X_t)=-X_t\left(1+X_t^2\right)^{-3/2}, $$ and therefore, $$ f(F(X_t))=\left(a-\frac12\right)\frac{X_t}{\sqrt{1+X_t^2}}. $$ Lastly, it remains to note that $\ln \left(X_t + \sqrt{1+X_t^2}\right)=\sinh^{-1}(X_t)$, so $$ f(Y_t)=\left(a-\frac12\right)\tanh\left(Y_t-C\right)=\left(a-\frac12\right)\tanh\left(Y_t-Y_0+\ln \left(X_0 + \sqrt{1+X_0^2}\right)\right). $$