I need show that $Ker \varphi = (X^2 + Y^2 - 1)$ . Clearly $(X^2 + Y^2 -1 ) \subset Ker \varphi$. To show the other inclusion, I did:
Let $f(X,Y) \in Ker \varphi$. Then
$$ f(X,Y) = (X^2 + Y^2 - 1)q(X,Y) + r(X,Y)$$
such that either $r(X,Y)=0$ or $r(X,Y)=g(X)+h(X)Y$. If $r(X,Y)=0$, the proof ends. Then, suppose that $r(X,Y)=g(X)+h(X)Y$.
Since that $\varphi(r(X,Y)) = 0$, $r(X,Y) \in Ker \varphi$; that is:
$$g(\frac{1-t^2}{t^2+1})+h(\frac{1-t^2}{t^2+1})(\frac{2t}{t^2+1}) = 0$$
The hint to end, is look on for the degree of this expression, but I can not see the contradiction.
Picking up where you left off, let $g(X), h(X) \in k[X]$ be nonzero polynomials such that
$$g\left(\frac{1-t^{2}}{t^{2}+1}\right) + h\left(\frac{1-t^{2}}{t^{2}+1}\right)\frac{2t}{t^{2}+1} = 0$$
Clearing denominators by multiplying by a suitable power of $t^{2}+1$, there exist non-negative integers $m, n \in \mathbb{Z}$ such that
$$(t^{2}+1)^{m}g(1-t^{2}) + (t^{2}+1)^{n}h(1-t^{2})2t = 0$$
whence
$$(t^{2}+1)^{m}g(1-t^{2}) = -(t^{2}+1)^{n}h(1-t^{2})2t $$
Now, whatever $n, m$ are, $(t^{2}+1)^{m}$ and $(t^{2}+1)^{n}$ are polynomials of even degree in $t$. Likewise, whatever the degrees of $g, h$ are, $g(1-t^{2})$ and $h(1-t^{2})$ are also of even degree in $t$, so $(t^{2}+1)^{m}g(1-t^{2})$ and $(t^{2}+1)^{n}h(1-t^{2})$ are both of even degree in $t$. But then this means that the LHS of the equation above has even degree in $t$, whereas the RHS has odd degree in $t$, which gives the desired contradiction.