$\phi (t) = \ t^2\ -\log t \ -t; \ t\in R_{++}$
By definition, the conjugate is:
$\phi^*(y) = \ sup_t \{ty-t^2+\log t +t\}$
So I took the derivative of the function inside the supremum (let's call it $f(t)$) and equalized to $0$:
$f^{'}(t)=y-2t+\frac{1}{t}+1=0 \Rightarrow ty - 2t^2 + t + 1 = 0$
$\Rightarrow f(t) = t^2 + \log t - 1$
Should I take now the derivative of $f(t)$ equalize to $0$ and use the resulting $t$ in the conjugate or there is another way?
Step 1: Take the derivative of $f(t) := ty-t^2+\ln(t)+t$ with respect to $t$ and get: $$y-2t+\tfrac{1}{t}+1.$$
Step 2: Set this equal to $0$ and solve for $t$. Note that this is a quadratic in $t$, so you need to pick the positive root which is $$\frac{y+1+\sqrt{y^2+2y+9}}{4}$$
Step 3: In Step 2, we found the maximizer of $f(t)$ where $t>0$, which you now must substitute into your $f(t)$. You will get the answer (with the help of Maple): $$\ln\Big(y+1+\sqrt{y^2+2y+9}\Big) + \frac{1}{8}(y+1)\sqrt{y^2+2y+9} + \frac{1}{8}y^2 + \frac{1}{4}y -2\ln(2)-\frac{3}{8} $$ which is your desired $\phi^*(y)$.