Suppose the Fourier cosine transform is given by:
\begin{align} F_c(k)=\mathcal{F}(f(x))&=\sqrt{\frac{2}{\pi}}\int_0^{\infty} f(x) \cos(kx) \end{align}
or any other form I'm not particular here.
1 What is the conjugate?
2 How can one make this into complex notation and get then congugate? (which should be easy in complex notation)
Your question implies $f(x) $ is complex, so the conjugate would be the same expression using the conjugate of $f(x)$. To answer your second question let $f(x)=f_R(x)+if_I(x)$. Then use $f_R(x)-if_I(x)$ in the expression for the transform to get the conjugate.