If $f: \mathbb R^n \to \mathbb R$ then $conj(f) : \mathbb R^n\to \mathbb R$ is defined as $conj(f)(y) = \sup_{x \in dom_f} (y^Tx - f(x))$ and it is called the conjugate function of $f$.
if $f(X) = -\log(\det X)$ where $X$ is a symmetric positive definite matrix, what is the domain of $conj(f)$ and how can we show that it is convex? ($Y^TX$ is defined as $trace(YX)$).
We can determine the domain of $f^*$ without having to determine $f^*$ fully, but that's not difficult, either.
Suppose we have a value of $Y$ satisfying $\langle Y, X \rangle \geq 0$ for some $X\succ 0$. Define $$\alpha\triangleq \langle Y, X \rangle\geq 0, \quad \beta\triangleq\log\det X > 0.$$ Now consider $\bar{X}=t X$ for $t>0$. Then $\bar{X}\in\textrm{dom} f$, and $$\langle Y, \bar{X} \rangle + \log\det\bar{X} = t\langle Y, X \rangle + \log\det tX = t\alpha + \log (t^n\beta) = t\alpha+n\log t + \log\beta.$$ $$f^*(Y) = \sup_{X\succ 0} \langle Y, X \rangle + \log\det X \geq \sup_{t>0} t\alpha+n\log t + \log\beta = +\infty.$$ Therefore this value of $Y$ cannot be in the domain of $f^*$. It must be the case, then, that $$Y\in\textrm{dom} f^* \quad\Longrightarrow\quad \langle Y, X \rangle < 0 ~ \forall X \succ 0 \quad\Longrightarrow\quad Y \prec 0$$ That is, the domain of $f^*$ is the set of negative definite matrices.
The key to computing $f^*$ is knowing that $\nabla_X \log\det(X) = X^{-1}$. So the optimality conditions for the maximization are $$Y+X^{-1} = 0 \quad\Longrightarrow\quad X=-Y^{-1}=(-Y)^{-1}.$$ It is clear by that $Y$ must be negative definite for this optimality condition to have a solution, confirming our claim about the domain of $f^*$. Substitution yields $$f^*(Y)=\langle Y, Y^{-1} \rangle + \log\det(-Y)^{-1}=n-\log\det(-Y).$$ copper.hat is right in his comment above about convexity. Because the convex conjugate is the supremum of linear functions of its argument, it is always convex, even if the original function is not. But here we also have $f^*(Y)=n+f(-Y)$, so convexity can also be established by the fact that $f(-Y)$ is the composition of a convex function with a linear function, and $n+f(-Y)$ is the sum of two convex functions.
EDIT: copper.hat brings up a fair point. The answer depends rather critically on the vector space one chooses. What I've done above assumes that $f$ is defined on the space of symmetric matrices $\mathcal{S}^n$. If you insist on defining it on $\mathbb{R}^{n\times n}$, then he is right about the domain: it's those matrices satisfying $Y+Y^T\prec 0$, and $f^*(Y)=n-\log\det\tfrac{-1}{2}(Y+Y^T)$. But then the function $f$ is kind of weird, because its domain has a non-empty interior: $$\mathop{\textrm{dom}} f = \{ X\in\mathbb{R}^{n\times n}~|~X=X^T,~X\succ 0 \}$$ I frankly think that's a bit messy, and it's better conceptually and in practice to think about this as a function on the space of symmetric matrices.