How can I solve the following unconstrained optimization problem in $\alpha \in \mathbb R$?
$$\text{maximize} \quad \log \det( \alpha A+I) + \mbox{tr} \left((\alpha A + I)^{-1} \alpha A \right)$$
where $A$ is a positive semidefinite matrix and $I$ is the identity matrix.
On
I assume you want $\alpha$ to be non-negative? It looks like your objective function is concave. If so, the minimum is achieved at the extreme points of the allowed range for $\alpha$. That is, at $\alpha=0$ or $\alpha=\infty$. Obviously $\alpha=\infty$ isn't it, so the answer is $\alpha=0.$
As a Rodrigo de Azevedo points out, the maximum is not attained. If your matrix has eigenvalues $\lambda_i$, the objective function is
$$ \sum_{i=1}^n \left( \log(\alpha \lambda_i+1) + \frac{\alpha \lambda_i}{\alpha \lambda_i + 1} \right) .$$
Its derivative, a rational function, is easy to write down, and is manifestly positive-valued for all $\alpha$.
Define a new matrix variable $$X=I+\alpha A$$ and note the following relationships $$\eqalign{ \alpha A &= X-I \cr \alpha AX^{-1} &= (X-I)X^{-1} = I-X^{-1} \cr dX &= A\,d\alpha \cr }$$ Now write the objective function in terms of $X$ $$\eqalign{ f &= \log\det X + {\rm tr}(\alpha AX^{-1}) \cr &= \log\det X + {\rm tr}(I) - {\rm tr}(X^{-1}) \cr }$$ The function is unbounded as $\alpha\to\infty\,\,$ since the trace term tend to zero while the log-det term increases.
Now find the differential and gradient $$\eqalign{ df &= d{\rm tr}(\log X-X^{-1}) \cr &= {\rm tr}((X^{-1}-X^{-2})\,dX) \cr &= {\rm tr}((X^{-1}-X^{-2})A)\,d\alpha \cr &= {\rm tr}(\alpha A^2X^{-2})\,d\alpha \cr\cr \frac{df}{d\alpha}&= \alpha\,{\rm tr}(A^2X^{-2}) \cr }$$ Setting the gradient to zero, means either $\alpha=0$ or the trace-term is zero.
Since $$\lim_{|\alpha|\to\infty} AX^{-1} = \lim_{|\alpha|\to\infty} I\alpha^{-1} = 0$$ one way to drive the trace-term to zero is to let $\alpha$ take on very large values.
It might be possible to find other real values for $\alpha$ which annihilate the trace, but those would depend on the properties of each specific $A$ matrix.