Is the following optimization problem convex?
$$\text{inf}\quad 5x_1^2 - 4x_1x_2 + 2x_2^2 + x_1x_3 \quad \text{ s.t.} \\ \quad x_1 - x_3 = 1 \\ \quad 4x_1^2 + 9x_2^2 - x_3^2 \leq 25 $$
I know that the first constraint is convex as it is linear. The second constraint is also convex as it is a cone. The intersection of two sets is convex, hence the feasible region is convex.
I calculated the Hessian of the problem and got this:
$$H(x_1, x_2, x_3 ) = \begin{bmatrix} 10 & -4 & 1 \\ -4 & 4 & 0 \\ 1 & 0 & 0 \end{bmatrix}$$
The determinant of the matrix is $-4$, hence it is negative definite and the problem is therefore not convex. Is this correct?
You almost got it right. A function is convex if and only if its Hessian is PSD. And a matrix is PSD iff all it's eigenvalues are non-negative.
The objective function is not convex since it has one negative eigenvalue ($\approx$-0.158). For $2\times 2$ matrices only, a matrix is PSD iff its determinant is non-negative.
The first constraint is indeed linear, hence it is convex.
The last constraint is again non-convex, as one of its eigenvalues is -2. This one is easy since the Hessian of the third constraint is a diagonal matrix, therefore it's eigenvalues are simply the diagonal entries.