Analytical solution to an optimization problem

Question

Say I have a matrix variable, $X$, and an objective function $f(X)$, which I am minimizing (or maximizing; for this question it doesn't make a difference).

I take the (partial) derivative with respect to one of the matrix elements, say, $X_{i,j}$ and set such derivative equal to 0. Then I solve for $X_{i,j}$ and find that it depends on other elements of the matrix (e.g. $X_{i,j-1}$, $X_{i,j+1}$, and/or others).

Does this mean that there is no analytical solution to the optimization problem, when optimizing with respect to the entire matrix $X$ (since to explicitly determine $X_{i,j}$, you must already know the values of other matrix elements, which themselves might depend on $X_{i,j}$)?

Answer 1

No, of course it doesn't mean that. You want to take the partial derivatives with respect to each $X_{ij}$ and solve the resulting system for all $X_{ij}$.

Answer 2

Usually you won't have analytic solutions, but the technique for solving these kinds of problems analytically is often to take the derivative of $f$ with respect to the whole matrix $X$ instead of an element, and set that equal to the zero matrix. Reference the bible, and this helpful calculator online to apply the technique to your specific problem.

Answer 3

In this case, $f$ must be a scalar valued function, thus it shouldn't matter if $X$ is a matrix or a vector.

Yes, the gradient being zero $\nabla f=0$ should normally give you a set of algebraic equations, which may of course be non-linear or transcendental, depending on $f$, while this is normal, it's also bothersome because finding a solution may be difficult, and there is no general analytical method for it. Fortunately, there's usually plenty of numerical methods that may be suitable for your need.

Yet, you may also need to check the hessian matrix $Hf$ in order to check if the solution corresponds to a minima, maxima or saddle point.