I have an optimization problem in which I need to perform gradient descent in a space of matrices. I'd like to understand how to perform the gradient updates while remaining in the space. If my matrices formed a Lie group, I know there can be ways to follow geodesics, but I don't think this is the case.
My matrices are complex symmetric and they can be decomposed as: $$ A = U D U^T $$ where $U$ is some unitary matrix (note there's a transpose on the right, not a conjugate transpose) and where $D$ is a diagonal matrix of real values in the interval $[-1,1]$ (extremes excluded). Also, note that the diagonal of $D$ does not contain the eigenvalues, the decomposition above is just a way of constructing $A$.
If I perform gradient updates naively as $A \leftarrow A - \epsilon\frac{\partial L}{\partial A^*}$ I eventually break the condition that the elements of $D$ should remain bounded between -1 and 1.
- Do the $A$ matrices form a known group?
- How should I perform the gradient updates?