I am currently trying to compute the maximal possible correlation between a predictor and a latent variable in a structural equation model given the amount of predictors, the collinearity between them and the multiple determination coefficient $R^2$. If necessary, all predictor effect sizes could be constraint to be equivalent. I found that the formula for $R^2$ is as follows $$R^2 = c^TR^{-1}_{xx}c,$$
with $R^{-1}_{xx}$ being the inverse of the correlation matrix among the predictors and $c$ being the vector of the correlation between the predictors and the outcome variable. Is there a way to solve for $c$?
Ignoring all the setup, it appears that you're asking whether given an $n \times n$ square matrix (which you've called $R_{xx}^{-1}$, but I'll call $M$, and a nonnegative number $H$ (which you've called $R^2$), can we solve for $c$ in $$ H = c^t M c. $$
The answer is evidently no, for if $M$ is the identity, and $H = 1$, then $c$ can be any point on the unit sphere.
Even if that were not the case, the function $$ f: \Bbb R^n \to \Bbb R: c \mapsto c^t M c $$ takes $\Bbb R^n$ to $\Bbb R$ smoothly. Unless $M$ is the zero matrix, you'd expect this to be a submersion at almost every point of $\Bbb R$, so that the preimage of a point would be an $(n-1)$-dimensional manifold. In particular, you'd expect $$ f^{-1}(R^2) $$ to be an $(n-1)$-dimensional manifold, hence consist of more than one point.