I want to solve a simltaneous equation below.
$\left(K+\lambda_1I\right)\alpha = \frac{1}{n}\left(y-Ah\right)$
$\left(A^2+\lambda_2I\right)h=A\left(y-nK\alpha\right)$
Here, $n, \lambda_1, \lambda_2>0$ are scalar. $\alpha, h$, and $y \in \mathbb{R}^n$, $A \in \mathbb{R}^{n\times n}$.
$K \in \mathbb{R}^{n\times n}$ is positive definite.
I want to solve for $\alpha$ and $h$, fixing other variables.
My answer is $\alpha=\frac{\lambda_2}{n}\left(\lambda_1\left(A^2+\lambda_2I\right)+\lambda_2K\right)^{-1}y$, and $h=\lambda_1 A \left(\lambda_1\left(A^2+\lambda_2I\right)+\lambda_2K\right)^{-1}y$. Is this correct and unique solution?
Write as
$$\begin{cases}\left(K+\lambda_1I\right)&\alpha+\frac{1}{n}A&h& = \frac{1}{n}y\\ nA^2K&\alpha+\left(A^2+\lambda_2I\right)&h&=Ay\end{cases}$$ which is an ordinary linear system.
If you solve it block-wise, for instance by Gaussian elimination, you will see negative terms appear, so I don't think that your solution is correct.