If I have a equation in the form
$${\lambda ^2}{I_N} + \lambda {M_1} + {M_2} = {0_N}$$
where ${I_N}$ is the identity matrix of order $N$, $M_1$ and $M_2$ are matrices of ($N\times N$) order and $\lambda \in \mathbb C$ belongs to the complex numbers .
What are the mathematical tools or the mathematical framework to solve this kind of equations?
For the equation to have a solution, your matrices $M_1$ and $M_2$ must necessarily commute. This would only be one requirement. To see this, consider the case for diagonalizable $M_2$ so that $ PM_2P^{-1} = D$ for some invertible $P$ and diagonal $D$.
\begin{align} P\left(\lambda^2 I_N + \lambda M_1 + M_2 = 0_N\right)P^{-1} \\ \lambda^2 I_N + \lambda PM_1P^{-1} + D = 0_N \end{align}
Here we can see that $M_1$ must not only have the same spectrum as $M_2$ (since otherwise the non-zero elements in the off-diagonal would not be canceled in the sum), but it must have appropriate eigenvalues such that the single $\lambda$ simultaneously solves for each diagonal term.
tl:dr: solve for $\lambda_1$ and $\lambda_2$ at any desired coordinate. Check if either one works globally. If not, then there is no solution.