Find all real value of $ k $ such that the system of equation \begin{align} a^2 + ab &= kc^2 \\ b^2 + bc &= ka^2 \\ c^2 + ca &= kb^2 \end{align} have positive real number solution for $ a $, $ b $, and $ c $.
I have already found the value of $ k $. Here is my solution. Adding all of the equation, we found that $$ a^2 + b^2 + c^2 + ab + bc + ca = k\left(a^2 + b^2 + c^2\right) $$ or $$ k - 1 = \frac{ab + bc + ca}{a^2 + b^2 + c^2}. $$ Since $ ab + bc + ca \leq a^2 + b^2 + c^2 $ for $ a, b, c \in \mathbb{R^{+}} $, $$ k - 1 \leq \frac{ab + bc + ca}{ab + bc + ca} = 1 \iff k \leq 2. $$ My question is, is my solution is wrong? Or did I missed something? Thanks.
You have proved that $k \le 2$ is necessary, but have not shown it is sufficient.
If $k=-1$, we have three equations like $a^2+ab=-c^2$. This cannot be satisfied with $a,b,c$ positive. This applies for any $k \le 0$.
Now it remains to show that there is a solution for any $k$ with $0 \lt k \le 2$
As the equations are homogeneous, for any solution $(a,b,c)$ you have $(da,db,dc)$ is also a solution, so you can choose $d=\frac 1a$ or (equivalently) set $a=1$. That reduces you to two unknowns and gives you an easy substitution.