I want to write the following coupled system in one variable (not solving it!).
$$ x'=ax+by \\ y'=cx+dy $$
The aim is to substitute with a $u=\alpha x+\beta y$ in a way that the system can be written as $u^{'}=K u$, where $K$ is some constant, some combination of $a,b,c,d$.
In the case $a=d=0$ the substitution can be $u=\sqrt{b}y+\sqrt{c}x$, but I cannot find the right coefficients when $a,d$ are non-zero.
Note: I am not trying to solve it, rather find a suitable substitution.
Let $X = \left(\begin{array}{c}x \\ y\end{array}\right)$ and $A = \left(\begin{array}{cc}a&b \\ c&d\end{array}\right)$, then $X'=AX.$ Now, diagonalize $A$ (i.e. write it as $P\Lambda P^{-1}$, where $\Lambda= \left(\begin{array}{cc}\lambda_1&0 \\ 0&\lambda_2\end{array}\right)$ is the matrix of eigenvalues of $A$, and $P$ is a matrix whose columns are the eigenvectors of $A$).
Define $U:=P^{-1}X$, then $U' = P^{-1}X' = PAX = P^{-1}P\Lambda P^{-1} X=\Lambda U$. In particular both $u_1' = \lambda_1 u_1$ and $u_2' = \lambda_2 u_2$ are answers to your question.
More explicitly: $$ P = \left(\begin{array}{cc}\frac{a-d-\sqrt{(a+d)^2-4 (a d- b c)}}{2 c}&\frac{a-d+\sqrt{(a+d)^2-4 (a d- b c)}}{2 c} \\ 1&1\end{array}\right). $$ $$U = P^{-1}X = \frac1{\sqrt{(a+d)^2-4(ad-bc)}}\left(\begin{array}{c}-cx+\frac{a-d+\sqrt{(a+d)^2-4 (a d- b c)}}2 y \\ cx-\frac{a-d-\sqrt{(a+d)^2-4 (a d- b c)}}2 y\end{array}\right)$$ $$ \lambda_1 = \frac12\left(d+a-\sqrt{(a+d)^2-4 (a d- b c)}\right),\quad \lambda_2 = \frac12\left(d+a+\sqrt{(a+d)^2-4 (a d- b c)}\right).$$