Given the dynamical system
$\dot x = f(x,y)$ and $\dot y = g(x,y)$
where f and g are homogeneous of degree n (i.e. $G(\alpha x,\alpha y)=\alpha^n G(x,y))$
show that straight line solutions of the form $y(t)=mx(t)$ exist for the system if and only if
$mf(1,m) = g(1,m)$
Starting with a simple example
$\dot x = 2x+y$ and $\dot y = x+y$
Both functions are homogeneous of degree 1 since
$f(\alpha x, \alpha y) = 2\alpha x + \alpha y = \alpha (2x+y) = \alpha f(x,y) $
and
$g(\alpha x, \alpha y) = \alpha x + \alpha y = \alpha (x+y) = \alpha g(x,y) $
If you want straight line solutions, then:
$mf(1,m) = g(1,m)$
$m(2+m) = 1+m$
$m=0.618, -1.618$
I see how the solutions are generated but how can I go about expressing this more formally for any function that is homogeneous of degree n
There are solutions satisfying $y(t)=mx(t)$ if and only if $$ g(x,mx)=mf(x,mx) $$ for all $x$. This is because solutions of that form exists if and only if the vector field along the corresponding line has the same direction. Being homogeneous of degree $n$ gives you the equation $$ g(1,m)=mf(1,m) $$ after canceling out $x^n$.