My approach:
I know only one way to relate inradius and sides of a triangle which is $\text{Area of triangle = (inradius)(semi-perimeter)}$
I am trying to get $RS$ and altitude of $\Delta AMB$ in terms of $r$ which equals to $b$ and $a/2 \; $ resp. Tell me if there's better way of doing this question
For convenience, define $p = a/2$ and $q = b/2$. Furthermore, let $MS = h$. Then $$|\triangle MBS| = hp = r\left(p + \sqrt{p^2 + h^2}\right),$$ hence $$h = \frac{2p^2 r}{p^2 - r^2}, \tag{1}$$ and $$MB = \sqrt{p^2 + h^2} = \frac{p(p^2 + r^2)}{p^2 - r^2}. \tag{2}$$ Similarly, by substituting $3r$ for $r$, $$b - h = MR = \frac{2p^2 (3r)}{p^2 - (3r)^2}, \quad MA = \frac{p(p^2 + (3r)^2)}{p^2 - (3r)^2}. \tag{3}$$ Hence $$|\triangle AMB| = pq = (2r)\frac{MA + MB + MR + MS}{2} = \frac{2pr(p^2 - 3r^2)}{(p-3r)(p-r)}. \tag{4}$$ Again letting $q = (MR + MS)/2$, we obtain $$pr(p^2 - 3r^2)(p^2 - 4pr - 3r^2) = 0, \tag{5}$$ and discarding obvious extraneous roots, we have $$p = r \sqrt{3}, \quad p = r(2 + \sqrt{7}). \tag{6}$$ But we can also eliminate $p = r \sqrt{3}$ since this would give $|\triangle AMB| = 0$ in $(4)$. The rest is straightforward and left as an exercise; we obtain $$\frac{a}{b} = \frac{p}{q} = \frac{3}{4}.$$