I already know that $b_{n+1} = a_n +3b_n$ and $a_{n+1} = 3a_n - b_n$. So
$a_{n+2} = 3(3a_n-b_n)-(3b_n+a_n) = 9a_n-3b_n-3b_n-a_n = 8a_n-6b_n$ and
$b_{n+2} = 8b_n+6a_n$.
So we can rewrite the whole thing as
$8a_n-6b_n+r(3a_n-b_n)+sa_n = 8a_n-6b_n+r(3b_n+a_n)+sb_n$ which, in turn is:
$(-4r+s)a_n = (4r+s)b_n$.
The original problem states that $(3+i)^n = a_n+ib_n$ so I tried using n=1 so a = 3 and b = 1
so then I have $-12r+3s=4r+s$ so
$2s=16r$.
From here I'm stuck. Do I just try random options? Honestly, I don't feel like I've done it right so far.
It looks like you made a substitution error when writing $$8a_n−6b_n+r(3a_n−b_n)+sa_n=8a_n−6b_n+r(3b_n+a_n)+sb_n$$ Here, you substituted $8a_n-6b_n$ for $b_{n+2}$ instead of $8a_n+6b_n$. This in turn leads you to the incorrect conclusion that $2s=16r$. Regardless, you'll be left with a single equation for $r$ and $s$, which will always have infinite solutions. This is why you run into a dead end - you have to find a second equation for $r$ and $s$ to get something conclusive (doable, but a little tedious and roundabout).
Here's an alternate approach. Since $b_{n+2}+rb_{n+1}+sb_n=0$, $ib_{n+2}+irb_{n+1}+isb_n=0$. Then, we can write: $$\left(a_{n+2}+ib_{n+2}\right)+r\left(a_{n+1}+ib_{n+1}\right)+s\left(a_n+ib_n\right)=0$$ Then, we use $(3+i)^n=a_n+ib_n$ to write: $$(3+i)^{n+2}+r(3+i)^{n+1}+s(3+i)^n=0$$ This can be reduced to: $$(3+i)^2+r(3+i)+s=0$$ $$9+6i-1+3r+ir+s=0$$ $$(8+3r+s)+(6+r)i=0$$ This gives $r=-6$ and $s=10$.