I believe I am having some trouble understanding the difference between a system of recurrence relations and a linear system of linear/nonlinear equations, and I could not find much information about systems of recurrece relations..
I have the following system for the recurrence relations $(F_n)_{n\geq 1}$ and $(G_n)_{n \geq 1}$, where the coefficients $f_1(n), f_2(n), g_1(n)$ and $g_2(n)$ are nonlinear functions of $n$ and other two parameters
\begin{align} F_{n+1} &=f_1(n)F_{n} +f_2(n-1)F_{n-1}\\ G_{n-1}&=g_1(n-1)F_{n-1}+g_2(n)F_{n}\\ G_n&=h_1(n-1)F_{n-1}+h_2(n)F_{n}\\ \end{align}
The goal is to study wether this system has a solution $\neq 0$ or not. The only restriction I have on the boundary conditions is $F_1\neq 0$.
Translating the second relation we get $$G_{n}=g_1(n)F_{n}+g_2(n+1)F_{n+1}$$
Since this relation must be consistent with the last one, equating those two we have
$$F_{n+1}=\frac{h_2(n)-g_1(n)}{g_2(n+1)}F_n+\frac{h_1(n-1)}{g_2(n+1)}F_{n-1}$$
The problem is that looking at the first relation we have $f_1(n)\neq\frac{h_2(n)-g_1(n)}{g_2(n+1)}$ and $f_2(n-1)\neq \frac{h_1(n-1)}{g_2(n+1)}$, but setting
$$\frac{h_2(n)-g_1(n)}{g_2(n+1)}F_n+\frac{h_1(n-1)}{g_2(n+1)}F_{n-1}=F_{n+1} =f_1(n)F_{n} +f_2(n-1)F_{n-1}$$
we get that
$$F_n=\left(\frac{h_2(n)-g_1(n)}{g_2(n+1)}-f_1(n)\right)^{-1}\left(f_2(n-1)-\frac{h_1(n-1)}{g_2(n+1)}\right)F_{n-1}$$
This last expression for $F_n$ indeed satisfies the system and is simpler to work with, but I'm not sure if I can do these computations, or if since the coefficients for the two different expressions for $F_n$ do not match there is not much I can do.