I'd like to find a closed-form solution for $X,Y$ in $\mathbb{R}$ for the following system of equations:
$$\left\lbrace \begin{array}{ll} a + (X - Y) (2 + 5 X - Y) = 0 \\ b X (2 + 5 X) + Y (-2 c (1 + 3 X) + d Y) = 0\end{array} \right.$$
Using Mathematica, I can get a closed-form, however it's a few hundreds of pages long. My question is whether there is a chance of having a reasonably short expression of the roots. If not, can I get a closed-form approximation of the roots?
Such a general system of equations does not have "short" closed-form solutions, see e.g. John Hughes's comment:
However, it might be possible to approximate the solutions in some cases. Considering ranges of values of the parameters such that the system has a solution, and that the solution corresponds to $X$ (or $Y$) sufficiently close from $0$, the second equation can be approximated by a straight line.
To illustrate my point, here (graph below) are the solution curves of the first equation (blue) and second equation (yellow) for $a =b = -1.2$, $c=d=0.3$. Both curves are hyperbolas and one of their intersection is "in the vicinity" of $X=0$ (green dot). Solving the second equation for $Y$, choosing the right solution and approximating it by a first-order Taylor expansion in $X$ in the neighbourhood of $0$ gives: $Y=bX/c$. Solving the first equation for this value of $Y$, i.e. finding the intersection between the blue hyperbola and the red straight line, gives among others the following solution:
$$X_0=\dfrac{(b - c) c + \sqrt{(c- b) c^2 ((-1 + a) b + c - 5 a c)}}{(b - 5 c) (b - c)}$$ and $Y_0=bX_0/c$. This gives the red dot below, which is reasonably close from the green dot.
The exact intersection is at about $({0.0966, -0.330})$ [green dot] while the approximation is $(0.0864,-0.346)$ [red dot].
Of course, this can only be done for some specific values of $a,b,c,d$, but it's better than nothing.