We have the model $y= ax(1-x)$ and we want to find the period 2 solutions such that $X_{n} = X_{n+2} $ and $X_n \neq X_{n+1}$. My teacher told us to do this problem with the quadratic formula. This is my attempt:
$y = ax (1-x)$, so for period 2 we'd want $x = a(ax(1-x))(1-ax(1-x))$. Factor out the parentheses and you get $$x = -a^3x^4 + 2a^3x^3-a^3x^2 -a^2x^2 + a^2x$$
Here I am confused because I am supposed to use a quadratic formula for a quartic function.
The two fixed points, $z_0 = 0$ and $z_1 = (a-1)/a$ are also solutions of $X_{n+2} = X_n$, hence you can divide those two out of the equation
$$a^3x^4 - 2a^3x^3 +a^2(a+1)x^2 - (a^2-1)x = 0$$
(you have miscalculated the coefficient of $x^2$), to obtain the quadratic equation
$$a^2x^2 - a(a+1)x + (a+1) = 0$$
for the two points of the 2-cycle.
First, for the fixed points, we have the equation $x = ax(1-x)$, which, expanded and rearranged becomes
$$ ax^2 - (a-1)x = 0$$
and we can read off the two solutions $z_0 = 0$ and $z_1 = (a-1)/a$ immediately.
Then, for the points with $X_{n+2} = X_n$, we have the equation $x = f(f(x))$, which, substituting $f(w) = aw(1-w)$ expands to
$$x = af(x)\bigl(1-f(x)\bigr) = a^2x(1-x)\bigl(1-ax(1-x)\bigr) = a^2x\bigl(1-x -ax(1-x) + ax^2(1-x)\bigr)$$
and then
$$x = a^2x(1 -(a+1)x +2ax^2 - ax^3) = x(-a^3x^3 + 2a^3x^2 - (a+1)a^2x + a^2).$$
Rearranging yields
$$x\bigl(a^3x^3 - 2a^3x^2 + (a+1)a^2x - (a^2-1)\bigr) = 0.$$
We already know that $z_0$ and $z_1$ are solutions (I have already factored out $x - z_0$), and thus we can write the polynomial expression on the left hand side as $(x - z_0)(x - z_1)Q(x)$ with a quadratic polynomial $Q$. Now we split off the factor $x - z_1$, or, better $a\cdot(x - z_1) = ax - (a-1)$ from the cubic factor.
$$\bigl(a^3x^3 - 2a^3x^2 + (a+1)a^2x - (a^2-1)\bigr) = \bigl(ax - (a-1)\bigr)\cdot Q(x)$$
To determine $Q$, we start with the highest-order term, and it's clear that must be $a^2x^2$ to obtain the highest-order term $a^3x^3$ of the cubic polynomial. $\bigl(ax - (a-1)\bigr)\cdot a^2x^2 = a^3x^3 - a^2(a-1)x^2$, so
$$\bigl(a^3x^3 - 2a^3x^2 + (a+1)a^2x - (a^2-1)\bigr) = \bigl(ax-(a-1)\bigr)\cdot a^2x^2 + R(x)$$
with $R(x) = -a^2(a+1)x^2 + a^2(a+1)x - (a^2-1)$. Splitting off $ax - (a-1)$ from $R(x)$ yields the next term of $Q(x)$, namely $-a(a+1)x$.
$$\bigl(ax - (a-1)\bigr)\cdot a(a+1)x = a^2(a+1)x^2 -a(a+1)(a-1)x$$
then gives us
$$\bigl(a^3x^3 - 2a^3x^2 + (a+1)a^2x - (a^2-1)\bigr) = \bigl(ax - (a-1)\bigr)\cdot\bigl(a^2x^2 - a(a+1)x\bigr) + S(x)$$
with $S(x) = a(a+1)x - (a^2-1) = (a+1)\bigl(ax - (a-1)\bigr)$, so wrapping up, we obtain the factorisation
$$x\bigl(ax - (a-1)\bigr)\cdot\bigl(a^2x^2 - a(a+1)x + (a+1)\bigr) = 0$$
and the two zeros of the last factor are the points with $X_{n+2} = X_n \neq X_{n+1}$.