I am very confused about factor groups of polynomials and I would appreciate some help.
I know that in the coset representation $x^2=-x-3$ and so all cosets are in the form $ax+b+A$. Now let us take the coset $x+1+A$ we know it is not the zero element as $x+1 \not \in A$. Now $$ax+b+A=(a(x+1)+b-a)+A=(x+1)+A +(b-a)+A$$ and so when $b=a$ we get $(x+1)+A$. Hence from $25$ elements, we get $21$.
I am almost sure this is wrong, and I am confused what I am doing wrong. Any help would be appreciated. Thanks!
As long as $\,f(x)$ is a monic polynomial of degree $2$, $\Bbb F_5[x]/\langle f\rangle$ will be a vector space of dimension two over $\Bbb F_5$, and thus have $25$ elements.
Indeed, calling $\langle f\rangle=A$, as you did, we see that $1+A$ and $x+A$ make up a basis of the factor group: if $\alpha+\beta x+A$ is zero in the factor group, then $\alpha+\beta x\in A$, i.e. $\alpha+\beta x=g\cdot f$ for some polynomial $g\in\Bbb F_5[x]$, impossible unless $g=0$, which says that $\alpha=\beta=0$, which shows linear independence of $1+A$ and $x+A$. That every $Z\in\Bbb F_5[x]/A$ is of form $\alpha+\beta x+A$ follows by applying Euclidean division to any polynomial $z(x)$ for which $z+A=Z$.