So plugging in $1$ gives $f(1) = 0$ which means $1$ is a root and $f$ has a factor $(x-1)$ which is $\equiv (x+4)$ in $\mathbb Z_{5}[x]$ ?
I then divide $f(x)$ by $(x+4)$ using polynomial long division and I get
$x^4-4x^3+16x^2-64x+255$ with remainder $-1020$
And that's looking kinda strange.. not sure what I'm doing at this point
I notice $f(0)$ also $= 0$ so maybe if I divide $f(x)$ by $x$ instead I would get $(x^4-1)$ but then I'm still stuck and don't know what else to do
You started out on a good track, but the coefficients of your division result are in $\mathbb Z$ instead of $\mathbb Z_5$.
The key to doing this more efficiently is Fermat's little theorem, which states that all residues $\bmod\,5$ are roots of this polynomial, so it factors as $x(x-1)(x-2)(x-3)(x-4)$.