Given an endomorphism $f$ of a vector space $V$, its characteristic polynomial, say $P(x)$, is defined as follows: $P(x) = \det(f -xI)$, where $I$ is the identity endomorphism. It is well known that, when we evaluate over $f$, we get $P(f) = 0$, as a consequence of Cayley-Hamilton's theorem.
But, working with the expression directly, we get $$P(f) = \det(f - fI) = \det(f - f) = \det(0) = 0$$ even without Cayley-Hamilton. What's wrong about this?
What's wrong is that $P(x)=\det(f-x\operatorname{Id})$ is a polynomial. Yes, you can compute $P(x_0)$ for a number $x_0$ by computing $\det(f-x_0\operatorname{Id})$, but $f$ is not a number; it's an endomorphism.
Conseder, for instance, $Q(x)=\operatorname{tr}(f-x\operatorname{Id})$. By your argument, $Q(f)=0$ too. But, in fact, since $Q(x)=\operatorname{tr}(f)-nx$, you actually have $Q(f)=(1-n)\operatorname{tr}(f)$.