The claim is not true as stated. The function $f(X)=0 \forall X$ is a counterexample.
With the extra assumption that $f$ is not constant function, the claim holds. Here are the hints:
Hint 1: Show $f(0)=0$.
Hint 2: If $f(Y) \neq 0$ for some $Y$ set $A=Y, B=I$. Deduce that $f(I) \neq 0$.
Hint 3: If $A$ is invertible set $B=A^{-1}$.
Hint 4: If $A$ is a diagonal matrix with $n-1$ ones and a zero on the diagonal, show that $f(A)=0$.
To do this, consider all such matrices $A_1,..., A_n$. Show that for each $j$ there exists some $P$ such that $A_j=PA_1P^{-1}$, and deduce that $f(A_j)=f(A_1)$.
What is $f(A_1...A_n)=?$
Hint 5 Use hint $4$ to deduce that if $A$ is a diagonal matrix with at least a zero on diagonal then $f(A)=0$.
Hint 6: If $A$ is not invertible show that there exists some $D,B$ with $D$ diagonal and $B$ invertible so that $A=DB$.
The claim is not true as stated. The function $f(X)=0 \forall X$ is a counterexample.
With the extra assumption that $f$ is not constant function, the claim holds. Here are the hints:
Hint 1: Show $f(0)=0$.
Hint 2: If $f(Y) \neq 0$ for some $Y$ set $A=Y, B=I$. Deduce that $f(I) \neq 0$.
Hint 3: If $A$ is invertible set $B=A^{-1}$.
Hint 4: If $A$ is a diagonal matrix with $n-1$ ones and a zero on the diagonal, show that $f(A)=0$.
To do this, consider all such matrices $A_1,..., A_n$. Show that for each $j$ there exists some $P$ such that $A_j=PA_1P^{-1}$, and deduce that $f(A_j)=f(A_1)$.
What is $f(A_1...A_n)=?$
Hint 5 Use hint $4$ to deduce that if $A$ is a diagonal matrix with at least a zero on diagonal then $f(A)=0$.
Hint 6: If $A$ is not invertible show that there exists some $D,B$ with $D$ diagonal and $B$ invertible so that $A=DB$.