I am reading the lecture notes. On page 14, example of $C_{c}^{\infty}(G)$. I am trying to show that the map $A$ takes $f$ to $g\mapsto f(g^{-1})$ is an invertible element of $\operatorname{Hom}_G(\lambda, \rho)$.
By the definition of invertible elements, we only need to show that $A$ satisfies the following identity $$ (\rho(g)A)(f)(x)=(A\lambda(g))(f)(x), x\in G. $$ We have $$ (\rho(g)A)(f)(x) = \rho(g)(A(f)(x))=f(g^{-1}x^{-1}), \quad (1) $$ $$ (A\lambda(g))(f)(x) = (Af(g^{-1}x))=f(x^{-1}g). \quad (2) $$ But (1) and (2) are different. I am confusing. Thank you very much.
Your second equation is not correct:
In fact, $$A(\lambda(g)f)(x)=A(f(g^{-1}x))=f(g^{-1}x^{-1}).$$
Why? Since $A$ takes $f$ to the function that sends $x$ to $f(x^{-1}),$ we find that $A(\lambda(g)f)$ sends $x$ to $\lambda(g)f(x^{-1})=f(g^{-1}x^{-1}).$
Hope this helps.