Consider a group $G$, a Hilbert space ${\mathbb{V}}$ with a dot product $\,\langle~,~\rangle\,$, and a space ${\cal{L}}^G_{\mathbb{V}}\,$ of functions: $$ {\cal{L}}^G_{\mathbb{V}}\;=\;\left\{~\varphi~\Big{|}~~~\varphi:\,~G\longrightarrow{\mathbb{V}}\,\right\}~~.\qquad\qquad\qquad (1) $$ Let $\,A\,$ be a representation of $\,G\,$ in the Hilbert space: $$ A~:\quad G~\longrightarrow~GL({\mathbb{V}})\;\;.\qquad\qquad\qquad\qquad\quad (2) $$ Fix the vectors $\,v,\,v^{\,\prime}\in{\mathbb{V}}\,$, treat $\,x\in G\,$ as a free variable, and define the function $$ \varphi(x)\;\equiv\;\langle\,v^{\,\prime}\,,\;A(x^{-1})\,v\,\rangle\;\;. $$
By changing the argument from $\,x\,$ to $\,x k\,$, where $\,k\in G\,$, we obtain $$ \varphi(xk)\;=\;\langle\,v^{\,\prime}\,,\;A(\,(xk)^{-1})\,v\,\rangle\;=\; \langle\,v^{\,\prime}\,,\;A(k^{-1})\,A(x^{-1})\,v\,\rangle \;\;. $$ Would it be legitimate of me to write this as $$ \varphi(xk)\;=\;D(k^{-1})\,\langle\,v^{\,\prime}\,,\;A(x^{-1})\,v\,\rangle\;=\;D(k^{-1})\,\varphi(x)~~, $$ where $D$ is acting in the space of functions and is somehow linked to $A$ acting in $\,{\mathbb{V}}\,$?
The middle part of your final line isn't sensible. $A(k^{-1})$ is a linear transformation with domain (and range) $\Bbb V$ so it cannot be applied to $<v', A(x^{-1})v>$ which is a real or complex number or, I guess, more generally, the coefficient field of $\Bbb V$. The same applies, for that matter, to the final element of the last line.