I'm reading an explanation of the group of operations on functions in a vector space and I'm confused about the order of operations. The explanation can be paraphrased as:
Let $U_i$ be an element of the group of transformations $G$ that act on elements, $x$, of a vector space $V$. A function that acts on such a vector space is denoted $f(x)$. What can we say about the the action of $f$ on the transformed vector $U_i^{-1} x$? Let us define an operator $\mathscr{U}_i$, which acts on functions of $x$ in such a manner that for all $f$
$$\mathscr{U}_if(x) = f(U_i^{-1}x).$$
Now consider the quantity $\mathscr{U}_i\mathscr{U}_jf(x)$. We have from the above equation
$$\mathscr{U}_i\mathscr{U}_jf(x) = \mathscr{U}_i f(U_j^{-1}x) = \mathbf{f(U_j^{-1}U_i^{-1}x)} = f([U_iU_j]^{-1}x). $$
Question: Going from the second to the third term (in bold), why does $U_i^{-1}$ act on x instead of on the newly transformed term? I reason that $U_j$ acts on $x$ and then $U_j$ acts on the transformed $x$, not $x$ itself, written as
$$ U_i^{-1}(U_j^{-1}x) = U_i^{-1}y = U_i^{-1}U_j^{-1}x = [U_j^{-1}U_i^{-1}]x, $$ where $y$ is the transformed $x$. Yet this cannot be correct because the text then uses this result to say that if $U_iU_j = U_k$ then $\mathscr{U_iU_j} = \mathscr{U_k}$ so the elements $\mathscr{U_1}, \mathscr{U_2}, ...$ form a group $\mathscr{G}$ that is isomorphic to $G$.
Reference
pg. 600 in Group Theory chapter: Byron & Fuller, Mathematics of Classical and Quantum Physics, Dover Publications, 1970.
In your bold term I would expect a different order of $i$ and $j$. The answer to your question is then: $U_i^{-1}$ acts on the transformed $U_j^{-1}x$ which you called $y$. Now, if $U_iU_j=U_k$ then $\mathscr{U}_kf(x)=f(U_k^{-1}x)=f(U_j^{-1}U_i^{-1}x)=\mathscr{U}_j\mathscr{U}_if(x)$. Here, $i$ and $j$ get reversed. In other words, the group element in $\mathscr{G}$ that corresponds to $U_iU_j$ is $\mathscr{U}_j\mathscr{U}_i$. It looks like the book just has a typo that flips the order of $i$ and $j$.