Let $G$ be some locally compact group and $\mu$ its associated Haar measure. I am trying to adapt this proof that convolution on the locally compact group $(\Bbb{R},+)$ is associative. Here's what I have so far:
$$((f \ast g) \ast h)(u) = \int_{G} (f \ast g)(x)h(x^{-1}u) ~d \mu (x)$$
$$= \int_{G} \left[ \int_{G} f(y) g(y^{-x}y) ~d \mu (y) \right] h(x^{-1}u) ~d \mu (x)$$
$$= \int_{G} \int_{G} f(y) g(y^{-1}x)h(x^{-1}u) ~d \mu (y)~ d \mu (x)$$
$$= \int_{G} \int_{G} f(y) g(y^{-1}x)h(x^{-1}u) ~d \mu (x)~ d \mu (y) ~~~~~~~~~~~\text{Fubini's theorem}$$
$$= \int_{G} f(y) \left[\int_{G} g(y^{-1}x) h(x^{-1}u) ~d \mu (x) \right] ~ d \mu (y) $$
So far all of this seems fine; just a literal translation from the abelian group case to the arbitrary group case. However, the part where we rewrite the inner integral is giving me some trouble. From what I make of it, they are doing the following:
$$\int_{G} g(y^{-1}x) h(x^{-1}u) ~d \mu (x) = \int_{G} g(yy^{-1}x) h (yx^{-1}u) ~ d \mu (x) ~~~~~~~~~~~\text{Translation Invariance}$$
$$ = \int_{G} g(x) h(yx^{-1}u) ~ d \mu (x)$$
But I'm having a hard time seeing why this is equal to $(g \ast h)(y^{-1}u) = \int_{G} g(x) h(x^{-1}y^{-1}u) ~ d \mu (x)$
First, I think there is a mistake in your question: in the second line, it should be $ g(y^{-1}x) $ instead of $ g(y^{-x}y) $.
You made a mistake in the substitution of $ x^{-1} $ in the function $ h $. Your idea is right, I just make the step more explicit. I'll make a substiution $ x=yx'$ or, equivalently, $ x^{-1} = x'^{-1}y^{-1} $:
\begin{align*} \int_{G} g(y^{-1}x) h(x^{-1}u) d\mu(x) &= \int_{G} g(y^{-1}yx') h(x'^{-1}y^{-1}u) d\mu(x') \\ &= \int_{G} g(x')h(x'^{-1}y^{-1}u) d\mu(x) = g \ast h (y^{-1}u) \end{align*}
If you now insert this in your equation in the last line, you're done.