Say I want to put action of $S_4$ on $V^{\otimes 4}$. If I want to act on the left, why can't I say
$$\sigma(v_1 \otimes v_2 \otimes v_3 \otimes v_4) = v_{\sigma(1)} \otimes \ldots \otimes v_{\sigma(4)}?$$
Everything seems to work out right e.g. take $\sigma = (123),\tau = (134)$. Then $\sigma\tau = 234$.
$$(\sigma\tau)(v_1 \otimes \ldots \otimes v_4) = v_1 \otimes v_3 \otimes v_4 \otimes v_2$$
and $\sigma\big( \tau(v_1 \otimes \ldots \otimes v_4) \big)$ is also the same thing. But books I see say we have to throw in an inverse for the left action? I am getting confused because it seems throwing the inverse does not make it an action! What's happening?
You aren't computing $\sigma(\tau(v_1 \otimes ... \otimes v_4))$ correctly. We have
$$\tau(v_1 \otimes ... \otimes v_4) = v_3 \otimes v_2 \otimes v_4 \otimes v_1.$$
Now rename $w_1 = v_3, w_2 = v_2, w_3 = v_4, w_4 = v_1$. Then
$$\sigma(w_1 \otimes ... \otimes w_4) = w_2 \otimes w_3 \otimes w_1 \otimes w_4$$
which is equal to
$$v_2 \otimes v_4 \otimes v_3 \otimes v_1$$
and so is not equal to $(\sigma \tau)(v_1 \otimes ... \otimes v_4)$. It is, however, equal to $(\tau \sigma)(v_1 \otimes ... \otimes v_4)$.
When you apply $\sigma$ after applying $\tau$, $\sigma$ doesn't remember what indices the $v_i$ had originally. It only sees their current positions, and then it permutes those.