Starting with some vector $x$, I am performing two operations: First, I convolve $x$ with another vector $g$ to compute $x*g$, where $~*~$ denotes convolution. Second, I pointwise multiply the result with a discretized rectangular function to compute $(x*g) \cdot r$, where $~\cdot~$ denotes point-wise product.
I know that point-wise multiplication with a rectangular function is equivalent to convolution with a sinc function (by the convolution theorem), so really we are finding $(x*g)*R$. And we also know that convolution commutes, so the answer should be equivalent to $(x*R)*g$, which could in turn be rewritten as $(x \cdot r)*g$. But it is obviously not the case:
For example, when $g$ is a discretized Gaussian, then you are smoothing $x$ with "Gaussian blur" and then clipping everything outside some window (specified by $r$ to be zero). Whereas, clipping $x$ and then smoothing does not include the influence that the clipped-off points can have on the smoothing (a single high-intensity peak that is clipped off would have multiplied with the tail of a Gaussian to influence a point in the actual result).
Who can figure out where my reasoning is (obviously) wrong?
You're a little bit off with your interpretation of the convolution theorem. In particular, what we really have is as follows: $$ \newcommand{\F}{\mathscr{F}} \F [(x*g)\cdot r] = \F(x*g) * \F(r) = \F(x*g) * R $$ Where $\F$ here denotes the Fourier transform. The reason things work out so nicely when your vector is a Gaussian is that the Fourier transform of a Gaussian is another Gaussian.
So, on the one hand, we would have $$ \F[(x*g)\cdot r] = X G*R $$ On the other hand, $$ \F[(x*r)\cdot g] = XR * G $$ Ultimately, it is this successive multiplication and convolution that cannot be interchanged.