I have ben having this confusion for quite some time. Here is my question. For simplicity, suppose we have two r.v.'s $X_1$ and $X_2$ defined on $\left(\Omega,\mathcal{A},P\right)$ and they are independent. I am interested in, say, finding an upper bound for $\mathbb{E}\left[\left|X_1 + X_2 \right|\right]$.
In statistics/machine learning literature, often I see arguments go like this: We "first treat $X_1$ as constant", and $\mathbb{E}\left[ \left|x_1 + X_2\right| \right]$ is upper bounded by a constant/function depends on $x_1$, and then we treat $x_1$ as a random variable and "take expectation wrt to $X_1$".
I have been self-studying measure theory based probability, and I have been trying to make this kind of argument rigorous but cannot achieve it. I am aware of Fubini Theorem, but we do not necessarily have product space/measure here. I am aware of conditional probability defined through Radon-Nikodym, but I don't see any connection. It seems that this kind of argument is using some sort of "iterated integral" but I can't formalize it.
It is the law of total expectation $$E[Z] = \int_{x=-\infty}^{\infty} E[Z|X=x]f_X(x)dx$$ and you can prove that by using a definition $$E[Z|X=x] = \int_{z=-\infty}^{\infty}z f_{Z|X}(z|X=x)dz$$ with $f_{Z|X}(z|X=x) = \frac{f_{X,Z}(x,z)}{f_X(x)}$. Here I am asssuming densities exist for simplicity.
Alternatively you can call it the "tower property" or "iterated expectations": $$E[Z] = E[E[Z|X]]$$ and a formal proof essentially is a 1-line application of the measure theory definition of $E[Z|X]$ as a conditional expectation given the sigma-algebra defined by random variable $X$, noting that the full space $\Omega$ is in $\sigma(X)$: $$ \underbrace{\int_{\Omega} E[Z|\sigma(X)] dP}_{E[E[Z|X]]} = \underbrace{\int_{\Omega} Z dP}_{E[Z]} $$