- Suppose $\alpha$, $\beta$ are increasing functions on $[a, b]$ such that $\alpha \in \mathscr{R}(\beta)$, and $f \in \mathscr{R}(\alpha) \cap \mathscr{R}(\beta)$. Show that $\int_{a}^{x} f(t) d \alpha(t)$, regarded as a function of $x$, belongs to $\mathscr{R}(\beta)$, and that \begin{equation*} \int_{t=a}^{b}\bigg(\int_{s=a}^{t} f(s) d \alpha(s)\bigg) d \beta(t) = \int_{t=a}^{b}(\beta(b) - \beta(t)) f(t) d \alpha(t) \end{equation*}
- Show that given continuous functions $u$ and $v$ on $[a, b]$, there exists a continuous function $w$ of two variables such that for any $f \in \mathscr{R}$ on $[a, b]$, one has \begin{equation*} \int_{t=a}^{x} u(t)\bigg(\int_{s=a}^{t} v(s) f(s) d s\bigg) d t = \int_{t=a}^{x} w(x, t) f(t) d t \end{equation*}
It has a clue for the 1st bullet, which is as follows,
First consider the case where $f$ is everywhere $ \geq 0$, since in this case, one can easily describe the least and greatest values of $\int_{s=a}^t f(s) d \alpha(s)$ on any interval as the values at the respective ends of the interval; then get the general case by writing any $f \in \mathscr{R}(\alpha) \cap \mathscr{R}(\beta)$ as a difference of two nonnegative-valued functions in $\mathscr{R}(\alpha) \cap \mathscr{R}(\beta)$.
For the 1st bullet, I know the result holds when $\alpha$ and $\beta$ are differentiable, but I have no idea how to prove it for the general case. Also, I have no clue about the 2nd bullet. This exercise is raised based on baby Rudin's first 6 chapters, so please do not resort to any advanced methods.
Define $F(x) = \int_a^xf(t) \, d\alpha(t)$. Following the hint and starting with the case where $f$ is nonnegative, we can show that $F$ is Riemann-Stieltjes integrable with respect to $\beta$ by showing that Riemann's condition holds. That is, for every $\epsilon >0$ there exists a partition $P$ of $[a,b]$ such that the difference between upper and lower sums of $F$ with respect to $\beta$ satisifes $U(P,F,\beta) - L(P,F,\beta) < \epsilon$.
For the partition $P: a = x_0 < x_1 < \ldots < x_n=b$ we have (for $f \geqslant 0$)
$$\tag{1}U(P,F,\beta) - L(P,F,\beta)= \sum_{j=1}^n[\sup_{x\in [x_{j-1},x_j]}F(x)-\inf_{x\in [x_{j-1},x_j]}F(x)]\,[\beta(x_j)- \beta(x_{j-1})]\\= \sum_{j=1}^n \int_{x_{j-1}}^{x_j}f(t) \, d\alpha(t)\,[\beta(x_j)- \beta(x_{j-1})]$$
SInce $f$ is Riemann-Stieltjes integrable it is bounded and there exists $M_f > 0$ such that $0 \leqslant f(x) \leqslant M_f$ for all $x \in [a,b]$ and
$$\tag{2}\int_{x_{j-1}}^{x_j}f(t) \, d\alpha(t)\leqslant M_f [ \alpha(x_{j}) - \alpha(x_{j-1})]$$
Together (1) and (2) imply
$$\tag{3}U(P,F,\beta) - L(P,F,\beta)\leqslant M_f \sum_{j=1}^n[ \alpha(x_{j}) - \alpha(x_{j-1})][ \beta(x_{j}) - \beta(x_{j-1})]$$
We can proceed in either of two ways to show that there exists a partition $P$ such that the sum on the RHS of (3) is less than $\epsilon/ M_f$. One way is to notice that the sum is the difference between the upper and lower sums of $\alpha$ with respect to $\beta$ and, since $\alpha \in \mathscr{R}(\beta)$, the difference can be made as small as desired with an appropriately fine partition. The other way is to use the fact that when $\alpha$ is Riemann-Stieltjes integrable with respect to $\beta$, then both functions cannot be discontinuous from the right or left together. Therefore, there exists a partition $P$ such that $U(P,F,\beta) - L(P,F,\beta) < \epsilon$ and $F \in \mathscr{R}(\beta)$.
Leaving the remaining details of the proof for the general case to you, we can complete the first part by noting that
$$\int_a^b \left(\int_a^t f(s) \, d\alpha(s)\right) \, d\beta(t)= \int_a^b \left(\int_a^b f(s) \mathbf{1}_{s \leqslant t}(s,t) \, d\alpha(s)\right) \, d\beta(t),$$
where the indicator function $\mathbf{1}_{s \leqslant t}(s,t) = 1$ if $s \leqslant t$ and $0$, otherwise. Applying Fubini's theorem it follows that
$$\int_a^b \left(\int_a^t f(s) \, d\alpha(s)\right) \, d\beta(t)=\int_a^b f(s)\left(\int_a^b \mathbf{1}_{s \leqslant t}(s,t) \, d\beta(t)\right) \, d\alpha(s)\\ = \int_a^b f(s)\left(\int_s^b \, d\beta(t)\right)\, d\alpha(s) = \int_a^b [\beta(b) - \beta(s)]f(s) \, d\alpha(s)$$
Another approach is an application of integration by parts. For Riemann-Stieltjes integrals this guarantees the implications $\alpha \in \mathscr{R}(\beta) \implies \beta \in \mathscr{R}(\alpha)$ and $ \beta \in \mathscr{R}(F) \implies F \in \mathscr{R}(\beta)$.
It is not difficult to show using Riemann-Stieltjes sums that
$$\int_a^b \beta(t) \, dF(t) = \int_a^b \beta(t)f(t) \, d\alpha(t),$$
and then with further application of IBP, we get
$$\int_a^b F(t) \, d\beta(t) = F(b)\beta(b) - \underbrace{F(a)\beta(a)}_{= \,0} - \int_a^b \beta(t)f(t) \, d\alpha(t)\\= \int_a^b [\beta(b) - \beta(t)]f(t) \, d\alpha(t)$$