Suppose that $\alpha$ is right continuous and increasing. Given $\epsilon > 0$ and $[c,d]\subset[a,b]$, construct a continuous function $f$ with $0\leq f\leq 1$ such that $\int_a^bf d\alpha\geq\alpha(d)-\alpha(c)-\epsilon$. The hint given is $f$ should "look like" the indicator function on the interval $[c,d]$
My idea was to create a function $f$, which is 0 on the interval $[a,c-\delta)$, and $(b+\delta,d]$. Then on interval $[c-\delta,d+\delta]$, we can then have something like a trapezoid, with the top reach the value $1$. Therefore, the function would be continuous. Also, the value of the integral would just be $\int_a^bf d\alpha = \int_{c-\delta}^{d+\delta}f d\alpha \geq\int_c^d 1 d\alpha $. Since $\alpha$ right continuous and increasing, $\int_c^d 1 d\alpha $ would just be $\alpha(d)-\alpha(c)$. So we have the desired result.
Is this right? I never used the information about $\epsilon$ though.