$f(x)$ is continuous on $[a, b]$, assumed to be Riemann integrable, and we let $\delta > 0$ be given.
Perhaps starting with boundedness may be the right way to go? We know that, since a continuous function on a closed interval attains its max value $M$ and min value $m$:
$m \cdot (b - a - \delta) \leq \int_{a + \delta}^{b} f(x)dx \leq M \cdot (b - a - \delta)$
but I'm unsure of where to go from here!
First, if $f(x)$ is continuous on $[a,b]$ this actually implies it is Riemann integerable, we don't need to add the assumption.
Let's use the definition of Riemann integration:
Consider the partition $P=\{x_0=a<x_1<x_2<...x_n= b\}$,
$\int_{a}^{b}f(x)dx=lim_{N\rightarrow \infty}\sum_{n}^Nf(x_n)(x_n,x_{n-1})$.
Now consider the partition $P_{\delta}=\{x_0=a+\delta<x_1<x_2<...x_n= b\}$ $\int_{a+\delta}^{b}f(x)dx=lim_{N\rightarrow \infty}\sum_{n}^Nf(x_n)(x_n,x_{n-1})+f(a+\delta)(x_0+\delta)$. Note the extra term at the end comes from the slightly modified partition.
Now if we compare the two partitions, note as $\delta\rightarrow 0$ $P_{\delta}=P$, and we have our result.