Riemann Sums with non-Riemann-integrable functions.

Question

Let $f$ be a Riemann-integrable function on $[0,1]$. Often, when working with Riemann sums, we wish to say that $$\begin{align} \lim_{N \to \infty} \sum_{k=1}^N \frac{1}{N} f(\frac{k}{N} ) = \int_0^1 f(x) \mathrm{d}x. \tag{1} \end{align}$$

This works as the definition of Riemann Integral suggests that the difference of the two values can be made arbitrarily small if we take a sufficiently fine partition. However, sometimes, we wish to apply this technique when $f$ is not Riemann-integrable. For instance $f = \log$, as in Limit using Riemann integral. These led me to the following question.

If $f$ is not Riemann-integrable, but $\int_0^1 f(x) \mathrm{d}x$ exists and is equal to $L$. Under what conditions is the right-hand side of equation 1 equal to $L$. Note that this is an improper integral. (It is not true for Lebesgue-integrable $f$, for instance the indicator functions of the rationals) Specifically,

1. The linked question suggests monotonicity is sufficient. How can we prove this?
2. Why aren't they equal in general? I feel that if $f$ is Riemann-integrable on $(0,1)$, this two expressions are equal. (By this I mean that it is integrable on all $[a,b] \subset (0,1)$) Is this the case?
3. If they are not equal in general, what conditions are sufficient for equality?

Answer 1

(1) Typically, convergence of Riemann sums to an improper integral (over a bounded interval) works only for special choices of intermediate points (tags). For example, as proved here, if $f$ is monotone and unbounded at the left endpoint, then the right Riemann sum converges. For monotone functions other choices may work as well as long as the intermediate points are sufficiently far from the singularity.

As far as I know there is no all-encompassing theory. It is case dependent and oscillating integrands are more difficult to handle.

(2) They are not equal in general if the integrand is unbounded. A function must be bounded to be Riemann integrable. This is often stated but seldom proved. The proof is facilitated by showing that that for any $I$ there exists $\epsilon_0 > 0 $ such that for any partition $P$ there exists an intermediate point in the subinterval containing the singularity such that $|S(P,f) - I| > \epsilon_0.$ Hence, not all Riemann sums converge.