I have a gamma correction function of the following form:
$$f(x) = x^\gamma \text{ for } 0 \le x \le 1, \gamma \in \mathbb{R}^+$$
For example, a simplified approximation of the sRGB electro-optical transfer function (EOTF) is $f(x) = x^{2.2}$, i.e. $\gamma=2.2$.
I'm trying to analyse the magnitude of deviation between this "ideal" gamma correction function, when calculated in a continuous manner, versus a variant where the space is quantised into an $n$-bit value.
The $n$-bit quantised version of the gamma function can be represented as:
$$f_Q(x) = \frac{\lfloor 2^n x^\gamma \rfloor}{2^n} \text{ for } 0 \le x \le 1, n \in \mathbb{N}^+, \gamma \in \mathbb{R}^+ $$
By subtracting $f_Q(x)$ from $f(x)$ we get a function that describes the magnitude of quantisation error across the input range:
$$f_{Q\Delta}(x) = f(x) - f_Q(x)$$
To turn this into a single value representing the total magnitude of quantisation error, I'd like to calculate the definite integral:
$$\int_0^1 f_{Q\Delta}(x) \text{ dx} = \int_0^1 x^\gamma - \frac{\lfloor 2^n x^\gamma \rfloor}{2^n} \text{ dx} \text{ for } 0 \le x \le 1, n \in \mathbb{N}^+, \gamma \in \mathbb{R}^+ $$
It's been 17 years since I last did definite integrals, so I'm more than a bit out of practice.
My first thought was to tackle the initial step where $\lfloor 2^n x^\gamma \rfloor = 0$. The upper bound of this first quantised step is the point where $\lfloor 2^n x^\gamma \rfloor > 0$, which is $\left(\frac{1}{2^n}\right)^{1/\gamma}$ or $2^{-n/\gamma}$.
This simplifies the definite integral to:
$$\int_0^{2^{-n/\gamma}} x^\gamma \text{ dx}$$
I'm pretty sure this is the solution of the integral for this first step:
$$\int_0^{2^{-n/\gamma}} x^\gamma \text{ dx} = \frac{\left(2^{-n/\gamma}\right)^{\gamma+1}}{\gamma+1} $$
This seems to match up with the numbers I got when calculating the integral through the Desmos graphing calculator online.
My next thought was to figure out the intervals for each quantised step. The end of each step should be:
$$\left(\frac{k}{2^n}\right)^{1/\gamma} = k^{1/\gamma} \space 2^{-n/\gamma}$$
So, the start of each step should simply be:
$$\left(\frac{k-1}{2^n}\right)^{1/\gamma} = \left(k-1\right)^{1/\gamma} \space 2^{-n/\gamma}$$
This lines up with the previous definite integral bounds: for $k=1$ we get $\left(\frac{1}{2^n}\right)^{1/\gamma}$ and $\left(\frac{0}{2^n}\right)^{1/\gamma} = 0$.
So this would mean that my original definite integral can be expressed as the following summation of definite integrals, each describing one quantised step:
$$\int_0^1 x^\gamma - \frac{\lfloor 2^n x^\gamma \rfloor}{2^n} \text{ dx} = \sum_{k=1}^{2^n} \int_{\left(k-1\right)^{1/\gamma} \space 2^{-n/\gamma}}^{k^{1/\gamma} \space 2^{-n/\gamma}} x^\gamma \text{ dx}$$
However, this is where I got stuck.
How do I rework these summed definite integrals to come up with a single formula for the whole definite integral? Or is there another approach I can use?
I don't know how you got that last line where you got stuck, but it does not seem numerically correct. Everything else looks correct, though.
Here is a slightly different method. We'll use the same bounds you defined for $n$ and $\gamma$. Then
$$ \begin{align} \int_0^1 f_{Q \Delta}(x)dx &= \int_0^1\left(x^{\gamma} - \frac{\lfloor{2^n x^\gamma\rfloor}}{2^n}\right)dx \\ &= \int_0^1x^{\gamma}dx -\int_0^1 \frac{\lfloor{2^n x^\gamma\rfloor}}{2^n}dx\\ &= \frac{1}{\gamma+1}- \frac{1}{\gamma2^{n}2^{\frac{n}{\gamma}}}\int_{0}^{2^{n}}u^{\frac{1}{\gamma}-1}\lfloor{u\rfloor}du \tag{1} \\ &= \frac{1}{\gamma+1}-\frac{1}{\gamma2^{n}2^{\frac{n}{\gamma}}}\sum_{k=1}^{2^{n}}\int_{k-1}^{k}u^{\frac{1}{\gamma}-1}\lfloor{u\rfloor}du \\ &= \frac{1}{\gamma+1}-\frac{1}{\gamma2^{n}2^{\frac{n}{\gamma}}}\sum_{k=1}^{2^{n}}\lim_{a \to (k-1)^+}\lim_{b \to k^-}\int_{a}^{b}u^{\frac{1}{\gamma}-1}\lfloor{u\rfloor}du \\ &= \frac{1}{\gamma+1}-\frac{1}{\gamma2^{n}2^{\frac{n}{\gamma}}}\sum_{k=1}^{2^{n}}\lim_{a \to (k-1)^+}\lim_{b \to k^-} \frac{u^{\frac{1}{\gamma}-1+1}}{\frac{1}{\gamma}-1+1}\lfloor{u\rfloor}\Bigg|_a^b \tag{2} \\ &= \frac{1}{\gamma+1}-\frac{1}{2^{n}2^{\frac{n}{\gamma}}}\sum_{k=1}^{2^{n}}\left(\lim_{b \to k^-} b^{\frac{1}{\gamma}}\lfloor{b\rfloor} -\lim_{a \to (k-1)^+} a^{\frac{1}{\gamma}}\lfloor{a\rfloor}\right)\\ &= \frac{1}{\gamma+1}-\frac{1}{2^{n}2^{\frac{n}{\gamma}}}\sum_{k=1}^{2^{n}}\left( k^{\frac{1}{\gamma}}(k-1) - (k-1)^{\frac{1}{\gamma}}(k-1)\right)\\ &= \frac{1}{\gamma+1}+\frac{1}{2^{n}2^{\frac{n}{\gamma}}}H_{2^{n}}^{\left(-\frac{1}{\gamma}\right)}-1. \tag{3}\\ \end{align} $$
$(1)$ We substitute $2^n x^{\gamma} = u$.
$(2)$ $\lfloor{u\rfloor}$ is constant (and continuous) on $(a,b)$ so we treat it like a constant and integrate $u^{\frac{1}{\gamma}-1}$ using the rules we all know.
$(3)$ We use the Generalized Harmonic Number $H_n ^{(r)}$, which I came up with using this.
I checked my work on Desmos to make sure I didn't make a mistake somewhere, and my work seems numerically correct.