Background:
I came across the following family of generalized Franel integrals, and found them quite interesting. I don't think I've seen anything about these integrals before, at least not generalized, and I want to know if this is a known family of generalized integrals. I am also interested behind the behavior of this family of integrals, specifically when $p=\frac{1}{2}$. I want to find out if there's a simple algebraic closed form expression for this family of integrals. Wolfram does not do a good job factoring the integrands, but to be fair it didn't factor $I(a,b,p)$ as I have done in this post. Also, how would you approach cases where $\gcd{(a,b,c,\dots)} \neq 1$?
Consider the family of generalized integrals as the following:
$${I}(a,b,p)=\int_0^{ab} \left( \bigg\{\frac{x}{a}\bigg\}-p\right) \left( \bigg\{\frac{x}{b}\bigg\}-p\right) \; dx$$ $${I}(a,b,c,p)=\int_0^{abc} \left( \bigg\{\frac{x}{a}\bigg\}-p\right) \left( \bigg\{\frac{x}{b}\bigg\}-p\right) \left( \bigg\{\frac{x}{c}\bigg\}-p\right)\; dx$$ $${I}(a,b,c,d,p)=\int_0^{abcd} \left( \bigg\{\frac{x}{a}\bigg\}-p\right) \left( \bigg\{\frac{x}{b}\bigg\}-p\right) \left( \bigg\{\frac{x}{c}\bigg\}-p\right) \left( \bigg\{\frac{x}{d}\bigg\}-p\right)\; dx$$ $$\ldots$$ Where $a,b,c,\ldots \in \mathbb{N}$, $p \in \mathbb{Q}^+$, and $\gcd{(a,b,c,\ldots)}=1$.
Calculations:
Express the integral as the following: $${I}(a,b,p)=\sum_{i=0}^{a-1} \sum_{k=0}^{b-1} \int_0^1 \left(\frac{t+i}{a}-p\right)\left(\frac{t+k}{b}-p\right) \; dt$$ Changing the order of the summations and integral and using some algebra: $${I}(a,b,p)=\int_0^1 \left(\frac{a-1}{2}+t-ap\right)\left(\frac{b-1}{2}+t-bp\right) \; dt$$ Expanding the integrand out and factoring yields: $${I}(a,b,p)=\int_0^1 \frac{ab}{4}{\left(2p-1\right)}^2+\frac{at}{2}\left(1-2p\right)+\frac{bt}{2}\left(1-2p\right)+\frac{(a+b)}{4}\left(2p-1\right)+{\left(t-\frac{1}{2}\right)}^2 \; dt$$ $${I}(a,b,p)=\int_0^1 \frac{ab}{4}{\left(2p-1\right)}^2+{\left(t-\frac{1}{2}\right)}^2 \; dt$$ And so: $$\boxed{{I}(a,b,p)= \frac{ab}{4}{\left(1-2p\right)}^2+\frac{1}{12}}$$ Calculated similarly,I got the following: $$I(a,b,c,p)=\frac{abc{\left(1-2p\right)}^3}{8}+\frac{c}{24}\left(1-2p\right)$$ $$I(a,b,c,d,p)= \frac{abcd}{16}{\left(1-2p\right)}^4+\frac{{(1-2p)}^2}{48}\left(ab+cd\right)+\frac{1}{80} $$ However, as @Varun Vejalla and @OliverDiaz pointed out in the comments, these results are illogical, and there actually is no closed form for $I(a,b,c,d,p)$.
Further Observations:
Interestingly enough, $p=\frac{1}{2}$ is a special case for this entire family generalized integrals. Why is this? Assuming the aforementioned conditions are met:
$$I\left(a,b,\frac{1}{2}\right)=\frac{1}{12}$$ $$I\left(a,b,c,\frac{1}{2}\right)=\int_0^1 {\left(t-\frac{1}{2}\right)}^3 \; dt=0$$ $$I\left(a,b,c,d,\frac{1}{2}\right)=\int_0^1 {\left(t-\frac{1}{2}\right)}^4 \; dt=\frac{1}{80} $$ $$I\left(a,b,c,d,e,\frac{1}{2}\right)=\int_0^1 {\left(t-\frac{1}{2}\right)}^5 \; dt=0$$ And so it seems that the following statement is true: $$I\left(a_1,a_2,\ldots,a_n,\frac{1}{2}\right)=\int_0^1 {\left(t-\frac{1}{2}\right)}^n \; dt=\cases{ 0 & $n \; \text{is odd}$ \cr \frac{1}{2^n\left(n+1\right)} & $n \; \text{is even}$ }$$ However, Wolfram Alpha computed $I\left(a,b,c,d,\frac{1}{2}\right)=0$ for valid $a,b,c,d$ values.
Final Remarks:
I wonder what other interesting observations can be made about this family of generalized integrals. Specifically, are there other interesting special cases, and if so why are they so special?
This just addresses the simplest questions in the OP, namely the estimation of $I(a,b;p)$ when $d:=g.c.d(a,b)>1$. In such case, there are integers $q,r$ such that $a=qd$, $b=rd$ and $g.c.d(q,r)=1$
The change of variable $u=x/d$ and the periodicity of $x\mapsto\{x\}$ results in
$$ \begin{align} I(a,b;p)&=d\int^{dqr}_0\Big(\big\{\tfrac{x}{q}\big\}-p\Big)\Big(\big\{\tfrac{x}{r}\big\}-p\Big)\,dx=d\sum^{d-1}_{j=0}\int^{(j+1)qr}_{jqr}\Big(\big\{\tfrac{x}{q}\big\}-p\Big)\Big(\big\{\tfrac{x}{r}\big\}-p\Big)\,dx\\ &= d\sum^{d-1}_{j=0}\int^{qr}_0\Big(\big\{\tfrac{x+jqr}{q}\big\}-p\Big)\Big(\big\{\tfrac{x+jqr}{r}\big\}-p\Big)\,dx\\ &=d^2\int^{qr}_0\Big(\big\{\tfrac{x}{q}\big\}-p\Big)\Big(\big\{\tfrac{x}{r}\big\}-p\Big)\,dx \end{align}$$
Form the case of relative prime one obtained (the OP provides a sketch of the proof for this case):
$$\begin{align} \frac{1}{ab}I(a,b;p)= \frac{1}{4}{\left(1-2p\right)}^2+\frac{1}{12}\frac{g.c.d(a,b)}{l.c.m(a,b)} \end{align}$$
where $l.c.m(a,b)$ is the lowest common multiple of $a$ and $b$. When $p=1/2$ we recover Franel's formula.
For higher order integrals, I have a few references that may be useful to those interested in this question:
Franel integrals of order three
Franel Integrals of over four
Multiple Franel integrals
Here I provide further details for the expression $I(a,b;p)$ when $g.c.d(a, b)=1$.
Without loss of generality assume $a>b$. The $a=bq+r$ for $q\in\mathbb{N}$ and $1\leq r<b$
$$ \begin{align} I(a,b;p)&=\sum^{b-1}_{k=1}\int^{(k+1)a}_{ka}\Big(\big\{\tfrac{x}{a}\big\}-p\Big)\Big(\big\{\tfrac{x}{b}\big\}-p\Big)\,dx = \sum^{b-1}_{k=0}\int^a_0\Big(\big\{\tfrac{x+ak}{a}\big\}-p\Big)\Big(\big\{\tfrac{x+ak}{b}\big\}-p\Big)\,dx\\ &=\sum^{b-1}_{k=0}\int^a_0\Big(\big\{\tfrac{x}{a}\big\}-p\Big)\Big(\big\{\tfrac{x+rk}{b}\big\}-p\Big)\,dx=\sum^{b-1}_{k=0}\sum^{a-1}_{\ell=0}\int^{\ell+1}_{\ell}\Big(\big\{\tfrac{x}{a}\big\}-p\Big)\Big(\big\{\tfrac{x+rk}{b}\big\}-p\Big)\,dx\\ &=\sum^{b-1}_{k=0}\sum^{a-1}_{\ell=0}\int^1_0\Big(\big\{\tfrac{x+\ell}{a}\big\}-p\Big)\Big(\big\{\tfrac{x+\ell+rk}{b}\big\}-p\Big)\,dx\ \end{align} $$ Since $g.c.d(a,b)=g.c.d(b,r)=1$, for each $0\leq\ell<a$ fixed, $\ell+kr$ ranges over $\{0,\ldots,b-1\}\mod \,b$ as $k$ over $\{0,\ldots,b-1\}$. Hence $$ I(a, b;p)=\sum^{b-1}_{j=0}\sum^{a-1}_{\ell=0} \int^1_0\Big(\big\{\tfrac{x+\ell}{a}\big\}-p\Big)\Big(\big\{\tfrac{x+j}{b}\big\}-p\Big)\,dx=\sum^{b-1}_{j=0}\sum^{a-1}_{\ell=0}\int^1_0 \Big(\tfrac{x+\ell}{a}-p\Big)\Big(\tfrac{x+j}{b}-p\Big)\,dx $$
The rest is as the OP indicated.
I ignore thus far whether a similar argument carries over for higher orders under the assumption that $g.c.d(a_1,\ldots,a_n)=1$.