Let $$\Omega = \left\{f(x) \in \mathcal{L}^2[0,T]: \frac{1}{T}\int_0^Tf(x)dx = \mu,~ a \le f(x) \le b,~\forall x \in [0,T]\right\},$$ where $\mathcal{L}^2[0,T]$ is the set of Lebesgue square-integrable functions on $[0,T]$ and $0 \le a \le \mu \le b$ are real numbers. Let $P_\tau(\Omega)$ denote the projection of $\Omega$ onto the space of $\mathcal{L}^2[0,\tau]$ functions for $\tau < T$; i.e., if $g \in P_\tau(\Omega)$ then there exists another function $\tilde{g}$ defined on $(\tau,T]$ where $$\hat{g}(t) = \begin{cases} g(t),&t\in [0,\tau] \\ \tilde{g}(t), & t \in (\tau,T] \end{cases} \qquad \textrm{and}\qquad \hat{g} \in \Omega. $$
My question: If $h(t) \ge 0$ is defined on $[0,\tau]$ and $h \not \in P_\tau(\Omega)$, then I want to solve: $$f^* = \begin{array}{rl} \arg \min & \|f - h\| \\ \textrm{s.t.} & f \in P_\tau(\Omega), \end{array} $$ where $\langle f,h\rangle = \int_0^\tau f(t)h(t)dt$ is an inner product with induced norm $\|f\| = \sqrt{\langle f,f \rangle}$.
I am pretty sure $f^*$ exists and is unique, but I would really like to characterize it as much as possible as a function of $h$.
Progress thus far: If $\tau/T$ is small enough, then for $x \in [0,\tau]$, (1) if $h(x) > b$, then $f(x) = b$, (2) if $h(x) < a$, then $f(x)=a$, and (3) otherwise $f(x)=h(x)$. Unfortunately, I can't figure out how $\mu$ comes into play when $\tau/T$ is larger.
Not a complete answer
It helps to state clearly what $P_\tau(\Omega)$ is. It is the set of $L^2$ functions $f$ on $[0,\tau]$ such that
The second inequality comes from the fact that $f$ we must have an extension to $[0,T]$ with total integral $\mu T$. The form of lower and upper bounds in (2) does not really matter: we could just call them $m$ and $M$. Of course, (2) may happen to follow from (1) in some cases.
Given $h$, we want to find the nearest $f$ that satisfies (1) and (2). If we only had (1) to worry about, the nearest would be $f = \min(b,\max(a,h))$, as you noted. If we had only (2), the nearest would be $h+c$ where $c$ is the smallest number by absolute value such that the integral of $h+c$ is within the bounds (2). (When you want to change the integral without changing the $L^2$ norm too much, the right way is to add a constant.)
Given both constraints, the answer should be: $$f = \min(b,\max(a,h+c)) \tag{ * }$$ where $c$ is the smallest number by absolute value such that the integral of $f$ is within the bounds (2).
Supporting evidence. Indeed, suppose $f$ is the nearest eligible function to $h$. Let $$A= \{x\in [0,\tau] : a<f(x)<b \}$$ On this set $f$ is unconstrained by pointwise bounds, and therefore is free to move as long as the integral is preserved. Therefore, $f_{|A}$ is minimal for the problem $\|f-h\|_{L^2(A)}\to\min$ with given $\int_A f$. It follows that $f-h$ is a constant function on $A$.