Let $X$ be a subset of $C([0,1])$ with $$ X=\big\{f\in C([0,1]): 0\le f(x)\le x,\text{$f$ is a polynomial}\big\} $$ where $C([0,1])$ denotes the space of continuous real-valued functions on $[0,1]$. Define a nonlinear functional $F$ on $C([0,1])$ as $$ F(f)=\int_0^1 |f(x)|^2\; dx-\left(\int_0^1 f(x)\; dx\right)^2 $$ Can $F$ achieve a maximum in the set $X$?
Remarks.
If one restricts the set to the linear functions $f$ of the form $f(x)=kx$ with $0\le k\le 1$, then this reduces to a min/max problem for values of $k$: $$ F(f) = \int_0^1 (kx)^2\;dx-\left(\int_0^1kx\;dx\right)^2=\frac{k^2}{3}-\frac{k^2}{4} = \frac{k^2}{12} $$ So the maximum is achieved at $f(x)=x$ for the restriction.
One could also restrict the functions to be parabolas of the form $f(x)=kx^2$ with $k\in[0,1]$, and get $$ F(f)=\frac{k^2}{5}-\frac{k^2}{9}=\frac{4k^2}{45}, $$ where the maximum is achieved at $f(x)=x^2$.
If one allows one more parameter and consider $f(x)=kx^m$, then $$ F(f)=k^2\cdot \left(\frac{1}{2m+1}-\frac{1}{(m+1)^2}\right)=\frac{k^2m^2}{(2m+1)(m+1)^2}\;. $$ Then one can study the maximum of $g(m)=\frac{m^2}{(2m+1)(m+1)^2}$ over positive integers $m$.
But this is far from analyzing all the polynomials in $X$.
Another observation is that the functional can be written as $F(f)=\|f\|_{L^2}^2-\|f\|_{L^1}^2$. But this seems not helpful at all.
This question was motivated by this unanswered question on the site. I was curious and attempted to solve that one but I didn't get anything. So I restrict the attention to the set of polynomials here.
For the sake of convenience, we will consider the functional
$$ \mathsf{Var}(f) = \int_{0}^{1} f(x)^2 \, \mathrm{d}x - \left( \int_{0}^{1} f(x) \, \mathrm{d}x \right)^2 $$
on the set $\overline{X}$ of all measurable functions $f$ on $[0, 1]$ such that $0 \leq f(x) \leq x$ for all $x \in [0, 1]$. Note that $\overline{X}$ is the closure of OP's set $X$ in $L^2([0,1])$, justifying the notation.
In order to find the maximum of $\mathsf{Var}(\cdot)$, for each $c \in [0, 1]$ we define $g_c : [0, 1] \to \mathbb{R}$ by
$$ g_c(x) = x \mathbf{1}_{[c,1]}(x) = \begin{cases} x, & x \geq c, \\ 0, & x < c. \end{cases} $$
This function is related to our maximization problem via:
Using this, we are ready to answer OP's question. Let $\alpha = \frac{\sqrt{5} - 1}{2}$. Then it is easy to check that $c \mapsto \mathsf{Var}(g_c)$ is maximized at $c = \alpha$. Now, since $g_{\alpha}$ can be approximated in $L^2$ by the elements of $X$,
$$ \sup_{f \in X} \, \mathsf{Var}(f) = \mathsf{Var}(g_{\alpha}) = \frac{5}{24}(3 - \sqrt{5}) \approx 0.159153. $$
However, this supremum cannot be attained on $X$, because $f \in \overline{X}$ and $\mathsf{Var}(f) = \mathsf{Var}(g_{\alpha})$ together imply that $f = g_{\alpha} \notin X$.