Here's the statement:
Let $f \in C([a,b])$ and $H$ be the set $\{h\in C([a,b]):h(a)=h(b)=0\}$. If $\int_a^bf(x)h(x)\,\text{d}x=0$ for all $h\in H$, then $f(x)=0$ for all $x\in [a,b]$.
I saw a lot of long proofs for this but I thought I could do better, but I think there could be a subtle error. Here's my attempt:
Consider the constant $c$ and the function $\phi$ defined as
$$\phi(x)=\int_a^xh(x)-c\,\text{d}x, c=\frac1{b-a}\int_a^bh(x)\,\text{d}x$$
obviously $\phi(a)=\phi(b)=0$, hence $\phi \in H$. Also, $h$ is integrable because it is continuous in $[a,b]$.
By hypothesis, we have
$$\int_a^bf(x)h(x)\,\text{d}x=0\Leftrightarrow\int_a^bf(x)\phi'(x)\,\text{d}x=0$$
Then we can use Du Bois-Reymond's lemma, which states
Let $H$ be the set $\{h\in C^1([a,b]):h(a)=h(b)=0\}$. If $f\in C([a,b])$ and $\int_a^b f(x)h'(x)\,\text{d}x=0$ for all $h\in H$, then $f(x)$ is constant for all $x\in[a,b]$.
The lemma can be used directly to get that $f(x)=k$, where $k\in\mathbb{R}$.
Then our hypothesis is simply
$$\int_a^bkh(x)\,\text{d}x=0$$
for this to be true for all $h \in H$, we must have $k=0$, because if $h(x)$ were, for example, a positive function in $(a,b)$ with $h(a)=h(b)=0$, it would not be true... thus the lemma is proved.
Now, where does it all fall apart?
Alright, my professor pointed out a possible flaw in the proof: is it really possible to represent every function in $H = \{h \in C^1([a,b]):h(a)=h(b)=0\}$ as $\phi(x)$, given its definition in the original post?
I'd guess that means we need to verify whether or not for every function $h\in H$ ($H$ in the sense of the original definition, where we consider functions in $C([a,b])$, there is a correspondent function $f\in H$ (in the $C^1([a,b])$ sense) such that $f(x)=\int_a^b h(x)-c\,\text{d}x$. Somewhat informally, we need to verify if this integral is a "surjection", I think. This is probably not true, but I don't think I know enough to prove it.