I am asked to produce a rigorous proof of the following:
Let $A$ be some area in $\mathbb{R}^2$ whose boundary $\partial A$ is smooth, let $g$ be a $C^2$ function on $\bar A$ (the closure of $A$) which vanishes on $\partial A$. Let $f$ be a continuous function on $\bar A$. If $\int_{A}fg=0$ for all such $g$, then $f=0$.
This problem arises in the study of the Poisson equation, so some conditions like the $C^2$ condition in particular arises naturally and might not be optimal.
The proof is supposed to be rigorous so if there is any mistakes or inaccuracies that you notice, even slight ones, please point them out!
My attempt:
Since $f$ is continuous on $\bar A$ which is a compact space (it is closed and bounded, and $\mathbb{R}^2$ is finite-dimensional), $f\in L^2(\bar A)$.
$L^2(\bar A)$ is endowed with the usual scalar product $\langle f,g \rangle=\int_{A}fg$ and it is known that such a space is a Hilbert space. This means that it suffices to show that $f$ is orthogonal to a dense family in that space to prove the claim $f=0$.
I know that compactly supported $C^\infty$ functions are dense in $L^2(\bar A)$. In particular, they are $C^2$ on $\bar A$.
Now I would like to show that I can add the condition that these functions vanish on $\partial A$ while keeping the density of the family of functions. If it were true, this would complete my proof that $f=0$.
This is not obvious to me that it is true. Indeed, while I can imagine that adding the vanishing condition on $\partial A$ is not a big deal since the smoothness of the boundary probably means that $\partial A$ has measure $0$, in principle I am simply modifying my functions in a way that does not affect distance in $L^2(\bar A)$ and therefore density should remain. However, due to the regularity of my functions, perhaps there is a chance that changing them on a set of measure $0$ implies change on a larger set of non-zero measure (due to continuity for example), and so I might lose the density property.
A different approach: Verify that the function $g$ on $\mathbb R^2$ defined by
$$g(x)=\begin{cases} (1-|x|^2)^3,& |x| \le 1\\0,& |x|>1\end{cases}$$
belongs to $C^2(\mathbb R^2).$ Note that $g>0$ on the open unit ball and g=0 elsewhere.
In your problem: Suppose $f(a)>0$ for some $a$ in the interior of $A.$ By the continuity of $f$ there exists $\overline {B(a,r)} \subset A$ on which $f>0.$ Then $h(x) = g((x-a)/r)$ is a $C^2$ function such that $h>0$ on $B(a,r)$ and $h=0$ elsewhere. We have
$$\int_A fg = \int_{B(a,r)} fg >0,$$
contradiction. A similar contradiction arises if $f(a)<0$ somewhere in $A.$ Thus $f\equiv 0$ in $A.$ The same thing holds on $\bar A$ by the continuity of $f$ on $\bar A.$