Consider the following $ u_{tt}=c^2u_{xx}+xt,\\ u(x,0)=0,\\ u_t(x,0)=\sin (x)$
and find the solution.
Solution.
We have that $u(x,t)$ is given by $$u(x,t)=\frac{1}{2}(g(x+ct)+g(x-ct))+\frac{1}{2c}\int_{x-ct}^{x+ct}h(u)du+\frac{1}{2c}\iint_\Delta f(y,s)dA,\cdots\cdots (1)$$ where $\Delta $ is a triangle with vertex $(x-ct,0),(x+ct,0),(x,t)$.
Thus in our case, $u(x,t)=\frac{1}{2c}\int_{x-ct}^{x+ct}\sin(u)du+\frac{1}{2c}\iint_\Delta xt \ dA.$
Using Green Theorem we have that $\frac{1}{2c}\iint_\Delta xt \ dA=\frac{1}{2}[\cos(x_0+ct)-\cos(x_0-ct)-2cu(x_0,t_0)],$ where I took a point $(x_0,t_0)$ to be able to integrate using the theorem.
and
$\frac{1}{2c}\int_{x-ct}^{x+ct}\sin(u)du=\frac{1}{2}[-\cos(x_0+ct_0)+\cos(x_0-ct_0)]$
Hence the final solution is $u(x,t)=\frac{1}{2}[\cos(x_0+ct_0)-\cos(x_0-ct_0)]+\frac{1}{2}[\cos(x_0+ct)-\cos(x_0-ct)-2cu(x_0,t_0)]$
Is it correct?
This is the first time I use formula (1), so if something can be improved don't hesitate to suggest.
Edit
Apparently the solution is wrong because I had to calculate the double integral over the area of the triangle and not the contour integral, that's what my professor said, but I don't know why, I didn't understand why. Could someone explain why is that?
I still don't see why application of Green theorem it's not correct in the double integral.
There's a lot of steps missing in your work, so I can't reliably tell you exactly what you did wrong. But I can tell you that most of it is wrong.
Here's the "usual" way to evaluate the area integral
$$ \frac{1}{2c}\iint_{D} ys \ dyds $$
The sides of the triangle $D(y,s)$ are given by
\begin{align} &S_1\{(x-ct,0)\to(x+ct,0)\} &&: s = 0 \\ &S_2\{(x+ct,0)\to (x,t)\} &&: y = x + c(t-s) \\ &S_3\{(x,t)\to(x-ct,0)\} &&: y = x - c(t-s) \end{align}
Then
\begin{align} \frac{1}{2c}\iint_{D} ys \ dyds &= \frac{1}{2c}\int_{0}^{t} \int_{x-c(t-s)}^{x+c(t-s)} ys\ dyds \\ &= \frac{1}{2c}\int_0^t \frac{\big[(x+c(t-s)\big]^2 - \big[x-c(t-s)\big]^2}{2}s\ ds \\ &= \frac{1}{2c}\int_0^t \big[{2c}x(t-s)\big]s\ ds \\ &= \int_0^t x(ts-s^2)\ ds \\ &= x\left(\frac{t^3}{2} - \frac{t^3}{3}\right) \\ &= \frac{xt^3}{6} \end{align}
Green's theorem method: You need to find two functions $M$ and $N$ such that $$ \frac{\partial M}{\partial x} - \frac{\partial N}{\partial t} = xt $$
These functions need not be unique, so we can arbitrarily choose $$ M(x,t) = 0, \quad N(x,t) = -\frac{xt^2}{2} $$
Then
$$ \iint_D \left( \frac{\partial M}{\partial y} - \frac{\partial N}{\partial s} \right) dyds = \int_{\partial D} Ndy + Mds = \int_{\partial D} -\frac{ys^2}{2}dy $$
Following the boundary of the triangle in the counter clockwise direction, we have to split the integral into
\begin{align} \int_{\partial D} -\frac{ys^2}{2}dy &= \int_{S_1} -\frac{ys^2}{2}dy + \int_{S_2}-\frac{ys^2}{2}dy + \int_{S_3}-\frac{ys^2}{2}dy \\ &= 0 + \int_{x+ct}^{x} -\frac{\big[x+c(t-s)\big]s^2}{2}(-c\ ds) + \int_{x}^{x-ct} -\frac{\big[x-c(t-s)\big]s^2}{2}(c\ ds) \\ &= \frac{c}{2}\int_{x+ct}^{x} \big[x+c(t-s)\big]s^2\ ds - \frac{c}{2}\int_{x}^{x-ct} \big[x-c(t-s)\big]s^2\ ds \end{align}
You can finish the integral if you want to, but this shows why using Green's theorem here is not a good idea. You get a bigger mess than the one you started with
Combined with the homogeneous piece (which is trivial to evaluate), the correct solution is
$$ y(x,t) = \frac{xt^3}{6} + \frac{1}{2c}\big[\cos(x-ct) - \cos(x+ct) \big] $$