Consider the picture above, where $ABCD$ and $EFGC$ are squares with areas respectively $A$ and $B$. Find the shaded area.
Well, I observerd the following: If the intersection between $BG$ and $EF$ is the midpoint of $EF$, then the area will be $\dfrac{A}{2} + \dfrac{3B}{4}$, since the square $EFGC$ will be divided into a rectangle plus a triangle with half the area of the rectangle, leaving out an identical triangle. But then when I change the square $EFGC$, the intersection will change, and the expression will change. Is there a more general formula that includes this case? I can't see how that's possible.
A colleague of mine did the following:
"Area of triangle $DBC$ is $\dfrac{A}{2}$. Area of triangle $GCB$ is $\dfrac{\sqrt{A}\sqrt{B}}{2}$. If $H$ is $BG \cap EF$, triangle $BEH$ has height $\sqrt{A}-\sqrt{B}$ and is similar to $GFH$, with ratio $\dfrac{\sqrt{A}-\sqrt{B}}{\sqrt{B}} = \dfrac{\sqrt{A}}{\sqrt{B}}-1$, then base of $BEH$ is $\dfrac{A}{\sqrt{B}} - \sqrt{A}$. Then the trapezoid $EHGC$ has area $\dfrac{\Bigg(\dfrac{A}{\sqrt{B}} - \sqrt{A}+\sqrt{B}\Bigg)\sqrt{B}}{2} = \dfrac{A - \sqrt{AB} + B}{2}$, then the shaded area is equal to $A + \dfrac{B-\sqrt{AB}}{2} $."
But I don't understand this part: "then base of $BEH$ is $\dfrac{A}{\sqrt{B}} - \sqrt{A}$".
I think his formula is not correct because I tried an example with $A=2$,$B=1$ and $H$ being midpoint of $EF$, and it didn't work, but maybe I'm doing something wrong.
Can someone please explain what's happening? Thanks.

$$ \triangle EHB \sim \triangle FHG \\ \implies \frac{FH}{EH} = \frac{FG}{EB} = \frac{\sqrt B}{\sqrt A - \sqrt B}$$ So $$ EH = FH ( \frac {\sqrt A -\sqrt B} {\sqrt B} ) $$ But we also know that $$ FH + EH = EF = \sqrt B $$ So $$FH(1+ \frac {\sqrt A -\sqrt B} {\sqrt B} ) = \sqrt B $$
$$ \implies FH(\sqrt B + \sqrt A - \sqrt B ) = B $$
$$\implies FH = \frac { B }{\sqrt A} $$
$$ A_{TOT}= \frac 12A + B - \frac 12 FH\cdot FG \\ = \frac 12A + B - \frac 12 \frac { B }{\sqrt A} \cdot \sqrt B \\ = \frac 12A + \left (1- \frac 12 \sqrt \frac BA \right )B $$