For reference: In figure $T$ and $K$ are points of tangency, $MT = a$ and $KN = b$; calculate area of region $ABC$.
(Answer:$2\sqrt{ab}(\sqrt a+\sqrt b)^2$)
My progress:
$$S_{ABC} = p \cdot r = \frac{r \cdot (AB+BC+AC)}{2}\\ AC +2R = AB+BC\\ S_{ABC} = AG \cdot GC \qquad \text{(property)} \\ S_{ABC} = (AC+R)R \qquad \text{(property)} \\ OTBQ:~\text{square} \implies TK = R\sqrt2 \\ \ldots ?$$



I continue your process and we already have $\text{TK}=\sqrt{2} R$.
According to intersecting chords theorem, we have $$\text{AT} \cdot\text{TB}=\text{MT}\cdot \text{TN}$$$$\text{BK} \cdot\text{KC}=\text{KN} \cdot \text{MK}$$ So $$\text{AT}=\sqrt{2} a+\frac{\text{ab}}{R}$$$$\text{KC}=\sqrt{2} b+\frac{\text{ab}}{R}$$ Because $S_{ABC}$ is a rectangular triangle, we have $\text{AB}^2+\text{BC}^2=\text{AC}^2$ and $$\text{BC}=\text{BC}+\text{KC}=\text{KC}+R$$$$ \text{AB}=\text{AT}+\text{TB}=\text{AT}+R$$$$\text{AC}=\text{AG}+\text{GC}=\text{AT}+\text{KC}$$ So solving the equation we get $$R=\sqrt{\text{ab}}$$ Therefore $$S_{ABC} = \frac{\text{AB}\cdot \text{BC}}{2}$$$$=\frac{1}{2} \left(\sqrt{2} a+2 \sqrt{\text{ab}}\right) \left(\sqrt{2} b+2 \sqrt{\text{ab}}\right)$$$$=\boxed{\sqrt{2} \sqrt{\text{ab}} (a+b)+3 a b}$$ I think the answer you give might be wrong.