Considering the attached image, please compute the area of the square.
$\\$To elaborate suppose we have a triangle $\Delta ABC$, that its angle $\angle ABC$ is equal to $45^o$. On side $\overline{AB}$ we have a point $E$ that divide the side to two segments, $\overline{BE}=4$ and $\overline{EA}=6$.
Find the area of the square $\square CDEF$, such that its $\overline{FC}$ side reside on $\overline{AC}$ side of the triangle $\Delta ABC$, its one vertex is on E point, and the alternate vertex of E is on vertex C.


$$\begin{align*} \angle ACE &= 45^\circ\\ \triangle ABC &\sim \triangle ACE\\ \frac{AC}{AB} &= \frac{AE}{AC}\\ {AC}^2 &= 60 \end{align*} $$
Consider $\triangle ACE$, taking sine law and considering only the case where $\angle AEC = \angle ACB$ are obtuse,
$$\begin{align*} \frac{\sin\angle AEC}{AC} &= \frac{\sin \angle ACE}{AE}\\ \sin\angle AEC &= \frac{AC\sin\angle ACE}{AE}\\ &= \frac{\sqrt {60}\cdot \frac1{\sqrt2}}{6} = \frac{\sqrt{30}}{6}\\ \cos\angle AEC &= \color{red}-\sqrt{1-\sin^2\angle AEC}\\ &= \color{red}-\frac{\sqrt6}{6} \end{align*}$$
The area of the square can be found from side $EF = AE\cos \angle AEF$:
$$\begin{align*} (AE\cos \angle AEF)^2 &= {AE}^2 \cos^2(\angle AEC-45^\circ)\\ &= {AE}^2\cdot \frac{1+\cos 2(\angle AEC-45^\circ)}{2}\\ &= {AE}^2\cdot \frac{1+\cos (2\angle AEC-90^\circ)}{2}\\ &= {AE}^2\cdot \frac{1+\sin 2\angle AEC}{2}\\ &=\frac{6^2}2\left[1+2\cdot\frac{\sqrt{30}}{6}\left(\color{red}-\frac{\sqrt6}{6}\right)\right]\\ &= 18 \color{red}- 6\sqrt5 \end{align*}$$
If otherwise consider an acute $\angle ACB$ and flip the red $\color{red}-$ sign, this gives another solution.