I've now been going around and around trying to solve this problem, and I just haven't managed. It's supposed to have a quick and smart way of solving, and the solution must be 6. Below, the problem is presented.
Segment EC = $\sqrt{12}$.
Point E and point F are exactly midway through segments AB and AD, respectively.
What is the area of the part of the square outside the triangle CEF?
On
If $E$ and $F$ are the midpoints of $AB$ and $AD$, respectively, then the area of the square $ABCD$ not contained in the triangle $\triangle CEF$ is simply $5/8$ of the area of the entire square. This is because $\triangle AEF$ is $1/8$ of the total area, and $\triangle BCE$ and $\triangle CDF$ are congruent and each equal to $1/4$ of the area of the square.
So it suffices to find the side length of the square; given that $EC = \sqrt{12}$, and $EB = BC/2$, we simply solve the resulting Pythagorean relationship $$(EB)^2 + (BC)^2 = (EC)^2,$$ or $$(BC/2)^2 + (BC)^2 = 12,$$ or $$BC^2 = \frac{4}{5} \cdot 12 = \frac{48}{5},$$ and since $BC^2$ is the area of the square, the desired answer is $$\frac{48}{5} \cdot \frac{5}{8} = 6.$$
On
Draw a line from A to C. The area of triangles AEC and BEC are equal and their summ is half of the area of square. The area of triangle AEF is half of the area of triangle BEC. so the area of region outside of triangle is:
$S=S_{ABCD}\times(1/2+1/8)=\frac{5}{8}S_{ABCD} $
$\tan(\angle CEB=\alpha)=2$ ⇒ $\cos \alpha=1/\sqrt 5 $
$BC= 2\times\sqrt 12/\sqrt 5$
$S_{ABCD}=4\times\frac{12}{5}=\frac{48}{5}$
$S=\frac{48}{5}\times \frac{5}{8}=\frac{48}{8}=6$
From $EBC$ the side of the square is $\frac 2{\sqrt 5}\sqrt {12}$
From $AEF$ we have $EF=\sqrt 2 \frac 1{\sqrt 5}\sqrt 12$
Draw the altitude from $C$ to $EF$.
Compute the height of $CEF$ and hence the area of $CEF$