In the figure below, a goat is tied at the point $A$ with a rope of length $5$ metres. Find the total area goat can graze if it cannot enter the triangular region.
My attempt: We have from $\Delta CBE$ $$BE=CE=11 \cos \left(\frac{\pi}{4}\right)=\frac{11}{\sqrt{2}}>5$$
Now if we draw a circle with centre $A$ and radius $5$ we get the following picture:
The circle meets $CB$ at $M$ and $CE$ at $K$.
Now the area goat can graze is given by: $$A_g=\text{Area of the circle}-\text{Area of the triangle}\:ACK-\text{Area of the sector}\:MAK$$ By cosine rule in $\Delta CAK$ we get $$\cos \left(\frac{\pi}{4}\right)=\frac{3^{2}+C K^{2}-5^{2}}{6(CK)}$$ $\implies$ $$C K=\frac{\sqrt{3}+\sqrt{41}}{\sqrt{2}}$$ Now by sine rule we get $$\frac{C K}{\sin (\angle CAK )}=\frac{A K}{\sin \left(\frac{\pi}{4}\right)}=5 \sqrt{2}$$ $\implies$ $$\angle C A K=\sin ^{-1}\left(\frac{\sqrt{3}+\sqrt{41}}{10}\right)$$ $\implies$ $$\angle M A K=\pi-\sin ^{-1}\left(\frac{\sqrt{3}+\sqrt{41}}{10}\right)$$
Thus the area of sector $MAK$ is:
$$\operatorname{ar}(M A K)=\frac{25}{2}\left(\pi-\sin ^{-1}\left(\frac{\sqrt{3}+\sqrt{41}}{10}\right)\right) \to (1)$$
and
$$\operatorname{ar}(C A K)=\frac{1}{2}(3)\left(\frac{\sqrt{3}+\sqrt{41}}{\sqrt{2}}\right) \frac{1}{\sqrt{2}}=\frac{3(\sqrt{3}+\sqrt{41})}{4} \to (2)$$
Thus the area goat can graze is:
$$A_g=25 \pi-\left(\frac{25 \pi}{2}-\frac{25}{2} \sin ^{-1}\left(\frac{\sqrt{3}+\sqrt{41}}{10}\right)\right)-3\left(\frac{\sqrt{3}+\sqrt{41}}{4}\right)$$
$\implies$
$$A_{g}=\frac{25 \pi}{2}+\frac{25}{2} \sin ^{-1}\left(\frac{\sqrt{3}+\sqrt{41}}{10}\right)-\frac{3(\sqrt{3}+\sqrt{41})}{4}$$
Is this the correct approach?
The method you proposed in your problem statement to calculate the area $A_\mathrm{g}$ is correct, but your execution of it is faulty. You have made a slight mistake in solving the quadratic equation for $CK$, the correct value of which is $$CK= \frac{3+\sqrt{41}}{\sqrt{2}}.$$
We hope that, if you rectify this error, you might get the correct answer for $A_\mathrm{g}$, which is $$A_\mathrm{g}=56.19314 \mathrm{\space m}^2.$$
We give below a method, which is very similar to yours, but requires fewer calculations. As illustrated in $\mathrm{Fig.\space 1}$, $A_1$ and $A_2$ are the areas of the red sector $MAK$ and green triangle $ACK$ respectively. The line $KN$ is the perpendicular dropped from $K$ to $CB$. We also assume that the rope tethering the goat to the pole at $A$ is allowed to overrun the off-limit triangle $BCE$. Please note that all the angles are measured in radians, unless stated otherwise. Therefore, we have $$A_\mathrm{g}= A_1 – A_2, \mathrm{\quad where} \tag{1}$$ $$A_1 = AK^2 \times \frac{\beta}{2} \mathrm{\quad and} \tag{2}$$ $$A_2 = \frac{AC\times KN}{2} = \frac{AC\times AK \times \sin \left(\omega\right)}{2}. \tag{3}$$
First. we apply sine rule to the $\triangle ACK$ to determine the value of $\measuredangle CKA$ (i.e. $\alpha$). $$\alpha =\sin^{-1}\left( \frac{3}{5}\sin \left(\frac{\pi}{4}\right)\right) = 0.43815 \mathrm{\space rad}$$
Now, we can calculate the value of $\measuredangle BAK$ (i.e. $\omega$) as shown below. $$\omega = \measuredangle ACK + \measuredangle CKA = \frac{\pi}{4}+ 0.43815 = 1.22355 \mathrm{\space rad}$$
$$\therefore \beta = 2\pi - \omega = 5.05964 \mathrm{\space rad}.$$
From (2), it follows, that $$ A_1 = 5^2 \times \frac{5.05964}{2} = 63.24548 \mathrm{\space m^2}.$$
Similarly, we obtain $A_2$ from (3). $$A_2 = \frac{3\times 5 \times \sin \left(1.22355\right)}{2}=7.05234 \mathrm{\space m^2}$$
Finally, use (1) to get the value of $A_\mathrm{g}$. $$A_\mathrm{g} = 63.24548 - 7.05234 = 56.19314 \mathrm{\space m^2}$$
$\underline{\bf{Note}}$
Both methods can be used if and only if the vertex $E$ of the off-limit triangle lies beyond $E_0$. In other words, the $\measuredangle ECB \ge \phi$ , where $\phi = 38.682188^o$. $E_0$ is the point of intersection between $CE$ and the tangent to the circle from point $B$. We leave OP to justify this claim.