The given data is unclear. So i will describe the area that i will compute in the sequel, then do this in "two different ways".
It seems that we have to calculate the area of the difference $Q-D'$, where
$Q$ is the compact, full square with vertices in $(0,0)$, $(5,0)$, $(5,5)$, and $(0,5)$, and
$D'$ is one quarter of the disk $D$ centered in $(5,5)$ with radius $5$. So $D'=D\cap Q$. The length of the quarter circle is indeed $5\pi/2$.
We expect to obtain the area $5^2-\frac 14\pi\cdot 5^2=5^2\left(1-\frac\pi4\right)$. Numerically, this is $5.36504591506379\dots$, and fits the approximation $5.4$ from the OP.
Using differential and integral methods, the function to be considered is $f:[0,5]\to[0,5]$, $f(x)=5-\sqrt{5^2-(5-x)^2}$. This is extracted from the equation of the circle with center $(5,5)$ and radius $5$, isolating $y$ in $(x-5)^2+(y-5)^2=5^2$, and taking the right sign of the square root.
Note that this is not the function $g(x)=\sqrt{5^2-x^2}$, obtained from isolating $y$ in $x^2+y^2=5^2$, which would be the "other quarter circle" passing through $(5,0)$ and $(0,5)$. (This would be the reflection of the drawn arc taken w.r.t. the segment from $(5,0)$ to $(0,5)$.) Using $g$ we of course get $\frac 14\pi 5^2=25\pi/4$... The wrong answer mentioned in the OP.
So we compute the "area $A$ under the graph" of $f$ as
$$
\begin{aligned}
A &=
\int_0^5f(x)\; dx
\\
&=
\int_0^5\left(5-\sqrt{5^2-(5-x)^2}\right)\; dx
\qquad\text{Substitution: }x=5-5\cos t
\\
&
=25-\int_0^{\pi/2}
\sqrt{5^2-5^2\cos^2t}\; d(5-5\cos t)
\\
&=25-\int_0^{\pi/2}
5\sin t\; 5\cos t\; dt
=25-25\int_0^{\pi/2}
\sin^2 t\; dt
\\
&=
25\left(1-\frac \pi 4\right)\ .
\end{aligned}
$$
0
Bumbble Comm
On
The area of the square is $5^2$, and the area of the circle is $\pi5^2$. To find the enclosed area, subtract a quarter of the area of the circle from the area of the square.
The given data is unclear. So i will describe the area that i will compute in the sequel, then do this in "two different ways".
It seems that we have to calculate the area of the difference $Q-D'$, where
We expect to obtain the area $5^2-\frac 14\pi\cdot 5^2=5^2\left(1-\frac\pi4\right)$. Numerically, this is $5.36504591506379\dots$, and fits the approximation $5.4$ from the OP.
Using differential and integral methods, the function to be considered is $f:[0,5]\to[0,5]$, $f(x)=5-\sqrt{5^2-(5-x)^2}$. This is extracted from the equation of the circle with center $(5,5)$ and radius $5$, isolating $y$ in $(x-5)^2+(y-5)^2=5^2$, and taking the right sign of the square root.
Note that this is not the function $g(x)=\sqrt{5^2-x^2}$, obtained from isolating $y$ in $x^2+y^2=5^2$, which would be the "other quarter circle" passing through $(5,0)$ and $(0,5)$. (This would be the reflection of the drawn arc taken w.r.t. the segment from $(5,0)$ to $(0,5)$.) Using $g$ we of course get $\frac 14\pi 5^2=25\pi/4$... The wrong answer mentioned in the OP.
So we compute the "area $A$ under the graph" of $f$ as $$ \begin{aligned} A &= \int_0^5f(x)\; dx \\ &= \int_0^5\left(5-\sqrt{5^2-(5-x)^2}\right)\; dx \qquad\text{Substitution: }x=5-5\cos t \\ & =25-\int_0^{\pi/2} \sqrt{5^2-5^2\cos^2t}\; d(5-5\cos t) \\ &=25-\int_0^{\pi/2} 5\sin t\; 5\cos t\; dt =25-25\int_0^{\pi/2} \sin^2 t\; dt \\ &= 25\left(1-\frac \pi 4\right)\ . \end{aligned} $$