I have a square with sides of 10cm and I have a circle with radius of 6cm. Now I've to find the area of the circle that is inside of the square.Here is the graph

I had an idea of finding the area of the arc(90 degrees) and subtracting it from 25(100/4), but then I noticed that the area of arc would still include the areas which are outside of the square.

Looking at the positive quadrant, you see two points of intersection of the circle and the square, at $x=5$ and at $y=5$. Compute these. This gives two triangles, that you can compute the ara of, hopefully. The circle part inbetween them is just a fraction of the circle area depending on the angle $\alpha$ (in radians) between these two points (inner product can help to compute the cosine e.g.) namely $\frac{\alpha }{2\pi}A_c$ where $A_c = 36\pi$, the area of the circle.
Then times 4 as we have 4 quadrants.