Problem: Say there are two points inside the circle; A and B, and they are both randomly drawn according to a uniform distribution where the boundary is the circumference of the unit circle/the surface of the unit sphere. What's the probability that the euclidean distance between two randomly drawn points inside a unit circle/sphere greater than 1?
This question has two versions; the 2D one and the 3D one. I have almost gotten down the expression of the integral in 2D one, but I still get stuck at the late stage of the problem, I haven't tried the 3-D version just yet, but I guess I will get stuck at a similar stage.
The following is my attempt on the 2-D version of the problem:
Phase 1:
for the sake of simplicity, we can "fix" the angle θ of A to a particular fixed value θa and only vary its r value in the polar coordinate system, so A could be (0,θa), (0.2,θa), (1,θa) etc. For B, we can vary everything including radius r and the angle θ of another point `B`.
Phase2:
The required probability should be equal to the sum of all of the conditional probability from r=0 to r= 1, where each increment of r is very very small:
ΣP{|A - B| > 1 | A= (r, θa) }
Upon taking this limiting process to the sum, this becomes a definite integral over the conditional probability density function from r=0 to r=1.
This is where I got stuck, I don't know how to transform the conditional probability to the conditional pdf inside the definite integral and possibly integrate it.
And after this 2-D version, the 3-D version is gonna be another beast that I need help in order to deal with that.
Note: These are the pictures of my drafts and my guesses, not sure whether they are helpful.


This is the solution for the 2D problem
Fix $A$ to be a distance $r$ from the center of the circle (i.e. fix $|A| = r$). Note that the probability density function of $|A|$ is
$$f_{|A|}(r) = \frac{2\pi r}{\pi} = 2r \qquad r \in [0,1]$$
The $2\pi r$ is the "area" of the circle representing all possible $A$ for which $|A| = r$; and the $\pi$ in the denominator is the total area of the unit circle.
Given that $|A| = r$, what is the probability that $|A-B| <1$?
Draw a circle of radius $1$ centered at $A$, and find the area common to both circles. After a bit of geometry, you should find that the answer is $2\cos^{-1} \frac r2 - \frac{r\sqrt{1-r^2}}{2}$. Thus,
$$\Bbb P(|A-B|<1 \, \big| \, |A| = r) = \frac 1\pi \bigg(2\cos^{-1} \frac r2 - \frac{r\sqrt{1-r^2}}{2}\bigg)$$
Finally, we use the law of total probability:
\begin{align} \Bbb P(|A-B|<1) & = \int_0^1 \Bbb P(|A-B|<1 \, \big| \, |A| = r) \cdot \color{red}{f_{|A|}(r)} \, dr \\ & = \int_0^1 \frac 1\pi \bigg(2\cos^{-1} \frac r2 - \frac{r\sqrt{1-r^2}}{2}\bigg) \cdot 2r \, dr \\ & = \frac 1\pi \int_0^1 \bigg(4r\cos^{-1} \frac r2 - r^2\sqrt{1-r^2}\bigg) \, dr \\ & = \frac 1\pi \bigg[ 2r^2 \cos^{-1} \frac r2 + 4\sin^{-1} \frac r2 - \frac 18 \sin^{-1} r - r\sqrt{4-r^2} + \frac 18 (r-2r^3)\sqrt{1-r^2} \bigg]^1_0 \\ & = \frac 1\pi \bigg(\frac{61}{48}\pi - \sqrt 3 \bigg) \\ & = \frac{61}{48} - \frac{\sqrt 3}{\pi} \end{align}
And of course, what we really want is
$$\Bbb P(|A-B|>1) = 1 - \Bbb P(|A-B|<1) = \frac{\sqrt 3}{\pi} - \frac{13}{48} = 0.2805 \dots$$
As pointed out above in red, you are missing a factor of $2r$ in the integrand.
Here's how to do the integration
\begin{align} & \int r\cos^{-1} \frac r2 \, dr \\ = & 4\int u\cos^{-1} u \, du && \text{substituted } u = \frac r2 \\ = & 4 \cdot \frac{u^2}{2} \cdot \cos^{-1} u - 4\int \frac{u^2}{2} \cdot \frac{-1}{\sqrt{1-u^2}} \, du && \text{integrated by parts} \\ = & 2u^2 \cos^{-1} u + 2\int u \cdot \frac{u}{\sqrt{1-u^2}} \, du \\ = & 2u^2 \cos^{-1} u + 2 \cdot u \cdot \big(-\sqrt{1-u^2}\big) - 2 \int 1 \cdot \big(-\sqrt{1-u^2}\big) \, du && \text{by parts again} \\ = & 2u^2 \cos^{-1} u - 2u\sqrt{1-u^2} + 2\int \sqrt{1-u^2} \, du \\ = & 2u^2 \cos^{-1} u - 2u\sqrt{1-u^2} + 2\int \sqrt{1-\sin^2 v} \cdot \cos v \, dv && \text{substituted } u = \sin v \\ = & 2u^2 \cos^{-1} u - 2u\sqrt{1-u^2} + 2\int \cos^2 v \, dv \\ = & 2u^2 \cos^{-1} u - 2u\sqrt{1-u^2} + 2 \bigg(\frac v2 - \frac{\sin (v) \cos (v)}{2} \bigg) \\ = & \frac 12 r^2\cos^{-1} \frac r2 - r\sqrt{1-\frac{r^2}{4}} + \sin^{-1} \frac r2 - \frac r2 \sqrt{1-\frac{r^2}{4}} \end{align}
The other integral can be done by writing
$$\int r^2\sqrt{1-r^2} \, dr = \int r \cdot \Big(r\sqrt{1-r^2}\Big) \, dr$$
and integrating by parts as suggested.