We have circle with radius $R$ and inside it we choose at random 2 points $A$ and $B$.Find the probability the circle with center A and radius $AB$ to be inside the circle(It is possible both circles to be tangent)
What I've concluded : the first point A has range from $[0;R]$ and the second point B must have range from $[0;R-OA]$ .But how should I find the probability?
First, I'll assume the two points are obtained with an uniform distribution in the circle.
You have a good conclusion there. Let's build from it. As you've noticed, once you fix the distance from $A$ to the center of the circle, $B$ has to fall inside the little circle:
$B$ can fall in an area of $\pi (R-r)^2$, from a total of $\pi R^2$. Here $r$ is the distance from the center of the original circle and $A$.
Now, if you use polar coordinates to express your points, the probability density function of the radius of a uniformly random point $A$ is $2r/R^2$. The only thing we have left is to compute the integral, which in words would mean "Compute the product of the probability of $A$ having distance $r$ from the center multiplied by the probability of $B$ falling close enough, and sum for each possible $r$":
$$\int_0^R \Big(\frac{R-r}{R}\Big)^2 \frac{2}{R^2}r \hspace{.2cm}dr = \int_0^R \Big(1-\frac{r}{R}\Big)^2 \frac{2}{R^2}r \hspace{.2cm}dr$$
Which is just a polynomial, so we can solve it. It's a bit tedious, so I'll leave it here with the answer of $\frac{1}{6}$ from wolfram alpha.