$40$ people are sitting around a table. In how many ways we can choose $5$ person of them so that between every $2$ persons that are chosen will be at least $3$ other people sited.
I think it is related to circular combination and permutation with repetition, but I do not know how?
I tried solving this question, but I could not. Please help me do that.

Fix a position at the table and assign it to be position $0$ and assign for example clockwise consecutively all postions from $0$ through $39$.
Any of your selections corresponds to a selection of increasing positions $p_1,\ldots , p_5$:
Now, consider the displacements between neighbouring positions:
$$d_k = p_{k}-p_{k-1} \text{ for } k=2,3,4,5 \text{ and set } d_1=p_1$$
Besides this the 5th position has a displacement of $d_6 = 39-p_5$ to the last position.
So, each selection corresponds to an integer solution of the equation
$$d_1 + d_2 + d_3 + d_4 + d_5 + d_6 = 39 \text{ where: }$$
So, introducing $d_k' = d_k - 4$ for $k=2,3,4,5$ we need the number of non-negative integer solutions of
$$d_1 + d_2' + d_3' + d_4' + d_5' + d_6 = 23 \text{ while } d_1+d_6 \geq 4$$
Since the number of non-negative integer solutions without any restrictions is $\binom{23+5}{5}$, we only need to subtract the solutions satisfying $0\leq d_1+d_6 = l$ for $l=0,1,2,3$:
$$d_2' + d_3' + d_4' + d_5' = 23-l$$
Noting that the number of non-negative integer solutions of $d_1+d_6 = l$ is $\binom{l+1}{1}=l+1$ we get
$$\binom{23+5}{5} - \sum_{l=0}^3(l+1)\binom{23-l+3}{3}$$