Assume that $m, n$ and $r$ are natural numbers such that $m=n+r^2$. Consider that we have $n+r$ families. $n$ families have only one member and $r$ families have $r$ members. Therefore, we have $m$ persons. We want to choose $k$ people.How many choices do we have, When in our choices are not two members of a family.
For example when $r=1$, we have $\left(\begin{array}{c} n+1 \\ k \end{array}\right)$ choices. Also, when $r=2$ and $k=2$, we have $\left(\begin{array}{c} n+4 \\ 2 \end{array}\right)-2$ choices.
From lulu's hint
$$\sum_{a=a_{min}}^{a_{max}}{n\choose a}\cdot{r\choose k-a}r^{k-a},$$
which $a_{max}$ and $a_{min}$ depends on $a \le\min(n,k)$ and $(k-a)\le r$, or combined as $\max(k-r,0)\le a\le\min(n,k)$. e.g. $r=2,\ k=2$ and $n=5$
$$\begin{alignat}{} \sum_{a=0}^{2}{5\choose a}\cdot{2\choose 2-a}r^{2-a}\\ = 1\cdot1\cdot 2^2 + 5\cdot2\cdot2^1 + {5\choose2}\cdot1\cdot2^0\\ = 4+20+10=34, \end{alignat}$$
which is the same as
$${5+4\choose2}-2.$$
But I'm wondering whether it is possible to represent the formula as $n,r,k$ only(without summation mark).