Considering a set of n points $\{P_i\}_{i \in [1,n]}$, I've defined a distance matrix $\{D_{ij}\}_{i,j \in [1,n]} = \lVert Pi - Pj \lVert$ and I would like to define a connectivity matrix $\{C^p_{ij}\}$ according to the $p$ closest neighbours. If $P_j$ is among the $p$ closest neighbours of $P_i$ then $C^p_{ij} = 1$ else $C^p_{ij} = 0$.
When $p=1$, I can write
\begin{align} C^1_{ij} = \left\{ \begin{array}{l} 1 \text{ if } j = {\arg\min}_k(D_{ik})\\ 0 \text{ else } \end{array} \right. \end{align}
what would be the correct notation in the general case ($p$ > 1)?
You could do it like so:
$$C_{ij}^k = \begin{cases}1 & \text{if }|\{m| D_{im} < D_{ij}\}| < k\\ 0&\text{otherwise}\end{cases}$$
however, I would advise you to instead describe what you are doing in text. Also, be careful what happens if ties are possible.