Assume a competition with $N$ "players". We represent all "games" using matrix $P$ of shape $(N, N)$, where the entry $P[i,j]$ is the probability of player $i$ winning player $j$.
I would like to rank players according to that matrix.
One heuristic could be "binarize the probabilities to $1$ if $p_{i,j}>0.5$ and $0$ otherwise", then count the number of wins, and sort by that number. This is, however, pretty coarse, because we do not account for the actual probability. Another option is to sum the probabilities, but then we might end with $0.99+0.01+0.01 < 0.35+0.35+0.35$. I was thinking about something like Page Rank.
Wondering if there is a "canonical" problem that my setting falls into, and what are the possible directions.
I think the comments answer the question pretty well, but I'm just going to elaborate a bit more.
Probably the most reasonable (and simplest) way of ranking the competitors is by their expected number of wins in a tournament where everyone plays each other once. This is also the same as ranking them based on a 'limiting' leaderboard, where the tournament carries on indefinitely with everyone playing each other infinitely often, but with each match occurring in equal proportions.
This boils down to calculating $\sum_{j=1}^Np_{ij}$ for each player $j$, and ranking in order of size.
But what happens when two players have an equal expected number of wins? The best we can do for this is to then compare the players based on their winning probabilities against each other. If these are equal, we can reasonably give them the same rank. However, depending on what characteristics you want your tournament to favour, you could compare them using different metrics such as their minimum win probability across all their opponents.