The probability that a specific relationship exists between two objects $(x,y)$ in a graph using probability model. How to write it down using a math notation. for example:
$f(z)= p$ if there is a relation between $x$ and $y$ $f(z)=0$ if no relation.
It looks like you are trying to define a piecewise function. You can write this in the following way:
$f(z)=\begin{cases} p & \text{ if } R(x,y),\\ 0 &\text{ otherwise.} \end{cases}$
Here, $R$ is whatever relation you have on $x$ and $y$. As your function also depends on $x$ and $y$, you can write $f_{x,y}(z)$ or $f(z, x,y)$ instead of $f(z)$.