If $p_y$ is a probability function for a density, which depends on the value of $y$ (for example, $y$ might be the mean in the poisson distribution).
Assuming that $y$ is random -- i.e. unknown -- how would one estimate the probability that $X < x$ for $X$ being distributed with $p_y$ as the probability function?
My approach so far has been to just generate a large set of $y$ and then using those values to generate an equally large sets of $X$ and then counting the number of values less than $x$.
Would/could this be considered a classic estimator?
What you are looking at, basically, is the empirical distribution function. This is a very classical thing.
According to the law of large numbers, you will eventually end up with very good estimates for $P(X < x)$. Further, by CLT, your estimator for this probability is asymptotically normal. There is also a result on the uniformity of the convergence and other results related to testing whether the underlying distribution has a specific form.