Generation of Hermitian invertible matrix with fixed number of non-zero elements

Question

I have a Hermitian matrix that is invertible, that is I can write it as:

$H = U^\dagger D U$,

where $U$ is a unitary matrix, and $D$ is a diagonal matrix with the eigenvalues of $H$, which must be real and distinct for it to be invertible.

If I want the resulting $H$ to have a certain number $d$ of non-zero elements, what are the constraints on $U$? And is there a way to generate $U$ randomly?