$\mathbf{H}$ is a $K\times N$ matrix of complex elements where $N>K$. I guess there must be some standard algorithms to find a set of orthonormal basis for its Kernel, i.e. set of all $\mathbf{X}\in\mathbb{C}^N$ such that $\mathbf{HX}=\mathbf{0}_K$. Also assume that the rank of $\mathbf{H}$ is $K$.
My field is not mathematics, however, I need to find the basis to solve a problem in my field. So citing some paper which proposes such an algorithm/analyses complexity etc. would be great, better if your own paper.
The Singular Value Decomposition (SVD) $H=USV^T$ does it.
All you have to do is to select the $N-r$ last colums of $V$, where $r$ is the rank of matrix $H$ (which is the number of non zero singular values)
See e.g., https://inst.eecs.berkeley.edu/~ee127a/book/login/l_svd_mat_prop.html