If i want to let $\mathbf H \vec f $ become zero,and $\mathbf H$ is a N by M matrix,$\vec f$ is a M by $1$ column vector,and both $\mathbf H$ and $\vec f $ cannot be a zero matrix or zero column .
Is there some condition that $\mathbf H$ must be satisfied ? for example,N > M? or M < N or M=N?
The set of vectors $\vec{f}$ that verify $H\vec{f} =0$ is called the kernel of $H$. Therefore your question is about the conditions on $H$ in order to have a non-zero kernel.
To give you a small example, whichever the dimensions of the matrix are, if it is filled with 0's then all vectors are in the kernel. Thus, the dimensions are not a necessary condition.
What you are interested in, is the rank of the matrix $H$. It is defined as the dimension of the image of $H$, or "column space", i.e. the set of all linear combinations of the column-vectors of your matrix.
This rank has the property $rank(H) \leq min(M,N)$, and also that $rank(H) + dim(kernel(H)) = N$.
So the necessary condition for having a non-zero kernel is to have a rank lower than N. This is immediately achieved if $M<N$ (sufficient condition).
However, if $M \geq N$, then you need to have a rank that is smaller than its maximum theoretical value considering the matrix dimensions. Matrices that have this property are called "singular" or "degenerate", and are non-invertible.