First define $x_i$ is determenistic variable, $v_i$ is random variable. Consider the expression:
$$\text E(\sum_{i=1}^{\ n} x_iv_i)= \sum_{i=1}^{\ n} \text E (x_iv_i)=\sum_{i=1}^{\ n} \text E (x_i) \text E(v_i) \text +{cov}(x_i,v_i)= E(x')E(v)+cov(x,v)\space \space \space \space \space \space = x'E(v)$$
Now, let's say we want to define the minimal assumption required so that the expression is zero. We can impose a restriction that $E(v_i)$= 0. However, I believe that it is too strong of a restriction to say ALL elements of $v_i$ have to have an expected value of 0. The problem here is that this can get quite complicated as the answer depends on the values of x. But, the question still remains, is there a comprehensive way to formulate the minimal condition so that the expression equals zero?
This applies to the OLS estimator, as in its derivation of bias it's assumed that a similar expression is zero if $\text E(v_i)=0$. However, I believe the assumption is too strict.
You have already written the condition: letting $ {\bf \mu} = E( {\bf v})$ you must have $ {\bf x}' {\bf \mu} =0$, i.e, the inner product must be zero, i.e., the expected value of $ {\bf v}$ should be orthogonal to ${\bf x}$.