Consider the following least squares problem in $X$:
$||AX-B||_2^2\rightarrow\min$, where $A$ and $B$ are known, real-valued matrices.
Is it there a regularisation method which ensures that the elements of $X$ are close to integers? (similar to how ordinary regularisation ensures that the elements of $X$ are bounded).
Note that such a penalty would generally lead to a non-convex objective. For example, the fractional part of $0$ and $1$ is $0$, however, the fractional part of $0.5$ is $0.5$. So, whichever objective function you choose, I suspect it will not be easy to calculate in practice.
Though in principle, this can be done. One way to is to penalize if the fractional parts of $X_i$ are not close to $0$ or $1$. The fractional part of $x\in\mathbb{R}$ is $x-\lfloor x \rfloor$, where $\lfloor x \rfloor$ is the integer part of $x$. So you could simply enforce that $x-\lfloor x \rfloor-1/2$ is close to zero. For example, you could minimize the following expression for some $\lambda>0$: $$ ||AX-B||_2^2+\lambda\left|\left|X-\lfloor X \rfloor -\frac{1}{2}\right|\right|_2^2 $$