What happens, if I change the acceptance criterion in a Metropolis Monte Carlo algorithm?
I do know the classic proof of detailed balance, which for symmetric transition matrices gives a set of states according to the Boltzmann distribution.
Would detailed balance still be fulfilled, if I change the acceptance criterion to e.g. include a third case or use an acceptance function different than the Boltzmann distribution? Would this just result in an unphysical but mathematically valid distribution of states?
As you can see Introduction to the Monte Carlo Method, by Andrea Pelissetto, on pag. 143 (section 6), detailed balance is satisfied as long as the acceptance probability for a move is \begin{equation}\tag{6.143} \frac{F(z)}{F(1/z)} = z \end{equation} where $$ z = \frac{\pi_j p_{ji}}{\pi_ip_{ij}} $$ and the $\pi$ denote the equilibrium probability of a given state and the $p$ the probability of proposing a given transition.
As long as this condition is satisfied, the Metropolis algorithm will work. Of course, since $F(z)$ is a probability, it has to be less than 1: $$ F(z) < 1\,.$$
Moreover, since the probability to undergo the reverse transition must also be less than one, we have $$ F(1/z) < 1 $$ and substituting this equation in the (6.143), we get $$ F(z) < z\,. $$ Therefore, it must be \begin{equation}\tag{1} F \leq \min(1,z) \end{equation} and if you want to maximise the acceptance probability (which you probably do, so that your system evolves as fast as possible), you can set $$ F(z) = \min(1,z)\,. $$
This leads us to finally answer your questions:
Can I choose a different F(z)? Yes, as long as the condition 1 is satisfied. E.g., $1/(1+z)$ is perfectly valid. However, your system will be moving slower than it should.
Do I have to use the Gibbs distribution? No. You can use the metropolis algorithm to sample any distribution you like, however, you are likely to be dealing with systems distributed according to the Gibbs distribution.
What if I introduce a third option, apart accepting or rejecting the function? Then it's not the Metropolis algorithm any more, but it's probably something more similar to the heath bath algorithm (same file and section as before (section 6))