In a classical implementation of Reversible Jump MCMC for Normal Mixtures (such as what Peter Green (1997) did in his paper On Bayesian Analysis of Mixtures with an Unknown Number of Components), people typically avoid non-identifiability by either imposing identifiability constraints on prior (e.g. impose a specific labeling or ordering to make parameters identifiable) or simply treating those parameters as an unordered multiset.
Despite that posterior non-identifiability (a.k.a. label-switching) is deprecated in Bayesian inference for many reasons, Peter Green (and other papers I can find) did not talk about what bad things can happen to the behavior of the Markov chain itself if we were to keep the parameters non-identifiably.
i.e. if we choose an exchangeable prior and a valid proposal so that the posterior is proper and theoretical convergence is guaranteed although the posterior manifests label-switching, can it still happen that the Markov chain fails to practically converge to the right posterior?
PS: according to this blog (section 3), the posterior will manifest multimodality with K! modes (K the number of labels) and it says the Markov chain will typically only explore one of the modes.
This seems to suggest that the chain will sometimes target $\pi$($\theta$,K|y)/K! instead of $\pi$($\theta$,K|y). This coincides with my result when I modified Peter Green's RJ-MCMC for normal mixtures by allowing non-identifiable parameters. I'm wondering if there's a resource that establishes this kind of behavior in more detail.
I run into this issue in my research. So far, the flat likelihood seems to be the root. Please Keep me posted if you do find more.