The Goldbach Conjecture asserts:
It is possible to write every even number greater that 2 as the sum of two primes.
Assume I can prove that the Goldbach Conjecture is unprovable from the Peano arithmetic. That would mean I could not prove the conjecture, nor the negation.
But wouldn't that mean that I can not find a counter-example? Because that would be a finite proof to show that its negation is true. And if I can't find an counter example, doesn't that imply that the conjecture is true?
In other words, wouldn't it be a proof to the conjecture if I can show that it is unprovable?
Unprovable ≠ Undecidable. If PA can prove neither the conjecture nor its negation, it is undecidable in PA.
If you ever prove such a result, you certainly cannot be working within PA, because PA cannot prove that it cannot prove something, otherwise PA can prove that it cannot prove contradiction, which is impossible by Godel's second incompleteness theorem (assuming PA is consistent). Thus there is no paradox; your proof of unprovability over PA has to be a proof in some system other than PA.
So let's fix your foundational system MS as any reasonable formal system (at least proves existence of a model of PA), and let's reason within MS. If PA does not disprove Goldbach, then PA does not prove its negation, which is a $Σ_1$-sentence, and hence that negation cannot be true because PA is $Σ_1$-complete. So if PA does not disprove Goldbach, then Goldbach is actually true.
The caveat is that you are working within MS, so you should at least convince yourself that MS is consistent. Observe that if MS is inconsistent, you (working within MS) would be able to prove that PA does not prove Goldbach, but that would not mean anything.
Note that this argument does not necessarily apply to other open problems. For example the twin-prime conjecture can be written as a $Π_2$-sentence, and currently it is not known to be equivalent to a sentence of lower complexity, so the argument does not work, since PA is not $Σ_2$-complete.