"Assume F is a consistent formalized system which contains elementary arithmetic. Then F does not prove F is consistent."
So, if Peano arithmetic is consistent, it can't prove "Peano arithmetic is consistent". But what if Peano arithmetic can prove "Peano arithmetic is inconsistent"? Would it still possibly be consistent?
Yes, it is consistent with PA that PA is inconsistent.
How does this work? By Goedel's completeness theorem, consistency is the same as satisfiability: a consistent theory has a model. So if PA+"PA is inconsistent" is consistent, it better have a model. How can that be?
Well, a model of $PA$ is a collection of things called "numbers", together with a notion of addition and of multiplication, which satisfy some properties. The usual natural numbers is merely one such model. In general, a nonstandard model of $PA$ will have "infinite" elements: that is, besides the usual natural numbers, such a model will have lots of $c$s such that $c>0, c>1, c>2, . . . $
Now, if a model thinks that PA is inconsistent, that means the model thinks "There is a proof of 0=1 in PA" - or, more specifically, there is a number coding such a proof. So the model will have an element $c$ which it thinks codes a proof of inconsistency. But, if such a $c$ is infinite, then it doesn't really code a proof! It codes a "nonstandard proof", something the model thinks is a proof.
In a comment, you ask how Goedel's theorem is really an "incompleteness theorem" if it leaves the possibility of PA proving its own inconsistency. Let's address that:
First of all, we usually assume that PA is true. A fortiriori, this means PA is consistent - and since it's true, that means it doesn't prove that it's inconsistent.
That's a rather strong assumption. Can we do better? Well, Goedel observed that we don't really need PA to be true, merely that it is correct about sentence of the form "$\exists x[P(x)]$" (and in fact even less is necessary - we can restrict attention to such sentences where $P$ uses only bounded quantifiers).
This was how Goedel left it, and he and others asked if the situation could be improved:
This was answered affirmatively by Rosser.
Here's what he did: whereas Goedel looked at the sentence $$G_{PA}:\quad\mbox{"There is no PA-proof of me,"}$$ Rosser looked at the sentence $$R_{PA}:\quad\mbox{"For any PA-proof of me, there is a shorter PA-disproof of me"}.$$ Via Goedel's original argument, $PA$ doesn't prove $R_{PA}$ if $PA$ is consistent; the new part is the other direction.
Suppose $PA$ proves $\neg R_{PA}$, via a proof $\pi$. Then $PA$ proves "There is a proof of $R_{PA}$ with no proof of $\neg R_{PA}$ shorter than it" (since this is exactly what $\neg R_{PA}$ says). Well, there are only finitely many proofs shorter than $\pi$ - so we can check them one by one in $PA$. Since $PA$ is consistent, this means there is in fact no proof of $R_{PA}$ at all; so in particular, there is no proof of $R_{PA}$ shorter than $\pi$. But that means that our check of proofs-shorter-than-$\pi$ shows that, in fact, $R_{PA}$ is true! Since it verifies that any proof of $R_{PA}$ would have to be longer than $\pi$.
Goedel's original argument was an incompleteness result: it showed that under reasonable assumptions $PA$ is incomplete. It's just that those assumptions were stronger than mere consistency. Rosser got the assumptions down to the absolute minimum (any inconsistent theory proves everything), but that shouldn't be seen as devaluing Goedel's argument.