Let $A=\{h\} \cup\{h_n\}$ where $h_n \in H$ is a sequence such that $h_n \to h$.
Is $A$ a compact subset of $H$?
I think so, since every subsequence of $A$ has the limit $h$ which is in $A$.
Similarly, $A\backslash \{h\}$ is not a compact subset of $H$.
Is this correct?
Yes. Another way of seeing it: given an open cover of $A$, take one open that covers $h$, and then only finitely many $h_n$ will be outside and can be covered by finitely many open sets in the cover.
The case about $A\setminus\{h\}$ is slightly different. It could be compact if all $h_n$ become equal eventually.