I have solved most of the problem, but I still can't figure out the last part.
If $P_{11}\in Syl_{11}(G)$, then $n_{11}\equiv 1 \pmod{11}$ and $n_{11} \mid 3\cdot 5$, i.e. $n_{11}=1$, so that $P_{11}$ is normal in $G$.
Hence an action of $G$ on $P_{11}$ induces an automorphism of $P_{11}$. The stabilizer of this action is $C_G(P_{11}$ (centralizer) and thus $G/C_G(P_{11})$ is isomorphic to a subgroup of $Aut(P_{11})$.
Since $Aut(P_{11})\cong (\mathbb{Z}/11\mathbb{Z})^{\times}\cong \mathbb{Z}/10\mathbb{Z}$, it follows that $|G/C_G(P_{11})|$ divides $10$.
On the other hand, since $P_{11}$ is abelian, $P_{11} \leq C_G(P_{11})$ and thus $|G/C_G(P_{11})|$ divides $165/11=15$.
Therefore, the two possibilities for $|G/C_G(P_{11})|$ are $1$ (in which case we are done) or $5$.
However, I can't figure out how $|G/C_G(P_{11})|=5$ would lead to $P_{11}\leq Z(G)$. Any help would be really appreciated.
As Sean Eberhard pointed out in the commentary, it is not true that the centre of such a group has elements of order $11$. Indeed, by applying the command $GAP$
we really make sure that there is exactly one non-Abelian group whose centre consists of three elements.
However, there is a simple way to prove this without using $GAP$.
Indeed, let $G$ be a group of order $165$. Let us prove that the centre of the group $G$ contains a subgroup of order 3. Hence it follows easily that there exists exactly one non-Abelian group of order $165$ and its centre consists of exactly three elements.