There was previous task was same but with $N = 185$. And I prove it by showing that number of Sylow subgroups is 1 for every prime $p\mid N$. But there I have some options $N_5 \in \{1, 51\}$, $N_{17} = 1$, $N_3 \in \{1, 85\}$.
I've tried to get contradiction from $N_5 = 51$ or $N_3=85$, but I didn't manage to do it
I understand that it's impossible to have $N_5 = 51$ and $N_3=85$ at the same time.
On
The key to $N=255$ is the observation that it is squarefree and coprime with its totient function.
Let $n_q$ denote the number of Sylow $q$-subgroups.
So, let us begin by the observation than any group $G$ of order $15$ is cyclic. Indeed, let us merely note $n_3\mid 5$, $n_3\equiv 1\text{ mod }3$ and so $n_3=1$. Similarly, $n_5\mid 3$ and $n_5\equiv 1\text{ mod }5$ which gives that $n_3=1$. Thus, we see that both the Sylow subgroups $P,Q$ of $G$ are both normal. Since evidently $P\cap Q=\{e\}$, and $PQ=G$ (since $\displaystyle |PQ|=\frac{|P||Q|}{|P\cap Q|}=\frac{15}{1}=15=|G|$) we may conclude that $G\cong P\times Q\cong\mathbb{Z}/3\times\mathbb{Z}/5\cong\mathbb{Z}/15$.
Now, to the case of $|G|=255$. Let $P$ be a Sylow $17$-subgroup. Note then that $n_{17}\mid 15$ and $n_{17}\cong 1\text{ mod }17$ gives $n_{17}=1$. Thus, we see that $P\unlhd G$. Now, here comes the step that is different than the standard approach. Since $P$ is normal we have that $G$ acts on $P$ by conjugation, giving us a group map $G\to\text{Aut}(P)$ whose kernel is $C_G(P)$. Now, note that since $P$ is just $C_{17}$ we have that $|\text{Aut}(P)|=16$. Note now that since $(|G|,16)=1$ you can conclude from the first isomorphism theorem that the image of the group map $G\to\text{Aut}(P)$ is trivial, and so the kernel is everything. This says that $C_G(P)=G$, and so $P\subseteq Z(G)$. Note then that $G/P$ is a group of order $15$ which, by what we have said in the previous paragraph, is cyclic. Now, it is a common theorem that if you mod out a group by a subgroup of its center and you get something cyclic, your group must have been abelian the whole time. So, from all of this we can conclude that $G$ is abelian in which case, by any method that tickles your fancy, you get that $G$ must just be $\mathbb{Z}/255$.
While the above may be ad hoc, it was actually just a simple application of the following (EXTREMELY USEFUL!) theorems:
Theorem: Let $G$ be a finite group and $N\unlhd G$. Then, if $(|G|,|\text{Aut}(N)|)=1$ then $N\leqslant Z(G)$.
and
Theorem: If $G$ is a group and $N\leqslant Z(G)$ such that $G/N$ is cyclic of finite order, then $G$ is abelian.
These allow us to prove the more general theorem:
Theorem: Let $G$ be a finite group such that $(|G|,\varphi(|G|))=1$ (where $\varphi$ is the totient function), then $G$ is cyclic.
As a side note, a cool fact is that the integer $n$ has the property that the ONLY subgroup of order $n$ is $\mathbb{Z}/n$ is equivalent to $(n,\varphi(n))=1$.
The proof of the above can be found on my blog here.
On
$$|N|=255=3\cdot 5\cdot 17$$
By Sylow theorem, the $\,17-$Sylow sbgp. $\,P_{17}\,$ is normal in $\,N\,$, so taking any $\,5-$Sylow sbgp. $\,P_5\,$, we get a sbgp. $\,K:=P_5P_{17}\,$ of index $\,3\,$ in $\,N\,$, and since this last is the minimal prime dividing $\,|N|\,$ , we get that $\,K\triangleleft N\,$ .
It's easy to prove that any group of order $\,85\,$ is cyclic , and thus its automorphism group has order $\,\phi(5)\phi(17)=64\,$ , so the only homomorphism $\,C_3\to Aut(K)\,$ ( the first group is the cyclic one of order $\,3\,$) there is is the trivial one (why?), and since by the above it follows that $\,N\cong K\rtimes P_3\,$ (with, of course, $\,P_3\cong C_3\,$), we get this semidirect product is in fact a direct one of abelian groups and, thus, $\,N\,$ is abelian and, in fact, cyclic.
On
By Sylow, the 17-Sylow subgroup $P_{17}$ is normal in $N$. The group $G/P_{17}$ is of order $255 / 17 = 3\cdot 5$. By $5 \not\equiv 1\bmod 3$ and the classification of $pq$-groups, $G/P_{17}$ is cyclic.
The cyclic group $G/P_{17}$ has a normal subgroup of index $5$. By the corresponcence theorem, also $G$ has a normal subgroup $N$ of index $5$ (with $P_{17} \subseteq N$). We have $\#N = 255/5 = 3\cdot 17$. By $17\not\equiv 1\bmod 3$ and the classification of $pq$-groups, $N$ is cyclic. So $N$ has a unique $3$-Sylow subgroup $P_3$. Then $P_3$ is also a $3$-Sylow subgroup of $G$. As all $3$-Sylow subgroups of $G$ are conjugate and $N$ is normal in $G$, we get that all $3$-Sylow subgroups of $G$ are contained in $N$. So $P_3$ is also the unique $3$-Sylow subgroup of $G$ and therefore, $P_3$ is normal in $G$.
In the same way, we show that there is also a unique (and hence normal) $5$-Sylow subgroup $P_5$ of $G$. Hence all Sylow subgroups $P_3$, $P_5$ and $P_{17}$ are normal in $G$ and cyclic (by their prime order). We get that $$G \cong P_3 \times P_5 \times P_{17} \cong \mathbb{Z}/3\mathbb{Z} \times \mathbb{Z}/5\mathbb{Z} \times \mathbb{Z}/17\mathbb{Z}$$ is commutative.
(The Chinese remainder theorem takes us one step further implying that $G \cong \mathbb{Z}/255\mathbb{Z}$ is even cyclic.)
On
Here is yet another solution, making use of the observation in the initial question that it's "impossible to have $N_5=51$ and $N_3=85$ at the same time." (As otherwise $G$ would contain $85\cdot 2 + 51 \cdot 4 = 374 > 255 = \#G$ elements of order $3$ or $5$.)
The observation implies that at least one of the groups $P_3$ and $P_5$ is normal in $G$ and hence $U = P_3 P_5$ is a subgroup of $G$. (For $p\in\{3,5,17\}$ let $P_p$ denote a $p$-Sylow subgroup.) As $P_3$ and $P_5$ have trivial intersection, $\#U = 3\cdot 5 = 15$. By the classification of $pq$-groups and $5\not\equiv 1\bmod 3$, the group $U$ is cyclic.
Moreover, $U$ is a complement of the normal subgroup $P_{17}$ in $G$ (as $U\cap P_{17} = \{e\}$ and $\#U \cdot \#P_{17} = \#G$). So $G \cong P_{17} \rtimes_\phi U$ with a group homomorphism $\phi : U \to \operatorname{Aut}P_{17}$. We have that $\#\operatorname{im}\phi\mid\#\operatorname{Aut}P_{17} = \varphi(17) = 16$. By the fundamental theorem on homomorphisms, moreover $\#\operatorname{im}\phi \mid \#U = 17$, forcing $\#\operatorname{im}\phi = 1$. Hence the homomorphism $\phi$ is trivial and the semidirect product $G$ is indeed the direct product $G\cong P_{17} \times U$ of two abelian groups. Therefore $G$ is abelian (and then cyclic, by the Chinese remainder theorem).
On
This recent question leads to still another solution, maybe the most elementary one. Again we make direct use of the observation in the initial question that it's "impossible to have $N_5=51$ and $N_3=85$ at the same time." (As otherwise $G$ would contain $85⋅2+51⋅4=374>255=\#G$ elements of order $3$ or $5$.)
By $N_{17} = 1$, the $17$-Sylow group $P_{17}$ is normal in $G$. By the above observation, there is a normal subgroup $N$ of $G$ of order $3$ or $5$. By the classification of $pq$-groups, the groups $G/P_{17}$ of order $15 = 3\cdot 5$ and $G/N$ of order $65 = 5\cdot 17$ or $51 = 3\cdot 17$ are both cyclic.
Consider the group homomorphism $$\phi : G \to G/P_{17} \times G/N, \quad g\mapsto (gP_{17},gN).$$ Then $\ker\phi = P_{17} \cap N = \{e\}$, so $\phi$ is an injection and $G$ is isomorphic to a subgroup of the abelian group $G/P_{17} \times G/N$ (abelian as a direct product of cyclic groups). Hence $G$ is abelian.
Suppose $G$ is a finite group of order $|G| = 255 = 3 \cdot 5 \cdot 17$. We will show that $G$ must be cyclic.
As you noticed, there must be only one Sylow subgroup $P$ of order $17$. Thus $P$ must be a normal subgroup. Since $17$ is prime the subgroup $P$ is cyclic, say $P = \langle a \rangle$. It is not difficult to see that every group of order $15$ is cyclic, so $G/P = \langle bP \rangle$ is also cyclic. Now $(bP)^{|b|} = P$, so $|bP| = 15 \mid |b|$.
If $|b| = 255$ we are done. Therefore we can assume $|b| = 15$. We will show that $ab = ba$ which implies that $ab$ has order $255$ since $15$ and $17$ are coprime. Because $P$ is normal, $bab^{-1} = a^i$ for some integer $i$. Using this fact we get $b^2ab^{-2} = ba^ib^{-1} = (bab^{-1})^i = a^{i^2}$ and by induction $b^kab^{-k} = a^{i^k}$ for all $k \geq 1$. Thus $a = b^{15}ab^{-15} = a^{i^{15}}$, which gives $i^{15} \equiv 1 \mod{17}$. By Fermat's little theorem $i^{16} \equiv 1 \mod{17}$ and since $15$ and $16$ are coprime, $i \equiv 1 \mod{17}$. Therefore $bab^{-1} = a$, proving the claim.
This exercise is a special case of a more general fact. Using the idea of this solution and the fact that in a group of squarefree order the Sylow subgroup corresponding to the largest prime is normal, you can prove that every group of order $n$ is cyclic when $n$ and $\varphi(n)$ are coprime ($\varphi$ is the totient function). This is done by Tibor Szele in the following short article (in German).