The problem asks to find a nonprincipal ideal and a nonprime irreducible in $R = \mathbb{Z}[\sqrt{-17}]$. Since $-17 \equiv 3 \pmod 4$, $R$ is the ring of integers of $\mathbb{Q}(\sqrt{-17})$. I know basic algebraic number theory, but I haven't spent much time working with concrete Dedekind domains.
In general, how should I go about finding a nonprincipal ideal, or a nonprime irreducible? For the second problem, it's not too difficult to show that a given rational prime number $p$ is irreducible in $R$. I thought I might try to find such a $p$ which also satisfied the condition that $X^2 \equiv 17 \pmod p$ was solvable, meaning that the ideal generated by $p$ would have to split as a product of two prime ideals in $R$. Should I just pick the $p$ at random until I find one that works?
There’s a nice theorem about rings of integers $R$ in an algebraic number field, that in case an ideal $I$ is principal, so $I=wR$, then $|R/I|=|\mathbf N(w)|$, where $\mathbf N$ is the field-theoretic norm. Notice that in your ring $R=\{m+n\sqrt{-17}\}$, the norm of $m+n\sqrt{-17}$ is $m^2+17n^2$.
Now I’ll exhibit an ideal $I$ such that $|R/I|=2$: It’s simply the ideal generated by $2$ and $1+\sqrt{-17}$. You prove this by noticing that $R/2R$ has four elements, and the ideal I described is clearly larger, but not all of $R$. But of course the norm form $m^2+17n^2$ never takes the value $2$, so the ideal is not principal. I’m sure you can show that the ideal in question is not principal directly, without calling in that useful theorem.
For an irreducible that doesn’t generate a prime ideal, you can take $2$.