A book I'm reading on category theory says that if $A$ and $B$ are topological spaces and $f:A\to B$ is continuous, then the "dual image" map $$f_*(U)=\{\,b\in B\mid f^{-1}(b)\subseteq U\,\}$$ restricts to open sets; that is, $f_*:\mathcal{O}(A)\to\mathcal{O}(B)$. (So then it's right adjoint to $f^{-1}:\mathcal{O}(B)\to\mathcal{O}(A)$.)
This seems wrong, since it would imply for example (taking $U=\varnothing$) that the image of a continuous function is always closed.
Are there natural conditions under which it does make sense to restrict to open sets?
You’re right that there’s a problem here.
Let $A=(0,1)\times(0,1)$ and $B=(0,1)$, each with the usual topology, and let $$f:A\to B:\langle x,y\rangle\mapsto x$$ be the projection to the $x$-axis; this is a continuous, open map. Let $$U=\{\langle x,y\rangle\in A:y>2x-1\}\;.$$ $U$ is open in $A$, but
$$f_*(U)=\left(0,\frac12\right]\;,$$
which is not open in $B$.
You do get the result if $f$ is closed and continuous. In that case let $U$ be open in $A$, and let $F=A\setminus U$. Suppose that $b\in B$; then $f^{-1}[\{b\}]\subseteq U$ iff $b\notin f[F]$, i.e., iff $b\in B\setminus f[F]$, so $f_*(U)=B\setminus f[F]$, which is open in $B$.