This answer gives a nice way of seeing why the category of sets is not isomorphic to its dual. I would like to know whether there is a proof from a certain different direction.
When considering the problem, the first idea I had was that there is precisely one arrow in $\mathbf{Set}$ going $\{1, 2\} \to \{1\}$, but two arrows in $\mathbf{Set}$ going $\{1\} \to \{1, 2\}$. Therefore, if the two categories $\mathbf{Set}$ and $\mathbf{Set}^*$ are isomorphic (via $\phi$, say, an arrow in the category of locally small categories), then $\phi$ cannot act trivially on both $\{ 1 \}$ and $\{1, 2 \}$ at the same time.
Is there a way of extending this into a proof that no such $\phi$ exists at all?
Yes.
In SET singletons are terminal objects. Then in its opposite category any singleton is an initial object. However in SET there is only one initial object (the empty set). This shows that the two categories are not isomorphic.