Assuming $F$ is a finite field such that $F[\sqrt{2}]$ and $F[\sqrt{3}]$ are both fields, I am trying to prove that they must both be isomorphic. Or is there a counterexample? Is there a counterexample where $F$ has prime order? I have looked hard for one and cannot find any.
For the avoidance of doubt $F[\sqrt{2}] := F[X]/(X^2-2)$, so assume that $X^2 -2$ and $X^2-3$ are irreducible over $F$.
On
Let $p$ be the characteristic of $F$. If $a$ and $b$ are non-squares mod $p$, it's a standard result that $ab$ is a square. This is because the nonzero squares form a subgroup of $(\mathbb{Z}/p)^\times$ of index $2$.
Under your assumption, both $2$ and $3$ are non-squares mod $p$, so there is some $x\in\mathbb{Z}$ with $x^2 \equiv 6\pmod{p}$. Then $F[\sqrt{2}]$ is isomorphic to $F[\sqrt{3}]$ via a map sending $\sqrt{2}$ to $x/\sqrt{3}$.
If neither $2$ nor $3$ are squares in the finite field $F$, then both $F[\sqrt{2}]$ and $F[\sqrt{3}]$ are extensions of $F$ having degree $2$, so both have $|F|^2$ elements. Hence they're isomorphic: if $p$ is a prime, then any two fields of cardinality $p^n$ ($n>0$) are isomorphic, because both are the splitting field of $x^{p^n}-x$.
The only case in which they aren't isomorphic, is when one among $2$ and $3$ is a square and the other isn't.