What is the definition of the non split Cartan subgroups of $GL_2(\mathbb{F}_p)$? And what are the explicit expression of a matrix of this subgroups?
I read on "Modular Functions of One Variable III" by W.Kuijk and J.P.Serre, that the non split Cartan subgroups of $GL_2(\mathbb{F}_p)$ corresponding to $W'$ or $W"$ consists of those elements of $GL_2(\mathbb{F}_p)$ which have $W'$ and $W"$ as eigenspacese, where $W'$ is any one-dimensional subspace of $\mathbb{F}_{p^2}$ which is not induced by a subspace of $\mathbb{F}_p$, and $W''$ is the conjugate of W' over $\mathbb{F}_p$.
I understand that $W'=\langle (a,b)\rangle$ with $a,b\in\mathbb{F}_{p^2}-\mathbb{F}_p$ and $W''=\langle(\phi(a),\phi(b))\rangle$ where $\phi:\mathbb{F}_{p^2}\longrightarrow \mathbb{F}_{p^2}$ is the Frobenius automorphism.
Is it true (about $W'$ and $W''$)?
The definition of $W''$ given $W'$ looks right, but I think you need a little more restriction on $a,b$ to ensure that $W'$ is not induced from a subspace coming from $\mathbf F_p$. For example if $a = b \in \mathbf F_{p^2} \smallsetminus \mathbf F_p$ then the subspace $W'$ is simply the subspace where the two coordinates are equal, which is induced by the subspace $\langle (1,1)\rangle \subseteq \mathbf F_p^2$.
The LMFDB has an alternative but equivalent description of this group at https://www.lmfdb.org/knowledge/show/gl2.nonsplit_cartan, so seeing the equivalence between the two definitions would be nice.