I have a problem understanding elliptic curves over finite non-prime fields, i.e., $E / {\Bbb F}_{p^k}$, where $p$ is prime and $k > 1$. Let's say we have a defining Weierstrass equation of the form $$E:\quad y^2 = x^3 + a x + b. $$ In my opinion, this then means that $a,b,x,y$ are all taken from the field the elliptic curve is defined over, i.e., $a,b,x,y \in \Bbb F_{p^k}$. Is this correct?
If so, how are then $a,b$ represented in a concrete case? If we take for instance $\Bbb F_{5^2}$ then, in my opinion, we can represent the field elements as $$\Bbb F_{5^2} = \{0, t, t + 3, 4t + 3, 2t + 2, 4t + 1, 2, 2t, 2t + 1, 3t + 1, 4t + 4, 3t + 2, 4, 4t, 4t + 2, t + 2, 3t + 3, t + 4, 3, 3t, 3t + 4, 2t + 4, t + 1, 2t + 3, 1\} $$ with the irreducible polynomial $p(t) := t^2 + 4t + 2$ and compute in this field accordingly. But this would mean that it is reasonable to consider e.g. the concrete elliptic curve, $$E_{(\Bbb F_{5^2}, t+3, 4t+4)}: \quad y^2 = x^3 + (t+3) x + (4t+4).$$ In order to test whether this is indeed an elliptic curve, I would then test the discriminant $\Delta := 4 a^3 + 27 b^2 \neq 0$ over this field and for $a:=t+3$, $b:=4t+4$. And similarly, if I want to test for supersingularity, I could e.g. use the Hasse invariant and test if the coefficient $c_{p-1}$ of the summand of $x^{p-1}$ in the polynomial $f(x)^{(p-1)/2}$, where $f(x) = x^3 + (t+3)x + (4t+4)$, is zero. Actually, I can do all these computations in SageMath. What I don't understand is, why I don't find any such things in books or papers. There are even in the general case of elliptic curves over non-prime fields $\Bbb F_{p^k}$, $k > 1$, only curves with integer coefficients, like $y^2 = x^3 + 31x + 4$. (I do understand that a multiple $31x$ makes sense also in $\Bbb F_{p^k}$.)
Why is this so? Aren't these curves then actually always just in the base fields? What about the other cases?