Exercise 5.2.4. in Ziegler, Tent: A Course in Model Theory states:
If (the $L$-structure) $\mathfrak{A}$ is $\kappa$-saturated, then all definable subsets are either finite or have cardinality at least $\kappa$.
This statement seems only correct, if $|\mathfrak{A}|\geq\kappa$, as the set $A$ defined by $\top$ could otherwise have a cardinality $\aleph_0\leq|A|<\kappa$ for $\kappa>\aleph_0$.
I didn't manage to come up with a viable idea for proofing the statement. I would be thankful for any hint.
A $\kappa$-saturated structure has cardinality at least $\kappa$: otherwise, consider $\{x\not=a: a\in\mathfrak{A}\}$ ... (Fine, unless it's finite.)
So the situation you're envisioning can't occur.
EDIT: And indeed, as Alex observes below, this should be thought of as a hint to the whole problem.