The transfer principle doesn't apply to second-order logic. For example, if I take a standard statement. $$\text{A lower bounded set of Reals has a greatest lower bound}$$
Is false for the hyperreals: If you take, say, the set of infinitesimals, they are bounded below by $-1$, but have no greatest lower bound.
It is true if we replace set with hyperset. There is no hyperset of (only the) infinitesimals.
Does the transfer principle apply to second-order logic if we transfer sets to hypersets and relations to hyperrelations?
As for what a hyperset is, if we use the ultrafilter definition, it is simply a sequence of sets of standard elements. For example, a hyperset might be:
$$\langle\{3\},\{3,3.1\},\{3,3.1\},\{3,3.1,3.14\},\{3.14,3.141\},\{3.1,3.1415\}\dots \rangle$$
This will contain for example
$$\langle3,3,3.1,3.14,3.14,3.1415,\dots\rangle$$
a number infinitely close to $\pi$.
Yes, the transfer principle will continue to hold if you pay careful attention to the ultraproduct construction to see what the correct transfer principle says. Recall that the transfer principle is essentially just the fact that an ultrapower of a model is elementarily equivalent to the original model.
To work with second-order models,$^{\text[1]}$ you would begin with a structure $$ M = (A, S, \ldots) $$ where $A$ is a set of objects, $S$ is a set of subsets of $A$, and $\ldots$ is a signature which can include functions and relations on $A$ and/or $S$. Among these will be the set membership relation $\in$. You then take an ultrapower of $M$ to obtain a new model $M^*$ of the form $$ M^* = ( A^*, S^*, \ldots) $$ By elementarily equivalence, there is a transfer principle between $M$ and $M^*$. Of course $S^*$ will not be the full powerset of $A^*$, usually. In the correct transfer principle, the set quantifiers in $M^*$ will range only over $S^*$, not over the powerset of $A^*$.
The elements of $S^*$ are equivalence classes of sequences of elements of $S$ - I believe you are calling these "hypersets". An element $\alpha$ of $A^*$ will belong to a set $X$ of $S^*$ if the collection of indices $i$ for which $\alpha(i) \in X(i)$ is in the ultrafilter.
Note [1]: to connect this with the literature, you have to be careful of terminology. A system where you have both natural numbers and real numbers is often called "second-order", as in second-order arithmetic. So if you also have subsets of the reals, then this would be a "third-order" system in that sense. Of course, you are indeed looking at the second-order theory of the real line - the terminology about the order of a theory can be confusing.
There has been a great deal of recent research on nonstandard analysis for second-order systems. For example,