I am interested in the set of oriented rectangles, that are centered on the origin, and can be described by their width, height and angle.
I am looking for a "good" parameterization that is non-ambiguous (a rectangle should correspond to only one set of parameters) and continuous (close rectangles should have close parameters).
For example, if I parameterize rectangles with a positive width and height and a 2D rotation in $[0, 2\pi)$, the parameterization is ambiguous as adding $\pi$ to the angle produced the same rectangle. If I restrict the angle to $[0, \pi)$, the parameterization is still ambiguous as adding $\pi/2$ to the angle and switching the width and the height produce the same rectangle. If I restrict the angle to $[0, \pi/2]$, the parameterization becomes unambiguous, but it is not continuous anymore, as the parameterization switches height and width when the angle crosses the 0/$\pi/2$ boundary.
At first I thought I could fall back to the parameterization of ellipses, but there is a major difference between rectangles and ellipses: when an ellipse collapse into a circle, its only parameter is its size. When a rectangle collapses into a square, it still has a rotation component. And the parameterization of this rotation should respect the order of rotational symmetry of 4 of the square, which is not shared with general rectangles which have an order of 2.
To be more rigorous, what I want is a homeomorphism between the space of rectangles and a manifold embedded in some $R^n$ (thanks @joriki). I think I also want at some point the manifold to be describable by a set of equations to be usable in practice, like $\|p\|^2=1$ for $U(1)$, but let's put that aside for now.
My question is then, what is a parameterization of rotated rectangles with these properties? Is it possible to find one, or on the opposite, can we prove that no such parameterization exists?
I am absolutely not knowledgeable in this matter and probably put inappropriate tags to this question, apologies in advance.
EDIT: I think this question might have to do with Lie groups and representation theory, and whether or not this group accepts a faithful representation (which I fear it doesn't).
I believe this yields an embedding of the $3$-dimensional manifold of rectangles in $\mathbb R^5$ that fulfils your criteria:
Denote the polar coordinates of two midpoints by $(a,\phi)$ and $(b,\psi)$, with $\psi\equiv\phi+\frac\pi2\pmod{2\pi}$. A rectangle is represented by
$$\left(s,\epsilon\cos2\phi,\epsilon\sin2\phi,\cos4\phi,\sin4\phi\right)\;,$$
where $s=\log\sqrt{ab}$ is the scale of the rectangle and $\epsilon=\log\sqrt\frac ab$ is its eccentricity. Adding $\pi$ to $\phi$ leaves this representation invariant. Adding $\frac\pi2$ to $\phi$ and swapping $a$ and $b$ also leaves it invariant: It adds $2\pi$ to $4\phi$ and $\pi$ to $2\phi$, and the resulting sign change in the second and third coordinates is cancelled by the sign change in $\epsilon$.
$a$, $b$ and $\phi$ can be reconstructed from $(s,x,y,u,v)$ as follows: $\epsilon=\sqrt{x^2+y^2}$, $a=\mathrm e^{s+\epsilon}$, $b=\mathrm e^{s-\epsilon}$. If $\epsilon\ne0$, $\phi$ is defined up to multiples of $\pi$ and given by $\frac12\operatorname{atan2}(y,x)$, whereas if $\epsilon=0$, $\phi$ is defined up to multiples of $\frac\pi2$ and given by $\frac14\operatorname{atan2}(v,u)$, where $\operatorname{atan2}$ is the two-argument arctangent.
The last two coordinates seem a bit wasteful, but I’m not sure whether this can be avoided, since, as you noted, we need to prevent the representation from shrinking to a point at $\epsilon=0$ and instead identify $\phi$ and $\phi+\frac\pi2$ when $\epsilon=0$, and I’m not sure whether this is topologically possible with a single extra coordinate.
If we go back to your original formulation where you essentially allowed elements of $U(1)$ as parameters, the two pairs of real coordinates can be combined into complex numbers, so we have a real number $s$, a complex number $\epsilon\mathrm e^{2\mathrm i\phi}$ and a phase $\mathrm e^{4\mathrm i\phi}$.