I'm reading Flegg's From Geometry to Topology, the author says that in Euclidean geometry, translation and rotation are:
$$T:(x,y)\to(x+a,y+b)$$ $$R:(x,y)\to(x \cos \phi - y \sin \phi, x \sin \phi +y \cos \phi)$$
And that reflection can be achieved through arbitrary combinations of reflection and rotation, although he assumes there is a formula for such, he argues that it's easier this way.
Then he argues that in these transformations, some properties are invariant, such as Length, area, the number of sides of a polygon, and so on. And that one of the properties that is not preserved is the orientation.
Then there's a chapter on non-euclidean geometries, and then he gives one example of affine transformation:
$$A:(x,y)\to(ax+by+c,dx+ey+f)$$
And says that some properties such as length, "angle" and shape are not preserved under this transformation but some other properties are. Further, there's also an example on projective geometry where he does the same thing, arguing about what's invariant and what's not.
Then he introduces topological transformations and says that the neighbourhoods are preserved and introduces an example of invariant objects under these transformations:
There are also examples for the transformations I mentioned before, I just decided to omit them in this question. He gives a formula for $T,R,A$, but there are no formulas for the transformation of the triangle in the previous image to an elipse, or a formula that transform any of these objects into another. I've read about it wikipedia, in the article about topology and it's mentioned only that topological transformations are called continuous maps, when opening the article on continuous maps, there's also no mention to a formula for such (or I'm not seeing it). So what's the formula for these transformations and how to obtain them?
The only hint I have at the moment (and I'm not really sure about that, it may be completely wrong!) is that it seems that these tranformations were completely made in set theory, which confuses me a little more.
In general, you there need not be a closed formula to describe this kind of map. And even if there is, the formula often is more complex than what you'd want to write down. Furthermore, the map isn't uniquely defined by the shape alone, but instead there will be many possible maps with the required properties.
Let's take this “triangle and ellipse” as an example. Let's assume them to be the sets
\begin{align*} A&=\{(x,y)^T\;\vert\;x\le1\text{ and }-y\le1\text{ and }2y-x\le1\}\\ B&=\{(x,y)^T\;\vert\;x^2+4y^2\le4\} \end{align*}
as depicted below.
Then the following function will be one possibility of how to continuously map from $A$ to $B$:
$$f:\begin{pmatrix}x\\y\end{pmatrix}\mapsto \frac{\max\{x, -y, 2y-x\}}{\sqrt{x^2+y^2}} \cdot\begin{pmatrix}2x\\y\end{pmatrix}$$
To see that this is really a bijection, here is the inverse:
$$f^{-1}:\begin{pmatrix}x\\y\end{pmatrix}\mapsto \frac{\sqrt{x^2+4y^2}}{\max\{2x, -4y, 8y-2x\}} \cdot\begin{pmatrix}x\\2y\end{pmatrix}$$
Both $f$ and $f^{-1}$ are strictly speaking not defined at $x=y=0$. But the singularity there is removable, so you can simply define $f(0,0)^T=f^{-1}(0,0)^T=(0,0)^T$.
The idea behind the formulas is simply the following: consider rays from the origin in every direction. These rays will intersect the triangle at some point, and the unit circle at some different point. Simply apply a linear scaling to all points along the ray with the scale factor chosen such that it will map the point of intersection with the triangle to the point of intersection with the unit circle. Afterwards, the unit circle is stretched by a factor of $2$ in the $x$ direction to obtain the ellipse. This idea of stretching every ray is the reason why Will Jagy in a comment wrote about star-shaped shapes: those are the ones for which this “scale every ray” approach works.
But keep in mind that this is only one of many possible maps, and even for shapes which are not star-shaped, you can still find continuous deformations between them as long as both shapes are of the same topological structure. Writing these deformations down as a formula will usually be more difficult than useful.