Problem: Let $\gamma: [a,b] \to \mathbb{R}^2$ be a regular smooth curve and $A$ be a $2 \times 2$ orthogonal matrix. Let $\gamma_A: [a, b] \to \mathbb{R}^2$ be a regular smooth curve defined by $\gamma_A(t) := A(\gamma(t))$. Find the relation between the curvature $\kappa_{\gamma}(t)$ and $\kappa_{\gamma_A}(t)$ for $t \in [a, b]$.
My approach: Consider $\gamma_t = (x(t), y(t))$, then $\gamma_A (t) = (Ax(t), Ay(t))$. After applying the standard formula for curvature for plane curves for both $\gamma_t$ and $\gamma_A (t)$, we get: $$ k_{\gamma}(t) = \frac{x'y'' - x''y'}{(x'^2 + y'^2)^{3/2}} $$ and $$ k_{\gamma_A}(t) = \frac{Ax'Ay'' - Ax''Ay'}{(A^2x'^2 + A^2y'^2)^{3/2}} = A^{-1}\left(\frac{x'y'' - x''y'}{(x'^2 + y'^2)^{3/2}}\right). $$ Since $A$ is an orthogonal matrix, its inverse exists. Therefore, $k_{\gamma}(t) = A^{-1}\cdot k_{\gamma_A}(t)$.
Is this correct?
The first thing you are doing wrong is writing
$\gamma_A (t) = (Ax(t), Ay(t))$
$Ax(t)$ and $Ay(t)$ are matrices, right? So $(Ax(t), Ay(t))$ is actually a tuple of matrices not a tuple of points, so how does it represent a point in $\mathbb R^2$? If you know matrix multiplication then you would know that multiplying $m×n$ matrix to $n×k$ matrix gives you $m×k$ matrix. In your case you have to multiply $2×2$ matrix $A$ to a $2×1$ matrix which represents a point in $\mathbb R^2$ to get another $2×1$ matrix which is again a point in $\mathbb R^2$. So $A$ is transforming points in $\mathbb R^2$ to points in $\mathbb R^2$. Specifically, the point
$\gamma(t) = \begin{bmatrix} x(t)\\ y(t) \end{bmatrix}$
gets mapped to the point
$\gamma_A(t) = A\begin{bmatrix} x(t)\\ y(t) \end{bmatrix}$
In particular if $A = \begin{bmatrix}a & b\\ c & d\end{bmatrix} $ then
$\gamma_A(t) = \begin{bmatrix}a & b\\ c & d\end{bmatrix}\begin{bmatrix} x(t)\\ y(t) \end{bmatrix} = \begin{bmatrix} ax(t) + by(t) \\ cx(t) + dy(t)\end{bmatrix}$
So the point $(x(t), y(t))$ gets mapped to the point $(ax(t) + by(t), cx(t) + dy(t))$ and in effect the curve $\gamma(t)$ gets mapped to the curve $\gamma_A(t)$. Let's write
$\gamma_A(t) = A\begin{bmatrix} x(t)\\ y(t) \end{bmatrix} = \begin{bmatrix} X(t)\\ Y(t) \end{bmatrix} $
Hence the curvature of our new curve is
$\dfrac{X'Y'' - X''Y'}{(X'^2 + Y'^2)^{3/2}}$
To calculate this note that
(1) $ X'Y'' - X''Y' = \det \begin{bmatrix} X' & X''\\ Y' & Y'' \end{bmatrix} $
(2) $\begin{bmatrix} X' \\ Y'\end{bmatrix} = A\begin{bmatrix} x'\\ y' \end{bmatrix}$
(3) $\begin{bmatrix} X'' \\ Y''\end{bmatrix} = A\begin{bmatrix} x''\\ y'' \end{bmatrix}$
(4) Combining (2) and (3) you get
$\begin{bmatrix} X' & X'' \\ Y' & Y''\end{bmatrix} = A\begin{bmatrix} x' & x'' \\ y' & y'' \end{bmatrix}$
Thus $ \det \begin{bmatrix} X' & X''\\ Y' & Y'' \end{bmatrix} = \det \Bigg(A\begin{bmatrix} x' & x'' \\ y' & y'' \end{bmatrix}\Bigg) = \det A \cdot \det \begin{bmatrix} x' & x'' \\ y' & y'' \end{bmatrix} $
(5) $X'^2 + Y'^2 = \begin{bmatrix} X' & Y' \end{bmatrix}\begin{bmatrix} X' \\ Y' \end{bmatrix} = \begin{bmatrix} x' & y' \end{bmatrix}A'A\begin{bmatrix} x' \\ y' \end{bmatrix} $
Now just use properties of orthogonal matrices. First of all for orthogonal matrices we have $A^{-1} = A'$ so that $A'A = I$ in (5) which gives $X'^2 + Y'^2 = x'^2 + y'^2$. This along with (4) will give you $k_{\gamma_A}(t) = \det A \cdot k_{\gamma}(t)$. Further, for orthogonal matrices $\det A = ±1$ so the curvature either remains same or changes only upto a sign depending on whether $\det A = 1$ or $\det A = -1$.
Most of these of calculations wouldn'td be needed if you know how orthogonal transformations work. Basically they don't change the distance between points. So if $(x, y)$ goes to $(X, Y)$ the distance $|X - Y|$ still equals $|x - y|$. So the curves $\gamma(t)$ and $\gamma_A(t)$ will look exactly the same. The only possible difference will be their orientation. For example if you take mirror image of all points across $X$ axis then the curve will get mapped to its mirror image which will look same as original curve except for the orientation. The curvature of new curve will be negative of the original curve's curvature. Also you can see that mapping to mirror images across any line through origin doesn't change the distance between two points. So this transformation is orthogonal. Yes, not only orthogonal transformations preserve distance but any distance preserving linear transformation is orthogonal. I hope that explains. If you have doubt about how the formulas in (1) - (5) appeared then simply do the calculations and you will see they are true. If you want to know more about orthogonal matrices/transformations then you will find many things right here on mathstackexchange.