The numbers $2,3$ and $6$ are initially written on a black board and you are allowed to replace two of them say $x$ and $y$ with the two numbers $\frac{3}{5}x + \frac{4}{5}y$ and $\frac{4}{5}x - \frac{3}{5}y$. Prove that it is impossible for a number greater than $7$ to appear on the blackboard.
I took a linear algebraic approach and noted that this operation is equivalent to the linear transformation $$\begin{pmatrix}\frac{3}{5}&\frac{4}{5}\\ \frac{4}{5}&-\frac{3}{5}\end{pmatrix}$$ acting on $\begin{pmatrix}x\\ y\end{pmatrix}$.
Now this is pretty much a rotation and a scaling and so for a number greater than $7$ to appear on the board one of the components of the scaled vectors should be $7$, but if we are starting with numbers less than $7$ this is impossible as we are only getting smaller values?
Is this correct? I guess we also cannot get $7$ ever either.
I like your linear algebraic approach, but when you write that this is ``pretty much a rotation and a scaling'', what are you scaling by?
In fact, this is just a rotation.
If we think of $\begin{pmatrix} x \\ y \end{pmatrix}$ as a point in $\mathbb{R}^2$, then it is a distance of $\sqrt{x^2+y^2}$ away from the origin. On the other hand, the point $\begin{pmatrix} \frac{3}{5}x+\frac{4}{5}y \\ \frac{4}{5}x-\frac{3}{5}y \end{pmatrix}$ is a distance of
\begin{align} \sqrt{\left(\frac{3}{5}x+\frac{4}{5}y\right)^2 + \left(\frac{4}{5}x-\frac{3}{5}y)^2\right)^2} &= \sqrt{\frac{9}{25}x^2 + \frac{12}{25}xy + \frac{16}{25}y^2 + \frac{16}{25}x^2 - \frac{12}{25}xy + \frac{9}{25}y^2 } \\ &= \sqrt{ \frac{9+16}{25}x^2 + \frac{9+16}{25}y^2 } \\ &= \sqrt{ x^2 + y^2 }. \end{align}
Any point $(x,y)$ such that either $|x|$ or $|y|$ is greater than $7$ will necessarily be a distance of at least 7 away from the origin; for example, if $x\geq 7$, then $\sqrt{x^2+y^2} \geq \sqrt{49+y^2} \geq \sqrt{49} = 7$.
However, using $2,3$ and $6$, the furthest point from the origin that we can create is $(3,6)$, which is only $\sqrt{45}$ away from the origin!