Given a fixed non-zero constant $k\in\mathbb{R}$, find all functions $f:\mathbb{R}\to\mathbb{R}$ satisfying $$f\big(f(x)\big)=kx\quad\text{and}\quad f\left(x^2\right)=xf(x).$$
If $f$ is continuous, then we can show that the solutions to the second equation are of the form $f(x)=mx$ (see here for example). With the first equation, this immediately implies that $f(x)=\sqrt{k}x$ or $f(x)=-\sqrt{k}x$. However, I am struggling to extend this to non-continuous $f$. Any help would be appreciated.
Let's suppose for a non-zero constant $k$ and every real number $x$, we have: $$f\big(f(x)\big)=kx\tag0\label0$$ $$f\big(x^2\big)=xf(x)\tag1\label1$$ By \eqref{0} you can find out that $f$ is injective. Now, by \eqref{0} and \eqref{1} you have: $$f\left(f(x)^2\right)=f(x)f\big(f(x)\big)=kxf(x)=kf\left(x^2\right)=f\bigg(f\Big(f\left(x^2\right)\Big)\bigg)=f\left(kx^2\right)$$ $$\therefore\quad f(x)^2=kx^2\tag2\label2$$ Therefore by \eqref{2} we have $k=f(1)^2$ and hence $k$ is positive. Again by \eqref{2}, for every real number $x$ we get $f(x)=\pm\sqrt kx$.
Now let's define $K^\pm:=\big\{x\ne0\big|f(x)=\pm\sqrt kx\big\}$. So we get $\mathbb R=\{0\}\cup K^+\cup K^-$. By \eqref{1} we have $f(0)=0$. You can reformulate \eqref{0} and \eqref{1} this way: $$x\in K^\pm\quad\text{iff}\quad\sqrt kx\in K^\pm$$ $$x\in K^\pm\quad\text{iff}\quad x^2\in K^\pm$$ It's easy to see that every function satisfying $f(0)=0$ and $f(x)=\pm\sqrt kx$ for $x\in K^\pm$ is a solution, where $K^\pm$ satisfy the above conditions.
The trivial cases happen when one of $K^\pm$ is empty, and give us the linear solutions $f(x)=\sqrt kx$ and $f(x)=-\sqrt kx$. For a nontrivial case, you can take $K^+=\{\pm k^q\vert q\in\mathbb Q\}$ and $K^-=\{\pm k^q\vert q\in\mathbb R\backslash\mathbb Q\}$.
EDIT:
As noted by @DanielWainfleet in the comments, the way my example above is presented, it only works for $k\ne1$. In fact, I could simply write $K^-=\mathbb R\setminus(K^+\cup\{0\})$, and then it would work for all $k>0$. But @DanielWainfleet has also provided an interesting example that works for $k=1$: $K^+=\left\{\pm 2^{2^n}\big\vert n\in\mathbb Z\right\}$.