$$f(x)= \begin{cases} x^2 + k & \text{if } x \leq 5 \\ k x\ &\text{if } x > 5 \end{cases} $$
I've seen where you set $x=5$ and plug in, but I don't know how that makes sense. Ive tried $x^2 + k = k x$ and $5^2 + k = k(5)$ so $25 + k = 5k$ but I don't know where to go from here.
You can't make it a constant function everywhere as this is a piecewise continous function, where part of it is strongly convex quadratic and another is linear. In particular, $f(1) \neq f(2)$.
From your current working, it seems that you are trying to make it a continuous function, then $$\lim_{x \to 5^-} f(x)=f(5)=\lim_{x \to 5^+} f(x)$$
and what you did is correct. that is you will get $$25+k=5k$$
then $$4k=25$$ and you solve for $k$ by dividing by $4$.