Let f be a real valued function $f:(0,\infty)\to(0,\infty)$
such that it satisfies the relation:
$$f(xf(y))=x^2\cdot y^a$$
where $a\in\mathfrak{R}$
then find $f(x)$ and the possible values of $a$.
My attempt:
Looking at the given relation, I can assume the function to be $f(x)=x^2$ on a hit-and-trial basis. Putting $f(y)=y^2$ in the given relation,
$f(xy^2)=x^2y^a$
or $(xy^2)^2=x^2y^a$
or $a=4$
Hence, $f(x)=x^2$ and $a=4$
Now, my question is: Is there an alternate method of solving such kinds of problems apart from the hit-and-trial method?
Problems of this sort come up quite frequently in mathematics contests (IMO, national olympiads, etc.). There are a bunch of techniques but so far as I know not much unifying theory. See e.g. here and here.
To solve a functional equation you need to find a solution or solutions, verify that they satisfy the equation, and prove that nothing else does. Usually the last of these is much the most difficult part, but even a conjectural answer can be very helpful in suggesting things to try.
Usually a large fraction of the work amounts to plugging in carefully chosen special values and seeing what facts come out. (The values being carefully chosen to make the resulting expressions simple and/or symmetrical in useful ways.) For instance, in the present case you learn a lot by writing $x=t/f(1)$ because then the LHS is just $f(t)$ and the $y^a$ on the RHS disappears.
In some cases you can do this "conditionally", saying something like "If there is any $y$ such that $f(y)=x$, then ...", and then deal with the case where the conditional doesn't hold in some other way (e.g., then you've found an $x$ that isn't the image of any $y$, which may be revealing).
When you have a good guess at what the solution(s) may be, it sometimes clarifies matters to consider a related function that is really simple when that guess holds. So, e.g., if you think the given equation holds when $f(x) = kx$ for some fixed $k$, you might consider $g(x)=f(x)/x$ or $g(x)=f'(x)$ or $g(x)=f(x+1)-f(x)$ and see whether the corresponding condition on $g$ is easier to work with than the original condition on $f$.
You may be able to transform your function in some other way to simplify the equation.
In some cases (especially if you know that your function is not too badly behaved) it may be helpful first to prove that some useful property holds for, say, integer arguments of the function (perhaps by induction -- maybe you can find $f(n)$ in terms of $f(0),f(1),f(n-1)$ or something), and then for rational arguments -- maybe you can find $f(m/n)$ in terms of $f(m),f(n)$, and then use continuity or something of the kind to establish it for all arguments. (A standard example is the "Cauchy equation" $f(x+y)=f(x)+f(y)$. First put $x=y=0$ to establish $f(0)=0$; then, by induction, for positive integer $k$ we have $f(kx)=kf(x)$, so $f(n)=nf(1)=an$, say; then the same holds for rational $n$ by letting $k$ equal the denominator; then use continuity if you have it. There actually are lots of "pathological" solutions to the Cauchy equation, but they are very pathological; e.g., I think the graph of the function has to be everywhere dense in $\Bbb{R}^2$ if it isn't $f(x)=ax$.)
Sometimes the statement of the problem doesn't explicitly include continuity or monotonicity or whatever, but you can infer it. (This usually depends on there being some sort of inequality in the problem statement -- e.g., values of the function always being positive.)
If your guess at the solution implies that the function must be injective or surjective or something of the kind, it is often not too hard to prove. (E.g., if you think the function is injective, suppose $f(x)=f(y)$ and see if you can find things to plug into the given equation that exploit that; or maybe you can get some relation like $f(xf(y))=x^2+y$ and note that the RHS takes every possible value.)
It is often useful to look for fixed points where $f(x)=x$ or, sometimes, variations on this theme -- $f(f(x))=x$, $f(x)=x+1$, etc.
If the foregoing looks like a grab-bag of random and/or obvious tricks, that's because it's more or less what it is. As I said, not much unifying theory here. The best way to get good at solving this sort of problem is to solve lots of them and accustom yourself to what works most often. There are some worked examples, and other examples to challenge yourself with, at the other end of the links above.
[EDITED to say more about techniques rather than just pointing to external resources.]