In my lecture notes I have this exercise:
Let $G$ and $H$ be cyclic groups. When is $Hom(G,H)<H^G$?
In the solution, before analyzing if $Hom(G,H)$ is a subgroup they define the operation:
$\varphi \psi: G \rightarrow H: x \mapsto \varphi(x) \psi(x)$
and argue that with such an operation $H^G$ is a group . For this part they don't use the fact that the groups are cyclic at all.
Then they find that since H is cyclic, we can use the fact that it is abelian in order to have a subgroup, but argue that there is a weaker condition for that being cyclic, which I guess is just to be abelian.
My question is how could I have known that is the operation I need to define in order to have a group over $H^G$ and that there is no other choice? If there are other choices, how do I know that for any other choice of an operation, $Hom(G,H)$ is still a subgroup under the same condition(to be abelian for the most general one if I was right) found when using the suggested operation?
First: I do not understand what “we can use the fact that it is abelian in order to have a subgroup” means. Do you mean, to prove that $\mathrm{Hom}(G,H)$ is a subgroup, or to just talk about subgroups? If the latter, that makes no sense to me. As soon as you have a group, you can have subgroups.
That said...
You may want to review any section in which they provide examples of groups, because the following is a common construction: if you have a group $G$, and a set $X$, then the set $$G^X = \{f\colon X\to G\mid f\text{ is a function}\}$$ is a group under pointwise product, $(fg)(x) = f(x)g(x)$. This holds for any group $G$ and any set $X$ (you don’t even need $G$ to be abelian, let alone cyclic). It is, in fact, a special case of the direct product with coordinate-wise multiplication, as it corresponds to the direct product of $|X|$ copies of $G$, using $X$ as the index set. It is possible that your source or your lecturer already established this as the standard interpretation of the group $G^X$ whenever $G$ is a group and $X$ is a set. This includes the instance of $H^G$, by looking at $H$ as a group and $G$ as a set. So you should review your notes to see if this has already been discussed earlier.
While there are “other choices”, none are particularly natural. This is a standard construction; just like if somebody told you “take the group $\mathbb{Z}$”, you are expected to know we mean the group under the operation of the usual integer addition, even though you can give it all sorts of other group structures by using transport of structure, it is unreasonable (almost perverse) to expect people to understand that “the group $\mathbb{Z}$” might refer to one of these many other group structures.
Now, because $\mathrm{Hom}(G,H)$ is certainly a subset of $H^G$, we can ask whether it is a subgroup of $H^G$. It is here that you would use the fact that $H$ is abelian, because otherwise the pointwise product need not make the resulting function a homomorphis. Specifically, if $f$ and $g$ are homomorphisms, you would need $$(fg)(xy) = f(xy)g(xy) = f(x)f(y)g(x)g(y)$$ to be equal to $$\Bigl((fg)(x)\Bigr)\Bigl((fg)(y)\Bigr) = f(x)g(x)f(y)g(y).$$ The equality is equivalent to having $f(y)g(x) = g(x)f(y)$ for all $f,g\in\mathrm{Hom}(G,H)$ and all $x,y\in G$.
This will certainly hold if $H$ is abelian; though it may hold in other situations by happenstance (for example, if every homomorphic image of $G$ in $H$ commute with one another. For an extreme example, $\mathrm{Hom}(A_6,A_5)$ is trivial, because the only morphism from $A_6$ to $A_5$ is the trivial homomorphism, so here $\mathrm{Hom}(A_6,A_5)$ is a subgroup of $(A_5)^{A_6}$, even though $A_5$ is not abelian).
As to your final question: since there is no other natural structure to give $H^G$, there is no condition you can give.