If $X$ and $Y$ are reasonably nice spaces (CW-complexes, simplicial complexes, manifolds...), is there a relation between the (co)homology of the space of the continuous maps $\mathcal{F}(X, Y)$ (with the compact-open topology) and the (co)homology of the two spaces $X$ and $Y$?
If there is no known/immediate relation for two generic (nice) spaces, is it possible to say something more if one of the two is a suitable "test" space (for example if $X\simeq S^n$ or $Y$ is an Eilenberg-MacLane space)?
Is there a good book/paper about the homology (or the cohomology ring) of map spaces?
Why this questions?
In this paper, the author claimed that the standard torus $T^2$ and $S^2\vee S^1 \vee S^1$ have the same homology groups but the two spaces $\mathcal{F}(S^1, T^2)$ and $\mathcal{F}(S^1, S^2\vee S^1 \vee S^1)$ have different homology groups. While the first part of the statement is quite straightforward to prove, I am not able to compute the homology groups of the two map spaces on my own and I could not find any reference on the topic by googling or looking it up here.
In general the topology of $Map(X,Y)$ will be very complicated, even for reasonable spaces $X$ and $Y$. This answer cannot possibly claim to be complete but hopefully will be useful to you. Apologies for its unwieldy length. We'll assume throughout that $X$ and $Y$ are pointed CW complexes, and use the more standard notation $Map(X,Y)$ for the space of unbased maps and $Map_*(X,Y)$ for the space of based maps. By a result of Milnor $Map(X,Y)$ and $Map_*(X,Y)$ are CW complexes.
Now it is true that if $A$ is an abelian group and $Y=K(A,n)$ is the Eilenberg-Mac Lane space in degree $n$, then $Map(X,K(A,n))$ is homotopy equivalent to a product of Eilenberg-Mac Lane spaces. This is true when $X$ is a finite complex, or more generally of finite type. It may still be true for more general $X$, although I would suggest care taken with this. I'll prove the statement for $X$ a simply connected finite complex. For more details look at Moller's paper Spaces of Sections of Eilenberg-Mac Lane Fibrations and the references therein.
To see that the statement should be true first consider the based mapping space $Map_*(X,K(A,n))$. Then $\pi_0Map_*((X,K(A,n)))=[X,K(A,n)]\cong \tilde H^n(X;A)$, and more generally
$$\pi_k(Map_*(X,K(A,n))=\pi_0(\Omega^kMap_*(X,K(A,n)))=\pi_0(Map_*(S^k,Map_*(X,K(A,n))))\cong\pi_0(Map_*(S^k\wedge X,K(A,n)))\cong\tilde H^n(\Sigma^kX;A)\cong\tilde H^{n-k}(X;A).$$
Thus $Map_*(X,K(A,n))$ has the same homotopy groups as the product $\prod K(\tilde H^{n-k}(X;A),k)$. There are several ways of increasing sophistication to realise the homotopy equivalence. We'll follow what is perhaps the simplest and assume that $X$ is a finite, simply connected CW complex, using induction on its cellular structure.
Now when $X=S^{k-1}$ it is clear that
$$Map_*(S^{k-1},K(A,n))=\Omega^{k-1}K(A,n)\simeq K(A,n-k+1)\simeq K(\tilde H^{k-1}(S^{k-1};A),n-k-1)$$
has the correct homotopy type, and more generally that so does
$$Map_*\left(\bigvee S^{k-1},K(A,n)\right)\simeq K\left(\tilde H^{k-1}\left(\bigvee S^{k-1};A\right),n-k-1\right)$$
when $X$ is a wedge of $(k-1)$-spheres.
Now assume the statement is true for all $X'$ of cellular dimension $<k$ and that $X$ is obtained from $X'$ by adding $k$-cells. Then there is a cofibration sequence
$$\bigvee S^{k-1}\xrightarrow{\varphi}X'\xrightarrow{i}X\xrightarrow{q}\bigvee S^k\rightarrow\dots$$
which induces a fibration sequence
$$\dots\rightarrow Map(X,K(A,n))\xrightarrow{i^*} Map(X',K(A,n))\xrightarrow{\varphi^*}Map(\bigvee S^{k-1},K(A,n)).$$
By assumption $Map(X',K(A,n))\simeq\prod K(\tilde H^{n-i}(X';A),i)$ and $Map(\bigvee S^{k-1},K(A,n))\simeq \prod K(A,n-k+1)$. For dimensional reasons $\varphi^*$ is trivial when restricted to all factors except $K(\tilde H^{k-1}(X';A),n-k+1)$, and on this factor it is the map corresponding to the change of coefficients induced by the homomorphism $\varphi^*$ in the long exact cohomology sequence
$$0\rightarrow \tilde H^{k-1}(X;A)\xrightarrow{i^*}\tilde H^{k-1}(X';A)\xrightarrow{\varphi^*} \tilde H^{k-1}\left(\bigvee S^{k-1};A\right)\rightarrow \tilde H^k(X;A)\rightarrow 0.$$
Thus the fibre of $\varphi^*$ resticted to $K(\tilde H^{k-1}(X';A),n-k+1)$ identifies with $K(\tilde H^{k-1}(X;A);k-1)$ and it follows that the fibre of $\varphi^*$ is homotopy equivalent to $K(\tilde H^{k-1}(X;A);k-1)\times\prod_{i<k-1} K(\tilde H^{n-i}(X';A),i)$. Using the fibration sequence above, this fibre is precisely $Map_*(X,K(A,n))$ .
Now since $\tilde H^{k-i}(X;A)\cong\tilde H^{k-i}(X';A)$ for $i<1$, we have from all this that
$$Map_*(X,K(A,n))\simeq \prod K(\tilde H^{n-i}(X;A),i)$$
and the proof is complete upon appealing to the inductive hypothesis.
Now consider the unbased mapping space $Map(X,K(A,n))$ and the evaluation fibration
$$Map_*(X,K(A,n))\xrightarrow{j} Map(X,K(A,n))\xrightarrow{ev} K(A,n).$$
Since $K(A,n)$ is an H-space so too is the mapping space $Map(X,K(A,n))$. The map $\theta:K(A,n)\rightarrow Map(X,K(A,n))$ defined by $\theta(a)(x)=a$ for $a\in K(A,n)$, $x\in X$, splits $ev$, and using the H-space multiplication we get a map
$$j+\theta:Map_*(X,K(A,n))\times K(A,n)\rightarrow Map(X,K(A,n))$$
which is seen to be a weak homotopy equivalence. Since $X$ is CW, so too are the mapping spaces and we conclude that this map is a homotopy equivalence. We can sum up the result with the following,
$$Map(X,K(A,n))\simeq \prod_{k\geq 0}K(\tilde H^{n-k}(X;A),K).$$
Now let us turn to your final question. Using the above we have
$$Map(S^1,S^1\times S^1)\cong Map(S^1,S^1)\times Map(S^1,S^1)\simeq S^1\times S^1\times \mathbb{Z}\times \mathbb{Z}.$$
On the other hand, using the fact that the composite $S^2\hookrightarrow S^1\vee S^1\vee S^2\xrightarrow{pinch} S^2$ is the identity we see that $Map(S^1,S^2)$ retracts off of $Map(S^1,S^1\vee S^1\vee S^2)$. In particular $H^*(Map(S^1,S^2))$ is a direct summand of $H^*(Map(S^1,S^1\vee S^1\vee S^2))$.
Now consider the rationalisation of this last function space. Since $S^1$ is a finite complex and $S^2$ is simply connected it holds that
$$Map(S^1,S^2)_\mathbb{Q}\simeq Map(S^1,S^2_\mathbb{Q})$$
Now there is a homotopy fibration sequence
$$S^2_\mathbb{Q}\rightarrow K(\mathbb{Q},2)\xrightarrow{\iota^2} K(\mathbb{Q},4)$$
to which we can apply the mapping functor and use the previous results to get a homotopy fibration
$$Map(S^1,S^2_\mathbb{Q})\rightarrow K(\mathbb{Q},1)\times K(\mathbb{Q},2)\xrightarrow{\ast\times \iota^2} K(\mathbb{Q},3)\times K(\mathbb{Q},4).$$
This shows that $Map(S^1,S^2_\mathbb{Q})\simeq K(\mathbb{Q},1)\times K(\mathbb{Q},2)\times S^2_\mathbb{Q}$.
The point is that this tells us that $H^*(Map(S^1,S^2))$ contains free $\mathbb{Z}$ factors in all even degrees, corresponding rationally to the homology of $K(\mathbb{Q},2)$. As we have already observed, this means that $Map(S^1,S^1\vee S^1,S^2)$ has integral summands in all even degrees. This is clearly not the case for $Map(S^1,S^1\times S^1)\simeq S^1\times S^1\times\mathbb{Z}\times\mathbb{Z}.$