I know the following rule: $\lnot \forall a (P(a)$ means $\exists a \lnot P(a)$ and $\lnot \exists a (P(a)$ means $\forall a \lnot P(a)$
But what if the problem is something like: $\lnot \forall x \lnot \exists y (P(x) \land P(y))$? Will the negation cancel out? To something like: $ \exists x \forall y (P(x) \land P(y))$?
And furthermore, what happens with functions $f$ then? If there is given: $\lnot \forall a \lnot \exists b(P(a) \land P(f(b)) ) \implies Q(f(f(b))))$ and I want to find a Model which satisfies the formula, how can I?
They don't cancel out in the way you write it, no. Don't forget, the rule in the form you cite is equivalent to: $$ \neg\, \forall x \,\neg\, \varphi(x) \iff \exists x\, \varphi(x) \\ \,\neg \,\exists x \,\neg\, \varphi(x) \iff \forall x\, \varphi(x) \\ $$ Even more concisely for all four equivalences, $$\begin{align} \neg \exists &= \forall \neg \\ \neg \forall &= \exists \neg \\ \neg \forall \neg &= \exists \\ \neg \exists \neg &= \forall . \end{align}$$
With your example, $\lnot \forall x\, \lnot \exists y\, (P(x) \land P(y))$ is equivalent to $\exists x\, \exists y \,(P(x) \land P(y))$.
If you have function symbols as well, the same quantifier rules still apply to variables, of course. Your second example sentence, plus a missing initial parenthesis, is $\lnot \forall a\, \lnot \exists b \left((P(a) \land P(f(b)) ) \to Q(f(f(b)))\right)$, which is equivalent to $\exists a\, \exists b\,((P(a) \land P(f(b)) ) \to Q(f(f(b))))$.
The question about how you then find a model is a very different question.