I have closed this formula:
$$[\forall x\forall y(P(x,y) \rightarrow P(h(x), h(h(y))))] \rightarrow \forall x(P(x, h(x)) \land P(h(h(x)), x))$$
How could I show that this formula is non-valid but satisfiable? Could I use the Method of analytic tableux here to prove satisfiability? Any suggestions would be helpful.
To show satisfiability you need to come up with an interpretation of the $P$ predicate and the $h$ function that will make the statement true.
And to show it is not valid, you need to come up with an interpretation that will make it false.
And yes, you can use a tableaux for this: if you leave the statement as is, and get to an open and finished branch, then that shows the statements is satisfiable. And if you create a tableaux for the negation of the statement and get an open and finished branch, then that shows it is not valid.
Unfortunately, it is possible that you get an infinite tableaux, in which case you do not get an open and finished branch. So, I would suggest just trying some things, and coming up with a concrete interpretation. Taking the domain of natural numbers is always a good start, so that for $P$ you can try things like > or < and for the $h$ function you can try successor or square.