If we speak simply about isomorphism, we say that it is a bijective homomorphism. But I have read recently another definition, where they say, it is more general if we say, that it is $\psi$, a bijective homomorphism where $\psi^{-1}$ is a bijective homomorphism too.
They say the reason for this is, that it is not by all algebraic structures the case, that by a bijective homomorphism, the inverse function is necessarily a bijective homomorphism.
Could you name me an example, where it is not the case? Thanks!
The generalization is a lot better suited to category theory. There we define that a homomorphism $\phi\colon A\to B$ is an isomorphism if there exists a homomorphism $\psi\colon B\to A$ such that $\psi\circ \phi=\operatorname{id}_A$ and $\phi\circ \psi=\operatorname{id}_B$.
First of all, to investigate bijectivity, $\phi$ needs to be a map between sets in the first place. In general categories, not all objects are sets with additional structure, and homomorphisms maps respecting the structure. The standard example is the category of homotopy classes of topological spaces.
Secondly, in certain categories for which objects are sets with structure it is still not true that a homomorphism that is bijective as a set-map is an isomorphism. The standard example is the category of topological spaces.