This question arises when looking at ways to describe isomorphism in certain categories.
In groups, rings, vector spaces: two objects $A,B$ are isomorphic if there exists a bijective morphism from $A$ to $B$.
In graphs, topological spaces, measure spaces: two objects $A,B$ are isomorphic if there exists a bijective morphism from $A$ to $B$, whose inverse is also a morphism.
Now, the second definition is a bit more general, it is easy to verify for the examples I have mentioned that the second definition reduces to the first because of the homomorphism structure.
My question is, is there a way to classify all such categories in which this reduction is possible? Or are homomorphisms simply that special when it comes to imbuing structure?
Edit: Let us restrict the discussion to categories where the objects are built upon sets and the morphisms are special set maps on these underlying sets.
As was already pointed out, the question doesn't make sense for arbitrary categories, because in general the notion of bijection doesn't make sense. A non-example would be the free walking arrow category $\bullet \rightarrow \bullet$ consisting of two objects and one nontrivial morphism between them. It has nothing to do with sets and functions.
What we can do is to restrict the question to categories $\mathcal{C}$, which are equipped with a functor $\mathcal{U:C}\longrightarrow\mathsf{Set}$ (e.g. many categories of sets with structure have such a forgetful functor). We can then say that a morphism $f:X\rightarrow Y$ in $\mathcal{C}$ is $\mathcal{U}$-bijective, if $\mathcal{U}(f)$ is a bijective function. Since in the category $\mathsf{Set}$ of sets being bijective and being an isomorphism is the same, $f$ is $\mathcal{U}$-bijective if and only if $\mathcal{U}(f)$ is an isomorphism. The condition that in $\mathcal{C}$ the notions of $\mathcal{U}$-bijective and isomorphism coincide is then precisely the requirement that $\mathcal{U}$ reflects isomorphisms i.e. that $\mathcal{U}$ is a conservative functor.
So far we have just rephrased the question, but I guess you would also like to know some conditions for when a functor $\mathcal{F:C}\longrightarrow \mathcal{D}$ is actually conservative. Two instances are:
I am unaware of a complete characterization of conservative functors and don't really think it is feasible, based on how different the sufficient conditions 1. and 2. are.
Fun fact: It is well known that the forgetful functor $\mathsf{Top}\longrightarrow\mathsf{Set}$ is not conservative (in other words: that a continuous bijection need not be a homeomorphism). However you can prove a variation of the compact-Hausdorff-trick by purely categorical means: The forgetful functor from the category $\mathsf{CHaus}$ of compact Hausdorff spaces and continuous maps to the category of sets is monadic. By 2. it is conservative, in other words a continuous bijection between compact Hausdorff spaces is a homeomorphism!
Remark 1: Why call it $\mathcal{U}$-bijective instead of just bijective? Well, it really depends on the functor. For example take the category $\mathsf{Set}\times\mathsf{Set}$ of pairs of sets. $\mathcal{U}$ could forget the second set, forget the first set or take a pair $(X,Y)$ to, say, their product set $X\times Y$. Now whether $\mathcal{U}$-bijections are isomorphism or not obviously depends on what $\mathcal{U}$ you choose...
Remark 2: One could also have a completely different take on this question and ask, what conditions you could put on your category $\mathcal{C}$ to even be able to formulate a definition of bijective morphism (e.g. as injective + surjective morphism, whatever this means now) inside that category. Then one could ask, under which conditions this notion coincides with isomorphism. Topos theory might be a good starting point to look for an answer to that question.