Let $ m , n \in ( 0 , + \infty ) $ be constants. Find all functions $ g : ( 1 , + \infty ) \to ( 1 , + \infty ) $ such that $$ g ( x ^ m y ^ n ) \le g ( x ) ^ { \frac { a ^ 2 } m } g ( y ) ^ { \frac { b ^ 2 } n } $$ for all $ x , y \in ( 1 , + \infty ) $ and all $ a , b \in ( 0 , 1 ) $ with $ a + b = 1 $.
I have no clue about this. I tried some things, but it seemed useless. Like making $ x = y $, or give some values… Nothing "happened".
It seems that the problem is too broad for achieving a nice characterization of all the solutions. But nevertheless, one can simplify it and get an equivalent formulation which can be handled more easily (see \eqref{3} below). Further restrictions on $ g $, $ m $ and $ n $ may result in more definitive solutions, but as we'll demonstrate at the end, even relatively powerful restrictions may still result in a huge class of solutions.
There are two major simplifications that one can consider:
Consider a function $ g : ( 1 , + \infty ) \to ( 1 , + \infty ) $ satisfying $$ g ( x ^ m y ^ n ) \le g ( x ) ^ { \frac { a ^ 2 } m } g ( y ) ^ { \frac { b ^ 2 } n } \tag 0 \label 0 $$ for all $ x , y \in ( 1 , + \infty ) $ and all $ a , b \in ( 0 , 1 ) $ with $ a + b = 1 $. As the functions $ \exp : ( 0 , + \infty ) \to ( 1 , + \infty ) $ and $ \log : ( 1 , + \infty ) \to ( 0 , + \infty ) $ are strictly increasing bijections, one can define a function $ h : ( 0 , + \infty ) \to ( 0 , + \infty ) $ with $ h ( x ) = \log \big( g ( \exp x ) \big) $ for all $ x \in ( 0 , + \infty ) $, and then substitute $ \exp x $ for $ x $ and $ \exp y $ for $ y $ in \eqref{0} and take logarithms from both sides to get $$ h ( m x + n y ) \le \frac { a ^ 2 } m h ( x ) + \frac { b ^ 2 } n h ( y ) \tag 1 \label 1 $$ for all $ x , y \in ( 0 , + \infty ) $ and all $ a , b \in ( 0 , 1 ) $ with $ a + b = 1 $. Then, noting that for any $ x , y \in ( 0 , + \infty ) $ and any $ a \in ( 0 , 1 ) $, we have $$ \frac { a ^ 2 } m h ( x ) + \frac { ( 1 - a ) ^ 2 } n h ( y ) = \left( \frac { h ( x ) } m + \frac { h ( y ) } n \right) \left( a - \frac { m h ( y ) } { n h ( x ) + m h ( y ) } \right) ^ 2 + \frac 1 { \frac m { h ( x ) } + \frac n { h ( y ) } } \text , $$ one can see that $ h $ satisfies \eqref{1} for all $ x , y \in ( 0 , + \infty ) $ and all $ a , b \in ( 0 , 1 ) $ with $ a + b = 1 $ iff it satisfies $$ h ( m x + n y ) \le \frac 1 { \frac m { h ( x ) } + \frac n { h ( y ) } } \tag 2 \label 2 $$ for all $ x , y \in ( 0 , + \infty ) $ (which is in fact an instance of \eqref{1} when $ a = \frac { m h ( y ) } { n h ( x ) + m h ( y ) } $ and $ b = \frac { n h ( x ) } { n h ( x ) + m h ( y ) } $). Again, as the function $ ( . \! ) ^ { - 1 } : ( 0 , + \infty ) \to ( 0 , + \infty ) $ is a stricty decreasing bijection, one can define $ f : ( 0 , + \infty ) \to ( 0 , + \infty ) $ with $ f ( x ) = \frac 1 { h ( x ) } $ for all $ x \in ( 0 , + \infty ) $, and then take reciprocals of both sides of \eqref{2} to get $$ f ( m x + n y ) \ge m f ( x ) + n f ( y ) \tag 3 \label 3 $$ for all $ x , y \in ( 0 , + \infty ) $. It's straightforward to take the above steps in reverse and see that any $ f : ( 0 , + \infty ) \to ( 0 , + \infty ) $ satisfying \eqref{3} gives rise to a solution $ g : ( 1 , + \infty ) \to ( 1 , + \infty ) $ of the original problem, which is of the form $ g ( x ) = \exp \left( \frac 1 { f ( \log x ) } \right) $ for all $ x \in ( 1 , + \infty ) $. Thus, we've arrived at an equivalent formulation of the original problem, which looks much simpler. No matter what the values of $ m $ and $ n $ are, you can see that for any $ c \in ( 0 , + \infty ) $, the function of the form $ f ( x ) = c x $ is a solution. For different values of $ m $ and $ n $, there are many other solutions as well.
The class of solutions of \eqref{3} is too broad to be nicely characterized. Even the special case of $ m = n = 1 $ gives rise to a broad class of solutions called the class of superadditive functions, which you can search the literature for. I only remark that in the other special case where $ m + n = 1 $ and $ f $ is considered to be continuous, one can prove that $ f $ must be concave: for every $ \mu , \nu \in [ 0 , 1 ] $ with $ \mu + \nu = 1 $ and every $ x , y \in ( 0 , + \infty ) $, one must have $ f ( \mu x + \nu y ) \ge \mu f ( x ) + \nu f ( y ) $. This is very remarkable, as \eqref{3} is only a single instance of the last inequality (with $ \mu = m $ and $ \nu = n $). While this may be a nice characterization of the solutions in this particular case, it again shows that even with further powerful restrictions, the class of solutions can be too broad, as the class of concave functions is still very large and diverse. I only sketch a proof of concavity: substitute $ m x _ 1 + n x _ 2 $ for $ x $ and $ m x _ 3 + n x _ 4 $ for $ y $ in \eqref{3} and use \eqref{3} itself to get $$ f \left( m ^ 2 x _ 1 + m n x _ 2 + m n x _ 3 + n ^ 2 x _ 4 \right) \ge m ^ 2 f ( x _ 1 ) + m n f ( x _ 2 ) + m n f ( x _ 3 ) + n ^ 2 f ( x _ 4 ) \text . $$ Iterate this way and for each positive integer $ K $ get an inequality of the form $$ f \left( \sum _ { i = 1 } ^ { 2 ^ K } m ^ { j _ i } n ^ { K - j _ i } x _ i \right) \ge \sum _ { i = 1 } ^ { 2 ^ K } m ^ { j _ i } n ^ { K - j _ i } f ( x _ i ) \text . $$ As $ m , n \in ( 0 , 1 ) $, you can choose a positive integer $ K $ sufficiently large so that every term of the form $ m ^ j n ^ { K - j } $ becomes arbitrarily small. Therefore, for any $ \mu , \nu \in [ 0 , 1 ] $, there is such large enough $ K $ for which you can partition the set of indices $ { 1 , \dots , 2 ^ K } $ into two sets $ \mathcal I $ and $ \mathcal J $ with the property that $ \sum _ { i \in \mathcal I } m ^ { j _ i } n ^ { K - j _ i } $ and $ \sum _ { i \in \mathcal J } m ^ { j _ i } n ^ { K - j _ i } $ are respectively arbitrarily close to $ \mu $ and $ \nu $. For such $ \mathcal I $ and $ \mathcal J $, let $ x _ i = x $ for all $ i \in \mathcal I $ and $ x _ i = y $ for all $ i \in \mathcal J $. Then by continuity of $ f $, you'll get an approximation of the inequality $ f ( \mu x + \nu y ) \ge \mu f ( x ) + \nu f ( y ) $, which can be made as precise as you need, and the conclusion follows.