If $0 < b < e < a$, how can I determine whether $a^b$ or $b^a$ is greater?
I know this question has been asked before, but I want to solve this question by this method. It worked fine for first two cases, and I want to know whether I can apply it to the third case as well.
We can equivalently compare $(b^{1 \over b})^{ab}$ and $(a^{1 \over a})^{ab}$. So define $$f(x)=x^{ 1 \over x} .$$ Now, $$f'(x)={1 \over x} \cdot (x)^{{1 \over x}-1}+x^{1 \over x} \cdot \ln{x} \cdot{{(-1)} \over {x^2}} =\bigg(\frac{x^{ 1 \over x}}{x^2}\bigg)\cdot \bigg(1- \ln x\bigg)$$ So for $x < e$ , $f'(x)>0$ , and so $f(x)$ is increasing, and for $x > e$ , $f'(x)<0$ , and so f(x) is decreasing.
We also have $\lim_{x \to 0}x^{ 1 \over x}=0$ and $\lim_{x \to \infty}x^{ 1 \over x}=e^{\lim_{x \to \infty}\frac{\ln x }{x}}=1$, so our function will increase from $x=0$ to $x=e$ and then decrease, tending to $1$ as $x \to \infty$.
Case 1: $a>b>e$: $a^{1 \over a} < b^{1 \over b}$, so $a^{{ab} \over a} < b^{{ab} \over b}$ and $a^{b} < b^{a}$. For example, $4^5 > 5^4$.
Similarly:
Case 2: $a<b<e$: $a^{1 \over a} < b^{1 \over b}$, so $a^{{ab} \over a} < b^{{ab} \over b}$ or $a^{b} < b^{a}$ For example, $\sqrt2^{\sqrt3}$ < $\sqrt3^{\sqrt2}$.
Case 3: $b < e < a$.
Here's where I'm stuck. Can we apply same method to solve it?
This is certainly the hardest of the three cases: In this case sometimes $a^b$ is larger, sometimes $b^a$ is.
Since $\log$ is increasing, it follows from your argument that we may as well work with the more tractable function $\log f(u) = \frac{\log u}{u}$, and so compare $$\frac{\log a}{a} \qquad \textrm{and} \qquad \frac{\log b}{b} .$$ Again since $\log$ is increasing, $\log f$ is increasing on $(0, e)$ and decreasing on $(e, \infty)$.
As a toy example, let's consider the case $a = 3, b = 2$, so that we're comparing $2^3$ and $3^2$. Of course we can evaluate both expressions directly and compare, but we'd like to use a method that also applies when we can't compute by hand. If we notice that $$\frac{\log 2}{2} = \frac{\log 4}{4},$$ then since $3$ and $4$ are both greater than $e$, we've reduced to case (1): $$\frac{\log 2}{2} = \frac{\log 4}{4} < \frac{\log 3}{3} .$$
We can generalize this method: For any $s > 0$ we have $$\frac{\log \left[\left(1 + \frac{1}{s}\right)^s\right]}{\left(1 + \frac{1}{s}\right)^s} = \frac{\log \left[\left(1 + \frac{1}{s}\right)^{s + 1}\right]}{\left(1 + \frac{1}{s}\right)^{s + 1}}$$ (check this!). But $\left(1 + \frac{1}{s}\right)^s < e < \left(1 + \frac{1}{s}\right)^{s + 1}$, so choosing a bound for $b$ of the form $\left(1 + \frac{1}{s}\right)^s$, gives a bound $\frac{\log b}{b} > \frac{\log c}{c}$ or $\frac{\log b}{b} < \frac{\log c}{c}$ for some $c > e$. If we choose $s$ appropriately, then, we can compare $\frac{\log a}{a}, \frac{\log b}{b}$ as desired. (In our above example, we took $s = 1$.)
Example Consider $a = e + \frac{1}{2}, b = e - \frac{1}{2}$---it is not reasonable to try to compute $a^b$ and $b^a$ by hand! Using a fourth-order Taylor polynomial estimate for $\exp$ we find that $$e - \frac{1}{2} < \left(1 + 1 + \frac{1}{2} + \frac{1}{6} + \frac{1}{24} + \frac{1}{24}\right) - \frac{1}{12} = \frac{9}{4},$$ and using a third-order Taylor polynomial estimate gives $$e + \frac{1}{2} < \left(1 + 1 + \frac{1}{2} + \frac{1}{6} + \frac{1}{6}\right) + \frac{1}{2} = \frac{10}{3} < \frac{27}{8}.$$ But taking $s = 2$ in the above identity gives $$\frac{\log \frac{9}{4}}{\frac{9}{4}} = \frac{\log \frac{27}{8}}{\frac{27}{8}},$$ and putting this all together gives $$\frac{\log\left(e - \frac{1}{2}\right)}{\left(e - \frac{1}{2}\right)} < \frac{\log \frac{9}{4}}{\frac{9}{4}} = \frac{\log \frac{27}{8}}{\frac{27}{8}} < \frac{\log\left(e + \frac{1}{2}\right)}{\left(e + \frac{1}{2}\right)} ,$$ and we conclude $$(e - 1)^{e + 1} < (e + 1)^{e - 1} .$$