I tried using the Rational Root Theorem on $x^3-6x^2+4x-5=0$. However, I could not find a rational root of the cubic. When I plugged the cubic into Wolfram Alpha, it yielded a very messy real solution:
How do you tell whether the RRT doesn't work on a particular cubic of the form $ax^3+bx^2+cx+d=0$? How do you tell whether $ax^3+bx^2+cx+d=0$ doesn't have any rational roots?
On
When you apply the rational root theorem, you find all the rational roots, if there are any. If the theorem finds no roots, the polynomial has no rational roots. (For a cubic, we would observe that the polynomial is irreducible over the rationals. This is because a factorization of the cubic is either the product of a linear factor and a quadratic factor or it is the product of three linear factors. Since in either case there is a linear factor, there would be a root in the rationals. With no rational root, we're done.)
Determining whether a polynomial has rational roots is done by means of the rational root test. There is no prior test one uses to determine whether to use the rational roots test; one starts with the test to find and remove all the "easy" factors. (One should also use the polynomial GCD and the derivative to remove repeated roots, but that's outside the scope of your question.) Only after all the easy roots are removed (or not found) do you move on to hard tools like the cubic formula.
For more discussion, see casus irreducibilis in the English Wikipedia.
On
After some extensive testing of my own on another polynomial that had zeros but could not be determined by the rational roots test I came to the following conclusion:
It appears, that the RRT will only give you $\mathit rational$ numbers as answers. Meaning, a polynomial might very well have zeros but the RRT will not give you any results. That is because said zeros are real numbers which cannot be represented by rational numbers. $(x\in\Bbb R, x \notin\Bbb Q)$
For OP's polynomial $x^3-6x^2+4x-5=0$, doing the RRT quickly leads one to believe that there are no zeros for it ;(it is left to the reader to confirm this as an exercise). It does however have one. Using an online zeros calculator one finds that the zero in question is $x=5.43316142415711$. This number is real (as in $\Bbb R$) and cannot be represented by a fraction.
In conclusion: The RRT will only give you results that are part of $\Bbb R$. There may however be polynomials that do very well possess zeros but those may not be part of $\Bbb R$. I hope this answers the question in the title.
The rational roots theorem yields potential rational roots. It says nothing about roots that are not rational. :)
To detect how many real roots there will be, take a look at the discriminant. If $a_1, a_2, a_3$ are roots of the polynomial, then:
$$\Delta = (a_1-a_2)^2(a_1-a_3)^2(a_2-a_3)^2 = b^2c^2-4ac^3-4b^3d-27a^2d^2+18abcd$$
Yuck! Note that $\Delta \geq 0 \iff$ there are $3$ real roots (counting multiplicity). Otherwise, there will be one real and two complex roots.
Actually finding these roots requires the God-awful cubic formula or certain numerical approximations, such as Newton's method, if you're fine with error.