I've been looking at f(x)=cosx, for which the Newton's formula becomes
$Xn = Xn+1 + cosx/sinx$ $= Xn+1 + 1/tanx$
Using 3 as my initial guess, the calculator computes the root to be -3pi/2. And using 5 as my initial guess, the calculator computes the root to be 3pi/2. The latter is the correct answer.
I understand that the method fails if the derivative at some point between the guess and the root is zero because of which the answer becomes indeterminate. But, here, the answer isn't indeterminate; in fact, the calculator is computing it almost correctly with just the sign being incorrect.
If I look at cosx, the graph makes a turn around 3, where its tangent is almost horizontal, hence giving sinx to be zero and the answer being indeterminate. If I look at 1/tanx, the graph never touches 3, so how does the value converge to -3pi/2 using this value since there is blank space for 1/tanx there? How are Newton's iterations traveling down the graph when there is no line present there? Finally, what makes it give a positive value if we take 5 as our initial guess?
Using just the calculator, it's easy to see that cosx/sinx give smaller negative values for x, which are small numbers to subtract from $5$, so the final answer is positive. The opposite is the case for 3: $cosx/sinx$ give negative values which are larger than $3$, so the answers obtained are negative, hence giving -3pi/2. What I'm confused about is the reasoning of the graphs behind this.

You’re wrong that any of the two are the only right guess. Both of them are valid roots of cosine being equalled to zero. It seems that in these two examples the algorithm is biased towards returning a root lying on the left. The specifics about why Newton-Raphson (NR) has yielded a left-lying optimum are left to the next answerer, but any root-finding algorithm that returns at most one root and is sensitive to initial conditions may return any root of the function and the root which it returns may vary with the initial condition. To recap, you are incorrect NR returned the “wrong” answer, and a general reason why root-finding algorithms’ “equilibrium” may vary is that they may be sensitive to initial conditions