Suppose a black box produces a sequence of vectors $(x_k)$ in $\mathbb{R}^n$.
Suppose that sequence is not convergent.
What are good measures of nonconvergence/chaoticness etc. to evaluate nonconvergent sequences?
I am interested in measuring how much the sequence wanders/exploits $\mathbb{R}^n$, and whether or not it exhibits "chaotic behavior" etc. I could monitor $\|x_k\|$ to see whether or not it blows up. Any other measures (ideally with literature references) would be appreciated.
It seems you have already got your answer in the comments. Nevertheless, I will add some more details.
Chaos itself is a vague concept and there does not exist a universally accepted definition for it. But from what I have seen, the most common way of detecting chaos is by calculating the Lyapunov exponents. Mathematically speaking
The Lyapunov exponent can also be considered as a measure of stability, i.e. in a linear system, the Lyapunov exponents are equal to the eigenvalues of its characteristic matrix. In this case, they can be easily calculated. But in general, there doesn't exist any straightforward way to calculate them.
There are several algorithms for estimating the Lyapunov exponents. You may refer to this article for a short survey on some of these methods. One simple method, as noted in the comments, is Wolf's algorithm. Another popular approach is proposed by Eckmann et al. and later optimized by Rosenstein et al.
In practice $-$if you are familiar with Python, you can use the
noldspackage, which implements the Rosenstein method for computing the Lyapunov exponents. It has a simple interface: