I have a problem where the regularizer is the nuclear norm and the matrix being regularized is $n \times d$ with $d < n$. I was initially not getting low rank for the desired performance, the matrix was still at rank $d$. When the rank started reducing from $d$, the performance was not good enough.
So, I started trying Schatten norm with $p = 0.5$. However, now I am getting a regularization path where there is suddenly a huge jump in the rank from $d$ to a low value (like $4$). Naturally, when the jump to the (much lower) rank happens, the performance is very poor. I tried scaling the loss part, to see whether it will stretch the regularization path. But the steep jump in the rank continues to happen.
Ideally, I want an optimal point where the rank of the matrix has reduced from $d$ and the desired performance is achieved at that point. I am a little surprised that low rank regularization has such practical difficulties.