In Fisher’s discriminant for multiple classes, How do you manage when $(Sw)$ is singular matrix (so you cant get $(Sw)^{-1}$)?

Question

I am trying to use Fisher’s discriminant for multiple classes to reduce the Dimension of the MNIST data set, similar to this post: https://towardsdatascience.com/an-illustrative-introduction-to-fishers-linear-discriminant-9484efee15ac

But in my Implementation in Python, when I compute $Sw$ it is a singular matrix so I can not get $(Sw)^{-1}$ to obtain the eigenvectors from $(Sw)^{-1}Sb$. Am I doing something wrong or what can I do to solve this problem?

I am following the steps given in Pattern Recognition and Machine Learning By Bishop.

Thanks.

Answer 1

You are right, it could be singular. This is known as the small sample size (SSS) problem.

A very typical way is to use the pseudoinverse to estimate the required directions.

There have been extensive research to this problem. Here, you can see a survey of various approaches such as regularized LDA, Orthogonal LDA, Uncorrelated LDA, and Null Space LDA.