Let $G=GL(2, \mathbb{F}_5)$, i.e., group of invertible $2\times 2$ matrices over a finite field with $5$ elements. Let $S$ be the subset of those elements of $G$ that can be written in the form $LU$, where $L$ is a lower triangular matrix and $U$ is an upper triangular matrix.
Find the cardinality of $S$?
I know that $$|G|=(5^2-1)(5^2-5)=480.$$ How to look for the matrices that can be decomposed as $LU$? Is there any direct formula? Please help.
On
The size of $G$ is 480. We need to find the size of $S$. We know that a $2\times 2$ Matrix is LU decomposible if it's principal minor of first order is non-zero. We note that number of such matrices is given by $$\begin{bmatrix}0&x\\y&z\\\end{bmatrix},$$ where $x,y,z$ are in $\mathbb{F}_5$.
Note that $x$ has $4$ choices as it can not be zero. Also $z$ has all the possible $5$ choices. Since determinant should be non-zero, number of choices for $y$ are $4$. In all there are $4*4*5=80$ choices. So, the matrices with LU decomposition are $$480-80=400.$$
Not an answer, but a nudge
I think that the point of this exercise is to get you to think about the shape of an LU-decomposition, and about the process for constructing one. For instance over the reals, if we look at \begin{align} \pmatrix{3 & 1 & 6 \\ 2 & 0 & 4 \\ 1 & 1 & -5} &= L U \\ &= \pmatrix{1 & 0 & 0 \\ * & 1 & 0 \\ * & * & 1} \pmatrix{a & * & * \\ 0 & * & * \\ 0 & 0 & *} \end{align} we can see that $a$ must be $3$. What does computing the product of the middle row of $L$ and the first column of $U$ now tell you about the (currently) unknown entry in the (2, 1) spot of $L$?
Working through this, there's essentially an algorithm for generating $L$ and $U$ from the original matrix $M$, but that algorithm fails in certain cases -- I leave it to you to discover which they are -- and for any other matrix you can find $L$ and $U$ by direct reasoning --- reasoning that applies over any field, not just the reals.
Fortunately, in your case the situation is $2 \times 2$, so the 'problem cases' are pretty limited.
When you're done, you'll have a count of the size of $S$, and if you're really energetic you can generalize to the 3x3 case, and pretty soon (OK, maybe that's optimistic!) you'll have the very "formula" that you were hoping to look for.
But in this small situation, you can probably just work it out faster that you can find a formula. I certainly can, and I've never seen a problem like this one in my life, and haven't really thought about LU since about 1990.