In order to decrypt a cipher text using hill cipher, we must first find the inverse matrix of a given matrix.
From this link http://en.wikipedia.org/wiki/Hill_cipher,
| 6 24 1 |-I | 8 5 10 |
| 13 16 10 | = | 21 8 21 |
| 20 17 15 | | 21 12 8 |
but from my java program , the output of the inverse after using Gauss Jordan Elimination is
| 6 24 1 | | 0.159 -0.778 0.508 |
| 13 16 10 | = | 0.011 0.159 -0.107 |
| 20 17 15 | | -0.024 0.857 -0.490 |
I don't really know why is it different and not sure which is correct. Please help.
You must work in the integers modulo $26$, not over the reals (which is what your program seems to be doing).
To check the inverse, just do the multiplication: e.g. the inner product of the row $(6\,\,24\,\, 1)$ with the column $(8\,\,21\,\,21)$ equals $6 \times 8 + 24 \times 21 + 1 \times 21 = 573$, which modulo 26 equals 1 (as $572$ is divisible by $26$), so we get a $1$ at the left upper number of the product matrix. Similarly for the other ones.
You can do Gaussian elimination, but division works differently in the integers modulo $26$. (For one thing, you cannot always divide, as 26 has non-trivial divisors!) Look into working inside that ring first....