I am junior in this field. I know the definition of a Lie algebra of a matrix group:
The Lie algebra of a matrix group $G\subset GL_n(\mathbb{R})$ is the tangent space to $G$ at $I$ (identity element). It is denoted by $T_IG$.
From this definition, we know that matrices in $G$ must be invertible. So if $G$ is not in $GL_n(\mathbb{R})$, then we cannot define its Lie algebra. (The operation of this group should be multiplication.)
My question is:
If today I can define a new group with a new group operation. And by this
operation, I can define inverse elements. So matrices in this new group are invertible under this new operation. Can I define its tangent space and Lie algebra?
Yes, if your group is a smooth manifold and multiplication and inversion are smooth maps. This is the definition of a Lie group. All Lie groups have a Lie algebra, but if your group is not a subgroup of $GL(n,\Bbb R)$, the Lie backet might be different from the standard one on $M(n,\Bbb R)$.