$N:=\{g\in GL(n,R) : g_{ij}=0 \forall j>i , g_{ii}=1 ∀i\}$.
For this matrix group, how can we show that it is a Lie group? I am at the beginning of the subject of Lie groups so I can not understand the mixture of a group and differential properties. No idea comes in my mind how to show that the group multiplication and the inversion are smooth.
In some lecture notes, I have read that the smoothness of the multiplication can be shown by using the fact that every bilinear form on a finite dimensional vector space is smooth, but anyway, without proving this statement also, I can not use it.
The group you're describing is the group of all unipotent, lower-triangular matrices. Unipotent just means all of its eigenvalues are $1$, or equivalently for lower-triangular matrices, all of its diagonal entries are $1$.
To see that this is indeed a smooth manifold, can you find a natural set bijection $\varphi$ from $N$ to $\mathbb{R}^{\frac{n(n-1)}{2}}$? $\varphi$ allows us to give $N$ the structure of a topological space (via the analytic topology on Euclidean space) such that $\varphi$ is actually a homeomorphism of topological spaces. The pair $(N,\varphi)$ is thus an atlas with only one chart. Since the only transition function is $\varphi\circ\varphi^{-1}=\mathrm{id}$, which is smooth on Euclidean space, we see that $N$ has the structure of a smooth manifold.
Basically, the previous paragraph just says we can view $N$ geometrically as Euclidean space of dimension $\frac{n(n-1)}{2}$.
To see that group multiplication and inversion are smooth maps, you should write out what happens when multiplying two matrices in $N$. The entries of the product are polynomial functions in the entries of the two given matrices, and polynomials are definitely smooth functions. Also, the inverse of a matrix in $N$ will have entries given by polynomial functions as well, because the determinant is $1$. This ensures that inverting a matrix involves no denominators (we're dividing each entry of the adjoint matrix by $1$).