Given a separable Hilbert space, we can find a Hilbert basis,which is countable and each element can be expressed as an infinite linear combination with $l^2$ coefficients. Ok this is clear and well-known in maths.
But in QM, they also use uncoutnable family of states and state that each element of the Hilbert can be written as an integral of those with "coefficients" a function in $L^2$
My question : what is the name of this, mathematically ? It is not an instance of Lebesgue integral, since the function we are integrating is not complex valued.
There are several ways to answer this question but the simplest one is the following: It's just a matter of terminology and you shouldn't take the word "continuous basis" literally, as in, it's a continuum set of basis elements. To be more concrete consider the fourier inversion theorem: For sufficiently well behaved $f$ (for instance satisfying the Dini condition), we have:
$f(x) = \sqrt\frac{1}{2\pi} \int \hat{f}(k) e^{ikx} dk $
where $\hat{f}(k)$ is the fourier transform of $f$
Also note that $\int e^{ik(x-x')}dk = \delta(x-x')$ in the sense of distributions
These are rigorous mathematical statements. Physicist then come and look at this formula noting its similarity to how a periodic function can be expanded in terms of a discrete sum of complex exponentials and just declare "exponentials form a basis of $L^2(\mathbb{R})$ in the same way that the discrete exponentials form a basis of $L^2(U)$ where $U$ is a bounded interval, also the exponentials are still orthogonal but the discrete Kronecker delta is replaced by its continuous analogue, the dirac delta function". This is actually how physics students are taught in class, even in math methods classes. Notice of course this abuses the term "basis" horribly because for one thing the exponentials are not even elements of $L^2(\mathbb{R})$, secondly $L^2(\mathbb{R})$ is still separable and thus has a countable orthonormal basis (in fact, the Hermite polynomials, which are eigenfunctions of the Fourier operator, form such a set). But all that doesn't matter because it's just a matter of terminology, whether you call the above Fourier formula as a "continuous expansion in terms of exponentials" or you call it Fourier inversion theorem is irrelevant to the calculations, it is still true regardless of how you choose to call it, hence physicists get away with it. It is harmless to call it a continuous expansion as long as you are using the correct rigorous formulas.
It's the same story with dirac delta function. According to physicists it's a function which evaluates to $\infty$ at $0$ and is $0$ otherwise whose integral is 1. Using modern integration theory we know this is nonsense but the key idea is that physicists only ever use the delta function in an integral sense i.e. the delta function only ever appears as integrating against some other function, or in mathematical terms, as a distribution. Hence, again, they get away with it but only because they use it in the write way despite calling it the wrong mathematical object. I hope you see the pattern here: You can abuse terminology all you want as long as you use the objects in the right way.
Now another way to look at it is through the Von Neuman Spectral theorem. This theorem requires quite a bit of formalism to even state and involves the notion of projection valued measures (essentially, they are measures which are operator valued). It states that any self adjoint operator is an integral over some projection valued measure. this decomposition is quite elegant and intuitively gives (at least some) rigorous meaning to otherwise vague statements made by physicists like "dirac deltas form an eigenbasis for the position operator" or "exponentials form an eigenbasis for the momentum operator". It all comes out quite naturally from this spectral theorem. The details for this you must learn from a book. One good reference I would recommend is "Quantum Theory for Mathematicians" by Brian Hall.