I am thinking on matrix-like entities with continuous indexes. I think, maybe such "continuous matrices" could be defined as complex-valued functions on $\mathbb{R}^+\times\mathbb{R}^+$.
I think it is trivially visible that the continuous analogy of the unit matrix is $\delta_{xy}$. Multiplication could be defined as
$$(f \circ g)(x,y)=\int_0^\infty f(x,u)g(u,y)du$$
I am looking for the $f \in \mathbb{R}^+\times\mathbb{R}^+ \rightarrow \mathbb{C}$ with a given $g \in \mathbb{R}^+\times\mathbb{R}^+ \rightarrow \mathbb{C}$, for which
$$\int_0^\infty f(x,u)g(u,y) du = \delta_{xy} | \forall \{x,y\} \subset \mathbb{R}^+ $$
Does this algebra has a name? What could be the continuous analogy of the matrix inversion in it?
The generalization of matrices to "continuous indices" are called linear operators. What you have in mind seem to be operators of spaces of functions and these operators are defined by "integral kernels" $k(x,y)$, i.e. a function $f$ is mapped to some $g$ by $$g(y) =\int k(x,y)f(x)dx $$ (omitting details about the domains).
The thing with these kind of operators is that it is not clear whether they are continuous and invertibility is much more subtle. For example the operator corresponding to a continuous kernel $k$ leads to a compact operator between the spaces of continuous functions (and the same holds for a square integrable kernel and square integrable functions). Compact operators have the property that they are not continuously invertible even when restricted to their range and when you factor out the null space (you need that the range of the operator is not finite dimensional for this to hold). You still can define the Moore-Penrose pseudo inverse for linear operators, but usually it is not defined everywhere and is not continuous (read unbounded).