In the linear least squares problem one minimizes the norm of the residual vector: $$ \chi^2=\|\vec{b}-A\vec{x}\|_2^2=\sum_i^m{|b_i-\sum_j^nA_{ij}x_j|^2} $$ where $A \in \mathbb{R}^{m\times n}, \vec{b}\in \mathbb{R}^{m}, \vec{x}\in \mathbb{R}^{n}$.
The solution to the problem has to satisfy normal equations: $$ A^\top A\vec{x}=A^{\top}\vec{b} $$
However how does the least square problem look like in the general case of an arbitrary operator A? In particular, how do the operators $A^\dagger$ and $A^{\dagger}A$ look like in case of integral operator: $$ A: x(r) \rightarrow \int{a(R,r)x(r)dr} $$
I found that the textbook on Advanced Linear Algebra by Steven Roman 1 on pages 433-434 gives an answer that I was looking for:
For an arbitrary operator $A$ the normal equation still looks the same except the transpose of $A$ is changed to the adjoint of $A$: $$ A^{\dagger}Ax=A^{\dagger}b $$
To find the $A^{\dagger}$ for an integral operator $A$ we can use the definition of the adjoint operator: $$ \langle Ax,y\rangle=\langle x,A^{\dagger}y\rangle=\int {dR \left( \int a(R,r)x(r)dr\right)y^*(R) }=\int dr\left( \int a^*(R,r)y(R)dR\right)^*x(r) $$ Then $$ A^{\dagger}x=\int{a^*(R,r)x(R)dR} $$ $$ A^{\dagger}Ax=\int{\int{a^*(R,r')a(R,r)x(r)}dr} $$