I am not sure if I have seen integral transforms in the right way, but given a transform like Fourier transform - it's actually a basis transformation right ?
$$ F(y) = \int K(x,y) f(x) \text{d}x $$ where $K(x,y) = \text{e}^{-ixy}$ for the case Fourier transform. The functions $F(y)$ and $f(x)$ can be seen as $\left<y|F\right>$ and $\left<x|f\right>$ respectively. In such a case the above integral equation can be rewritten as -
$$ \left< y|F \right> = \left<y|\mathbb{\hat I}|F\right> = \sum_x \left<y |x\right> \left<x |f\right> $$
So is $\left<y |x\right>$ one way of looking at the integral kernel for all general cases ? If not, I wish to understand how one can precisely look at integral kernels.
EDIT 1: I also wish to know that can transforms like Laplace, Mellin etc. also be treated like that as Transformation matrix, also in which case it might not be unitary matrix in all cases, but rather just a map from one inner product space to another.
Intuitively, $K(x,y)$ is a "continuous" matrix $K$ acting on a "continuous" row vector vector $f$: $$ \sum_{x}f(x)K(x,y) = \int f(x)K(x,y)dx $$ The associated quadratic form would be $$ \sum_{x}f(x)K(x,y)g(y)^{\star} = \int\int K(x,y)f(x)\overline{g(y)}dxdy $$ This would be selfadjoint if $K(x,y)=\overline{K(y,x)}$, analogous to a Hermitian matrix, and would lead to an observable. A type of condition that allows everything to work really well is the Hilbert-Schmidt condition: $$ M^{2}= \iint |K(x,y)|^{2}dxdy < \infty. $$ That may be a little restrictive for what you're doing. I'm not sure, but it's very nice from a Mathematical point-of-view because it allows one to define a bounded linear operator $K : L^{2}(\mathbb{R})\rightarrow L^{2}(\mathbb{R})$: $$ Kf = \int K(x,y)f(x)dx. $$ This follows from Cauchy-Schwarz: $$ |Kf(y)|^{2} \le \int |K(x,y)|^{2}dx\int |f(x)|^{2}dx \\ \int |Kf(y)|^{2}dy \le \int\int |K(x,y)|^{2}dxdy\int|f(x)|^{2}dx \\ \|Kf\|^{2} \le \int\int|K(x,y)|^{2}dxdy \|f\|^{2} \\ \|Kf\| \le \left(\int\int|K(x,y)|^{2}dxdy\right)^{1/2}\|f\| \\ \|Kf\| \le M\|f\|. $$ Here $M$ is the Hilbert-Schmidt constant defined above. This describes how Hilbert originally started in generalizing matrices. Eventually von Neumann's operator theory replaced these ideas because (1) Matrices cannot distinguish between different linear operators and (2) linear operators allowed more general objects that were better suited for Quantum.