So I was doing some physics and I can recover a variant of the Maxwell distribution (the answer is a gaussian)
$$P(k)dk = \int_{-\infty}^\infty P(k-x)P(x) dx dk $$
I was trying to extend this to the relativistic case. How would one find the general form of this probability distribution $P$? (Is it some kind of Bessel function?)
$$P(k) dk= \int_{0}^\infty P(\lambda)P(k/ \lambda) d \lambda dk $$
Edit:
Calculations of the first claim: $$P(k) dk = \int_{-\infty}^\infty (a^2/ \pi) e^{-a^2(k-x)^2-a^2x^2} dx dk= (a^2/ \pi) \int_{-\infty}^\infty e^{-a^2 (k^2 - 2x^2 - 2kx)}dx dk = (a^2/ \pi) \int_{-\infty}^\infty e^{-a^2 (2(k/2 +x)^2 +k^2/2)}dx dk$$
Taking $x \to k/2 + x$
$$(a^2/ \pi) \int_{-\infty}^\infty e^{-a^2 (2x^2 +k^2/2)}dx dk= |a| \sqrt{\frac{2}{\pi}} e^{-a^2k^2/2} dk $$
where $k \to \sqrt2 k$
$$\frac{|a|}{\sqrt{\pi}} e^{-a^2k^2} dk = P(k) dk$$
I agree with previous posters that a Gaussian is not a solution of $P*P=P$, as can be seen easily by working in Fourier space $\hat P \cdot \hat P = \hat P$ can only be solved if $\hat P =1$. Your mistake is precisely where you write $k \to \sqrt2 k$, which means you are replacing $P(k)$ with $P(\sqrt 2 k)$. I suppose you want to say something like $P(x)*P( x)=P(\sqrt 2 x)$. This shouldn't be a surprise since this is an expression for the distribution of the sum of two Gaussian random variables (as @Andrew has pointed out in the comments above).
In order to deal with equations like the second part $$\int_{0}^\infty P(\lambda)P(k/ \lambda)d\lambda $$, one performs the equivalent of the Fourier transform, but in scale space: the Mellin transform. $$\int_0^\infty k^\alpha \int_{0}^\infty P(\lambda)P(k/ \lambda)d\lambda dk$$ I wont go into the math, but if you look carefully, you will see that what is going on here is that ${k\over \lambda} = e^{\log(k)-\log(\lambda)}$ so $\log(k),\log(\lambda)$ behave like $k,x$ in the convolution example. Please note that you probably also have a normalization issue in the second integral as well which you should fix, before attempting to solve.