Let $\xi$ be a fixed unit vector in $\mathbb{R}^n$. I'm interested in evaluating the following Fourier transform $$F_\xi(k)=\int_{\mathbb{R}^n} d^nx \dfrac{1}{(\xi\cdot x)^\alpha}e^{ik\cdot x}\tag{1}$$
where $\alpha\in \mathbb{C}$. I imagine that not for all values of $\alpha$ will (1) converge, so I would also want to determine the values for which (1) makes sense, even if as a distribution in the $\alpha$ dependence.
Now, one way I thought about evaluating (1) was to choose the axes in $x$ space such that $\xi = (0,\dots, 1)$. Further, denoting $x=(x^1,\dots, x^{n-1}, z)$, $k=(k^1,\dots, k^{n-1},\lambda)$ and also $x_\perp = (x^1,\dots, x^{n-1})$ and $k_\perp=(k^1,\dots, k^{n-1})$ this would mean that
$$F_\xi(k)=\int_{\mathbb{R}^{n-1}}\int_{-\infty}^\infty d^{n-1}x_\perp dz \dfrac{e^{i\lambda z }e^{ik_\perp \cdot x_\perp}}{z^\alpha}=\int_{-\infty}^\infty dz \dfrac{e^{i\lambda z}}{z^\alpha}\int_{\mathbb{R}^{n-1}}d^{n-1}x_\perp e^{ik_\perp\cdot x_\perp}=(2\pi)^{n-1}\delta^{n-1}(k_\perp)\int_{-\infty}^\infty dz \dfrac{e^{i\lambda z}}{z^\alpha}\tag{2}.$$
Now, I'm confused on how to proceed with the last integral. As far as I know, for $\alpha=1$ the integral is $i\pi\operatorname{sgn}(\lambda)$ and for $\alpha$ integer I think it is also known. For generic $\alpha$ I'm unsure about what it gives.
In that case:
Is my approach to (1) correct in that it just reduces to $\int_{-\infty}^\infty dz \frac{e^{i\lambda z}}{z^\alpha}$?
How can I evaluate this integral (possibly as a distribution in $\alpha$) for $\alpha$ as general as possible?
I won't do all the computations, because it's too much algebra with many minus signs and factors of $i$, but hopefully this is enough to get you started.
Question 1.
For your first question, yes that's essentially right. To be more explicit, assume for simplicity $\xi=e_n= (0,\dots, 1)$; we can always return to the general case by pulling back/composing the final distributions by a certain rotation. We shall write $x\in\Bbb{R}^n$ as $x=(y,z)\in\Bbb{R}^{n-1}\times\Bbb{R}$. We consider the holomorphic branch of the logarithm by removing say the lower half of imaginary axis, so $-\frac{\pi}{2}<\arg<\frac{3\pi}{2}$. I shall write the exponent as $\lambda$ rather than $-\alpha$, since that's what I happened to write on paper. So, we wish to consider the mapping $f_{\lambda}:\Bbb{R}^n\to\Bbb{C}$, \begin{align} f_{\lambda}(x)&:=\begin{cases} z^{\lambda}&\text{if $z\neq 0$}\\ 0&\text{otherwise} \end{cases} \end{align} Define also $f_{\lambda,+}(x)=z^{\lambda}$ if $z>0$ and $0$ otherwise, and $f_{\lambda,-}(z)=|z|^{\lambda}$ if $z<0$ and $0$ otherwise. Then, we can write $f_{\lambda}=f_{\lambda,+}+ (-1)^{\lambda}f_{\lambda,-}$. Because of this decomposition, I shall only deal with $f_{\lambda,+}$. Sometimes, we may denote this function by $z_{+}^{\lambda}$.
Now, suppose $\text{Re}(\lambda)>-1$. Then, the pairing with any Schwartz function $\phi\in \mathcal{S}(\Bbb{R}^n)$ is defined as \begin{align} \langle z_+^{\lambda},\phi\rangle&:=\int_{\Bbb{R}^n}z_{+}^{\lambda}\phi(x)\,dx =\int_{\Bbb{R}^{n-1}\times (0,\infty)}\phi(y,z)\cdot z^{\lambda}\,dy\,dz. \end{align} Note that this is well-defined and continuous with respect to $\phi$ because $\phi$ being a Schwartz function means it satisfies the inequality $|\phi(y,z)|\leq \frac{M_{ab}}{(1+|y|^2)^a(1+|z|^2)^b}$, for all $a,b\in\Bbb{N}$. Here, $M_{ab}$ is a constant related to one of the Schwartz seminorms. Because of this bound on $\phi$, by choosing $b$ large enough and $a=1$, we get a finite integral, and because of the $M_{ab}$ bound, this proves continuity. Hence, $z_+^{\lambda}$ is a tempered distribution, for each $\text{Re}(\lambda)>-1$. Next, some basic complex analysis shows that for each $\phi$, we have a holomorphic function of $\lambda$.
We now aim to provide an analytic continuation. Note that \begin{align} \langle z_+^{\lambda},\phi\rangle&:=\int_{\Bbb{R}^{n-1}\times (0,\infty)}\phi(y,z)\cdot z^{\lambda}\,dy\,dz\\ &=\int_{\Bbb{R}^{n-1}}\int_0^{\infty}\phi(y,z)\cdot z^{\lambda}\,dz\,dy\\ &=\int_{\Bbb{R}^{n-1}}\left[\int_0^1[\phi(y,z)-\phi(y,0)]\cdot z^{\lambda}\,dz+ \frac{\phi(y,0)}{\lambda+1}+\int_1^{\infty}\phi(y,z)\cdot z^{\lambda}\,dz\right]\,dy\\ &=\int_{\Bbb{R}^{n-1}}\left[\int_0^1[\phi(y,z)-\phi(y,0)]\cdot z^{\lambda}\,dz+\int_{1}^{\infty}\phi(y,z)\cdot z^{\lambda}\,dz\,\right]\,dy+ \frac{1}{\lambda+1}\int_{\Bbb{R}^{n-1}}\phi(y,0)\,dy. \end{align} It is straightforward to verify that although we defined the LHS only for $\text{Re}(\lambda)>-1$, the RHS is in fact defined for all $\lambda$ such that $\text{Re}(\lambda)>-2$ and $\lambda\neq -1$, and that it is again a continuous functional of $\phi$. We thus use the RHS as a definition for the tempered distribution $z_+^{\lambda}$ when $\text{Re}(\lambda)>-2$, $\lambda\neq -1$. We can keep going in this manner: "regularize" the distribution by subtracting more and more terms of the Taylor expansion about $z=0$, which will yield corresponding "boundary terms". Each time we do this, we can extend the definition further, while keeping poles at the negative integers.
The net result of all this is that we now have a well-defined holomorphic mapping $\lambda\mapsto z_+^{\lambda}$ from $\Bbb{C}\setminus\{-1,-2,\cdots\}$ into $\mathcal{S}'(\Bbb{R}^n)$. One final remark is that Fourier transform is a continuous mapping on tempered distributions, so it preserves holomorphicity (i.e if $\lambda\mapsto g_{\lambda}$ are holomorphic tempered distributions, then so is the Fourier transform $\lambda\mapsto \mathcal{F}(g_{\lambda})$).
Now, we can come to the first question. Let $\phi_1\in\mathcal{S}(\Bbb{R}^{n-1})$ and $\phi_2\in \mathcal{S}(\Bbb{R})$, and consider the "tensor product" $\phi=\phi_1\otimes \phi_2$ defined as $\phi(y,z):=\phi_1(y)\phi_2(z)$. Now, for $\text{Re}(\lambda)>-1$, we have \begin{align} \langle\mathcal{F}(z_+^{\lambda}), \phi\rangle&:=\left\langle z_+^{\lambda}, \mathcal{F}(\phi_1\otimes \phi_2)\right\rangle\\ &= \left\langle z_+^{\lambda}, \mathcal{F}_{\Bbb{R}^{n-1}}(\phi_1)\otimes \mathcal{F}_{\Bbb{R}}(\phi_2)\right\rangle\\ &=\int_{\Bbb{R}^n}z_+^{\lambda}\mathcal{F}_{\Bbb{R}^{n-1}}(\phi_1)(y)\mathcal{F}_{\Bbb{R}}(\phi_2)(z)\,dy\,dz\\ &=\int_{\Bbb{R}^{n-1}}\mathcal{F}_{\Bbb{R}^{n-1}}(\phi_1)(y)\,dy\cdot \int_{\Bbb{R}}z_+^{\lambda}\mathcal{F}_{\Bbb{R}}(\phi_2)(z)\,dz\\ &=(2\pi)^{n-1}\phi_1(0)\cdot \langle z_{+}^{\lambda}, \mathcal{F}_{\Bbb{R}}(\phi_2)\rangle\\ &=\langle (2\pi)^{n-1}\delta_{\Bbb{R}^{n-1}},\phi_1\rangle\cdot \langle \mathcal{F}_{\Bbb{R}}(z_+^{\lambda}),\phi_2\rangle \end{align} Here, $\mathcal{F}$ is the full Fourier transform on $\Bbb{R}^n$, and $\mathcal{F}_{\Bbb{R}^{n-1}}$ is the partial Fourier transform on the first $n-1$ variables and so on; finally there's a slight abuse of notation here, on the RHS, the term $z_+^{\lambda}$ should be viewed as a tempered distribution defined on $\Bbb{R}$ rather than $\Bbb{R}^n$. Note that here, since $\text{Re}(\lambda)>-1$, everything is in $L^1$, which is why we could use Fubini. By the existence and uniqueness of tensor product of tempered distributions, it thus follows that \begin{align} \mathcal{F}(z_+^{\lambda})&=(2\pi)^{n-1}\delta_{\Bbb{R}^{n-1}}\otimes \mathcal{F}_{\Bbb{R}}(z_+^{\lambda}). \end{align} We only showed this for $\text{Re}(\lambda)>-1$. But now by the previous comments, both sides are holomorphic (tempered-distribution valued) functions of $\lambda$, which agree on a non-trivial open set; therefore, they agree for all $\lambda\in\Bbb{C}\setminus\{-1,-2,\dots\}$.
You can repeat the same story with $z_-^{\lambda}$, and thus get the same equation with $z^{\lambda}$. This addresses the first part.
Question 2.
Now, we shall consider $z_+^{\lambda}$ as a tempered distribution on $\Bbb{R}$ not $\Bbb{R}^n$. Restrict once again to $\text{Re}(\lambda)>-1$, so that we have a function rather than distribution. Consider the family of functions $u_{\epsilon}(z)= z_+^{\lambda}\cdot e^{-\epsilon z}$, for $\epsilon>0$. We can easily see that $u_{\epsilon}\to z_{+}^{\lambda}$ pointwise on $\Bbb{R}$ and also in the topology of tempered distributions $\mathcal{S}'(\Bbb{R})$ (because of dominated convergence). Since Fourier transform is continuous on $\mathcal{S}'(\Bbb{R})$, we can first calculate the Fourier transform of $u_{\epsilon}$ (which is much easier since it belongs to $L^1(\Bbb{R})$), and then take limits $\epsilon\to 0^+$.
Let us now calculate the Fourier transform of $u_{\epsilon}$: \begin{align} \mathcal{F}_{\Bbb{R}}(u_{\epsilon})(\zeta)&=\int_0^{\infty}z^{\lambda}e^{-\epsilon z}e^{iz\zeta}\,dz =\int_0^{\infty}z^{\lambda}e^{i(\zeta+i\epsilon)z}\,dz= \int_R\left(\frac{iw}{(\zeta+i\epsilon)}\right)^{\lambda}e^{-w}\frac{i}{\zeta+i\epsilon}\,dw \end{align} where we performed a change of variables. This results in the contour of integration $(0,\infty)$ being rotated to a certain ray $R$ emanating from the origin. Note that since $z=\frac{iw}{\zeta+i\epsilon}$, the multiplication by $i$ transforms the positive real axis to the positive imaginary axis, so dividing by $\zeta+i\epsilon$ reduces the argument slightly, so that $R$ is a ray lying in the right half plane. In this half-plane, $w^{\lambda}e^{-w}$ decays exponentially, so we can use Cauchy's theorem to shift the contour of integration back to the positive real axis. Hence, \begin{align} \mathcal{F}_{\Bbb{R}}(u_{\epsilon})(\zeta)&= \left(\frac{i}{\zeta+i\epsilon}\right)^{\lambda+1}\int_0^{\infty}w^{\lambda}e^{-w}\,dw=\left(\frac{i}{\zeta+i\epsilon}\right)^{\lambda+1} \cdot \Gamma(\lambda+1), \end{align} where we used the definition of the Gamma function. Now, we have to take the limit $\epsilon\to 0^+$. By the way, I think it is standard notation to write $(\zeta+i0)^{-(\lambda+1)}$ to mean the tempered distribution $\lim\limits_{\epsilon\to 0^+}(\zeta+i\epsilon)^{-(\lambda+1)}$ (the limit exists, since the LHS has a limit in $\mathcal{S}'(\Bbb{R})$ as $\epsilon\to 0^+$). Thus, \begin{align} \mathcal{F}_{\Bbb{R}}(z_+^{\lambda})&=i^{\lambda+1}\cdot \Gamma(\lambda+1)\cdot (\zeta+i0)^{-(\lambda+1)}\tag{$*$}, \end{align} where this holds for all $\text{Re}(\lambda)>-1$. Since both sides are defined and holomorphic for $\lambda\in\Bbb{C}\setminus\{-1,-2,\dots\}$, the equality $(*)$ holds for all $\lambda$ in this set.
If we want to obtain a more explicit expression for the tempered distribution $(\zeta+i0)^{-(\lambda+1)}$, note that the pointwise limit is given by \begin{align} \lim_{\epsilon\to 0^+}\mathcal{F}_{\Bbb{R}}(u_{\epsilon})(\zeta)&=i^{\lambda+1}\Gamma(\lambda+1) \begin{cases} |\zeta|^{-(\lambda+1)}&\text{if $\zeta>0$}\\ e^{-(\lambda+1)i\pi}|\zeta|^{-(\lambda+1)}&\text{if $\zeta<0$} \end{cases} \end{align} A slightly more tedious verification shows that the same is true in the sense of tempered distributions as well provided $\lambda\notin \Bbb{Z}$: \begin{align} \mathcal{F}_{\Bbb{R}}(z_+^{\lambda})&= i^{\lambda+1}\Gamma(\lambda+1) \cdot \left[\zeta_+^{-(\lambda+1)}+ (-1)^{-(\lambda+1)}\zeta_-^{-(\lambda+1)}\right],\tag{$**$} \end{align} where $\zeta_{\pm}^{\text{powers}}$ are the tempered distributions defined previously (the LHS is defined away from $\{-1,-2,\dots\}$ and the RHS is defined away from $\{0,1,2,\dots\}$). Therefore, the equality holds for all $\lambda\in\Bbb{C}\setminus \Bbb{Z}$. Note that $(**)$ is just a more explicit way of saying $(*)$, in terms of the more familiar distributions $\zeta_{\pm}^{\text{powers}}$.
Once again, you can play the same game with the Fourier transform of $z_{-}^{\lambda}$, and then add the two results to get the Fourier transform of $z^{\lambda}$. My guess is that when you add the two results, the singularities at the integers should cancel out which then allows us to analytically continue over the entire complex plane., though I haven't carefully verified this, so I'm not completely sure.