In Quantum mechanics we said that $\langle x'|\psi \rangle = \psi(x)$, where $\langle \phi|\psi \rangle $ is the dot product in $L^2(\mathbb{C})$.
I found out, that this is true, if you set x' to be the delta function $\delta(x)$
Now I also found $\langle p'|\psi \rangle = \tilde{\psi}(p)$, where $\tilde{\psi}$ is the fourier transform of $\psi$. My question is: Does anybody here know what $p'$ could be, so that this expression makes sense?
What you want to keep in mind is that when we say a quantum state is represented by a state $\mid \psi \rangle$ in a Hilbert space we haven't yet committed ourselves to a particular Hilbert space.
When a system has a classical analogue we introduce hermitian operators $X$ and $P$ which obey a canonical commutation relation $[X,P]=i\hbar $. These operators have eigenvectors $ \mid x \rangle $ and $ \mid p \rangle $ which form a complete orthonormal basis in our space. They also generally have a continuous spectrum of eigenvalues $x$ and $p$.
So we have, $$X \mid x \rangle = x \mid x \rangle, \qquad P \mid p \rangle = p \mid p \rangle,$$
and any state $ \mid \psi \rangle $ can be written as a linear combination of these eigenvectors,
$$ \mid \psi \rangle = \sum_x \psi(x) \mid x \rangle \quad (\text{discrete spectrum }) $$
$$ \mid \psi \rangle = \int dx \quad \psi(x) \mid x \rangle \quad (\text{continuous spectrum }) $$
Where $\psi(x)$ are the coefficients of the $\mid x \rangle$'s in the expansion of the state $\mid \psi \rangle$. Using the orthonormality of the basis vectors we can conclude that $\psi(x) = \langle x \mid \psi \rangle$ this function of the eigenvalues is usually what we call the wavefunction. Since there is a $1-1$ correspondence between wave functions and the state vectors they represent we often become sloppy and refer to them as if they are the same thing.
Now a reasonable question to ask is what is the wave functions that correspond to the eigenstates of $P$? I'm not going to derive it here but it is possible to show (starting from the canonical commutation relation) that,
$$ \langle x \mid P \psi \rangle = \frac{\hbar }{i} \frac{\partial}{\partial x} \langle x \mid \psi \rangle = \frac{\hbar}{i} \frac{\partial \psi(x)}{\partial x}$$
Using this it is easy to show that the wave function for $\mid p \rangle$ is, $$ p(x) = \langle x \mid p \rangle = \frac{1}{\sqrt{2\pi \hbar}} e^{ixp/\hbar}$$
This gives us a way of converting wave functions in the $x$-basis to wave functions in the $p$ basis. Starting with the projection of $\mid \psi \rangle$ onto the $p$-basis we expand $\mid \psi \rangle$ in the $x$-basis and perform the integration,
$$ \psi(p) = \langle p \mid \psi \rangle = \int dx \quad \langle p \mid x \rangle \langle x \mid \psi \rangle = \frac{1}{\sqrt{2\pi\hbar}} \int dx \quad e^{-ixp/\hbar} \psi(x) $$
The last line above is obviously the fourier transform of $\psi(x)$.
To answer a question in the comments,
The form of the states $\mid x \rangle $ depends on the basis you represent them in. If I were to represent these states in the $p$-basis they would look like,
$$ x(p) = \langle p \mid x \rangle = \frac{1}{\sqrt{2\pi\hbar}} e^{-ixp/\hbar} $$
If I were to represent them in their own basis I would get dirac delta functions,
$$ x'(x) = \langle x \mid x' \rangle = \delta(x-x') $$
Similarly if I expand the $p$'s in their own basis I would get a delta function,
$$ p'(p) = \langle p \mid p' \rangle = \delta(p-p')$$
Think of the wave functions as the components of a good old arrow vector. You can get different components by using different basis vectors but at the end of the day the vector itself is unchanged. Similarly we can get different looking wave functions by looking at their expansions in different basis sets but at the end of the day they all correspond to the same state vector.