Consider the ill-posed system $$ \mathbf A \mathbf x= \mathbf b.$$
One method to regularize the solution is the Tikhonov method which effectively minimizes $ ||\mathbf A \mathbf x - \mathbf b ||^2 + || \mathbf \Gamma \mathbf x||^2$.
Letting the Tikhonov matrix $\mathbf \Gamma = \lambda \mathbf I$ favours solutions with smaller norms, where $\mathbf I$ is the identity matrix, and the parameter $\lambda$ is chosen empirically of a given system. Singular value decomposition ($\mathbf A = \mathbf{U \Sigma V}^H$) may then be used to calculate the solution via $\mathbf x = \mathbf{V \hat{\Sigma} U}^H \mathbf b$, where
$$\hat{\Sigma}_{ii} = \frac{\Sigma_{ii}}{\Sigma_{ii}^2 + \lambda^2}.$$
My question is how to find $\mathbf x$ which instead favours low frequency solutions?
i.e. I wish to minimize something like $ ||\mathbf A \mathbf x - \mathbf b ||^2 + \lambda^2 \sum_{high \, frequencies} FFT \{ \mathbf x \}^2$.
Well, it seems like you said how to do it: Let $ T $ be the discrete fourier transform matrix, ie. something like: $$ T_{a,b} = e^{\frac{2 \pi i}{N} ab} $$ or whatever is appropriate for your situation. Then let $ P $ be the projection on to high frequency modes, or more generally something like $ P = \text{diag}(0,0,\cdots,w_{n-2},w_{n-1},w_{n}) $. For some weights $ w_i $ you can choose. Then let $ \Gamma = PT $.
Maybe there is some natural choice for the weights. For example, it might be computationally easier if you take $ \Gamma $ to be some finite difference matrix, eg: $$ \Gamma = \begin{pmatrix} -1 & 1 & 0 & \cdots \\ 0 & -1 & 1 & \cdots \\ && \ddots & \end{pmatrix} $$