I've been asked to compute the integral of $f(x, y, z)= 1 - x^2 - y^2 - z^2$ over the surface of the plane $x + y + z = t$ cut off by the sphere $x^2 + y^2 + z^2 = 1$ for $t \leq \sqrt3$ and prove it is equal to $\frac{\pi}{18}(3-t^2)^2$.
A hint is provided to introduce a new coordinate system $(x_1, y_1, z_1)$ where $z_1$ is normal to the plane and use polar coordinates in the $x_1 y_1$ plane to parametrize the surface as a circle.
I started to do this, letting $z_1 = \frac{1}{\sqrt3}(1,1,1)$, choosing $x_1 = \frac{1}{\sqrt2}(1, 0, 1)$ since it is orthogonal to $z_1$, and letting $y_1 = (\frac{-1}{\sqrt6},\sqrt\frac{2}{3}, \frac{-1}{\sqrt6})$ (their cross product).
Then I tried to take the surface integral: $$ \int\int_S{f(x_1,y_1,z_1) dS}=\int\int_S(1-x_1^2-y_1^2-(t-x_1-y_1)^2)dS $$
Then by the polar parametrization $g(r, \theta)=(r\cos\theta, r\sin\theta, t-r\cos\theta-r\sin\theta)$, this becomes $$\int_0^{2\pi}{\int_0^1{(1-2r^2-t^2+2rt(\sin\theta+\cos\theta)-2r^2\sin\theta\cos\theta)\sqrt3r}dr}d\theta$$
This integral does not evaluate to anything resembling what I'm supposed to prove it does. Where am I going wrong? I feel it has something to do with switching coordinate systems, but I'm not exactly sure where I've made my mistake.
Your ideas are good, but execution is failing right away. I think you mean to say that the new coordinates are related to the old by an orthogonal transformation $$\begin{bmatrix}x_1\\y_1\\z_1\end{bmatrix}=\begin{bmatrix}\frac1{\sqrt2}&-\frac1{\sqrt2}&0\\ \frac1{\sqrt6}&\frac1{\sqrt6}&-\frac2{\sqrt6}\\ \frac1{\sqrt3}&\frac1{\sqrt3}&\frac1{\sqrt3}\end{bmatrix}\begin{bmatrix}x\\y\\z\end{bmatrix}$$ Note that your $x_1$ vector was not orthogonal to your $z_1$ vector. Typo? Anyhow, because this transformation is orthogonal it has two advantages: the first two rows of the transformation are orthogonal to each other and normalized to unity so the areal element will take on the simple form $d^2A=dx_1dy_1$ and since it preserves lengths, $$x_1^2+y_1^2+z_1^2=\frac12(x-y)^2+\frac16(x+y-2z)^2+\frac13(x+y+z)^2=x^2+y^2+z^2$$ So now you are working on the plane $z_1=\frac1{\sqrt3}(x+y+z)=\frac t{\sqrt3}$ and inside the circle $x_1^2+y_1^2+z_1^2=x_1^2+y_1^2+\frac{t^2}3=x^2+y^2+z^2=1$. Your transformed function is $f(x_1,y_1,z_1)=1-x^2-y^2-z^2=1-x_1^2-y_1^2-z_1^2=1-x_1^2-y_1^2-\frac{t^2}3$. So now we can transform to polar coordinates $x_1=r\cos\theta$, $y_1=r\sin\theta$ and we have $$\begin{align}\int\int_Sf(x_1,y_1,z_1)d^2A&=\int_0^{2\pi}\int_0^{\sqrt{1-\frac{t^2}3}}\left(1-r^2-\frac{t^2}3\right)r\,dr\,d\theta\\ &=2\pi\left(1-\frac{t^2}3\right)^2\left(\frac12-\frac14\right)\\ &=\frac{\pi}{18}(3-t^2)^2\end{align}$$ Just like the book said.