I'm having a really hard time with question b of the image below, (Find any non-trivial A and B such that they are orthogonal) and question c, the proof.
I know that non trivial means a nonzero solution, but how do I interpret the subscript? And does part c utilize Cauchy Schwarz? It looks almost like the pythagorean theorem but I'm not sure if I'm way off.
Any help or guidance you can provide would be super helpful, thanks! 
On
You can map each of this matrix to a vector of size 3. In each matrix there is a a zero entry. Ignore that zero and take the other three entries as components of a vector )that is the x,y,z co-ordinates)..
After this process it is clear that this question is not really about matrices, it is the same question as for $\mathbf{R}^3$.
If you have written computer programs the above process can be viewed this way. In a program we use, 2d-arrays, or 3d-arrays etc. A 2d-array can be scanned row by row and mapped to a 1-d array (that is what we have done).
For b) we need only find $A$ and $B$ non-zero so that the inner product is $0$, i.e., $$ \langle A, B \rangle = \left\langle \begin{bmatrix} a_{11} & a_{12}\\ 0 & a_{22} \end{bmatrix}, \begin{bmatrix} b_{11} & b_{12}\\ 0 & b_{22} \end{bmatrix}\right\rangle = a_{11}b_{11} + a_{12}b_{12} + a_{22}b_{22} = 0$$ So one easy example would be taking $a_{11}=a_{12}=b_{22}=0$. This would give a sum of zeros, even if $A$ or $B$ were non-trivial. For instance,
$$ A= \begin{bmatrix} 0 & 0\\ 0 & 1 \end{bmatrix}, \quad B= \begin{bmatrix} 3 & -17\\ 0 & 0 \end{bmatrix},\qquad \text{then $\langle A,B\rangle = 0$}$$
As for c), you are correct in recognizing the Cauchy-Schwartz inequality. As we have an inner product, it must satisfy $$\langle A, B\rangle^2 \leq \langle A, A\rangle \langle B,B\rangle$$ where $$\langle A, B\rangle^2 = (a_{11}b_{11}+a_{12}b_{12}+a_{22}b_{22})^2$$ $$\langle A, A\rangle \langle B,B\rangle = (a_{11}a_{11}+a_{12}a_{12}+a_{22}a_{22})(b_{11}b_{11}+b_{12}b_{12}+b_{22}b_{22}) = (a_{11}^2+a_{12}^2+a_{22}^2)(b_{11}^2+b_{12}^2+b_{22}^2)$$