Is it possible to define a tensor Singular Value Decomposition (SVD)?
For example for the 3 tensor $$\left[\begin{array}{rr}1&-1\\1&1\\\hline 1&-1\\-1&-1\end{array}\right]$$
Can be written as a linear combination (in octave code)
[1,-1].*[1;0].*cat(3,1,1) + [1,1].*[0;1].*cat(3,1,-1)
The rank 1 tensors which we make outer product between are pairwise orthogonal in each direction [1,-1] against [1,1], [1;0] against [0;1] and 1,1 against 1,-1 in the third dimension.
Is this a reasonable generalization of the SVD in some sense or am I thinking wrongly?
What you have found is a tensor rank decomposition, which is something that the SVD gives you for $2$-tensors. In this way, your decomposition can indeed be thought of as a generalization of the SVD. In fact, what you get is the same thing that you would get by applying the "higher order SVD" algorithm. See the interpretation section of that second page for details.