I asked this question as a mathematica question: https://mathematica.stackexchange.com/questions/41689/finding-a-certain-invariant-polynomial-using-matrix-coordinates
but maybe it will get more attention on this site. Let
$a = \begin{pmatrix} a11 & a12 & a13 \\ 0 & a22 & a23 \\ 0 & 0 & a33 \\ \end{pmatrix},$ $c = \begin{pmatrix} c11 & c12 & c13 \\ 0 & c22 & c23 \\ 0 & 0 & c33 \\ \end{pmatrix},$ $r = \begin{pmatrix} r11 & r12 & r13 \\ 0 & r22 & r23 \\ 0 & 0 & r33 \\ \end{pmatrix},$ $s = \begin{pmatrix} s11 & s12 & s13 \\ 0 & s22 & s23 \\ 0 & 0 & s33 \\ \end{pmatrix},$ $u = \begin{pmatrix} 1 & u12 & u13 \\ 0 & 1 & u23 \\ 0 & 0 & 1 \\ \end{pmatrix}$
be $3\times 3$ matrices. Consider the conjugation of $a,c,r,s$ by $u$:
$uau^{-1}= \small\begin{pmatrix} a11 & a12 + (a22 - a11 )u12 & a13 + a23 u12 - a12 u23 + (a33 - a11 )u13 + (a11- a22 )u12 u23\\ 0 & a22 & a23 + (a33- a22 )u23 \\ 0 & 0 & a33 \\ \end{pmatrix},$
$ucu^{-1}= \small\begin{pmatrix} c11 & c12 + (c22 - c11 )u12 & c13 + c23 u12 - c12 u23 + (c33 - c11 )u13 + (c11- c22 )u12 u23\\ 0 & c22 & c23 + (c33- c22 )u23 \\ 0 & 0 & c33 \\ \end{pmatrix},$
$uru^{-1}= \small\begin{pmatrix} r11 & r12 + (r22 - r11 )u12 & r13 + r23 u12 - r12 u23 + (r33 - r11 )u13 + (r11- r22 )u12 u23\\ 0 & r22 & r23 + (r33- r22 )u23 \\ 0 & 0 & r33 \\ \end{pmatrix},$
$usu^{-1}= \small\begin{pmatrix} s11 & s12 + (s22 - s11 )u12 & s13 + s23 u12 - s12 u23 + (s33 - s11 )u13 + (s11- s22 )u12 u23\\ 0 & s22 & s23 + (s33- s22 )u23 \\ 0 & 0 & s33 \\ \end{pmatrix}.$
I would like to find an $U$-invariant polynomial $f$ in the coordinates of $a,c,r,s$ (not necessarily all) such that at least one of the monomials of $f$ is divisible by the $(1,3)$-entry of $a,c,r$, or $s$.
$\textbf{Example:}$ $(a11-a22)c12-(c11-c22)a12$ is an invariant but none of the monomials are divisible by $a13$ or $c13$.
Here's another homogeneous degree $2$ polynomial $(c11 - c33) (a13) - (a11 - a33) (c13) + c12 a23 - a12 c23$ but it is not quite an invariant because under the group action, it becomes $a13 c11 + a23 c12 - a11 c13 + a33 c13 - a12 c23 - a13 c33 + (a23 (c22 - c33) + (a33 - a22 )c23 ) u12 + (a12 (c22-c11 )+ (a11 - a22 )c12 ) u23.$