Let $A\in\mathbb{R}^{n\times n}$ and define conjugation by $GL_n$ on $\mathbb{R}^{n\times n}$ in the usual way (e.g. for all $A\in\mathbb{R}^{n\times n}$ and $T\in GL_n$, $A\mapsto T^{-1}AT$). Are there any ways to derive conditions on $T$ so that a given subset of the entries of $A$ will be invariant under conjugation?
For example, let $$ A = \begin{bmatrix}a & b\\ c & d \end{bmatrix} $$ and suppose we want $b$ to be invariant under conjugation. Then we are looking for the subset $U \subset GL_n$ such that, for all $T \in U$, $$ T^{-1}AT = \begin{bmatrix}a' & b \\ c' & d' \end{bmatrix}. $$ Due to the application from which this question arises, it is also enough to be able say whether or not $U$ consists of just the identity matrix.
I would also appreciate any pointers to relevant literature.
Let $T=\begin{pmatrix} t_1 & t_2\cr t_3 & t_4\end{pmatrix}$ with $\det(T)=1$. Then $T^{-1}AT$ has upper right corner equal to $b$ if and only if $$ at_2t_4 + bt_4^2 - ct_2^2 - dt_2t_4=b. $$ This follows by a direct computation. Suppose that $c\neq 0$. Then we can solve the quadratic equation for $t_2$ over the complex numbers. This will give, in general, two non-zero solutions, hence there are more than just the multiples of the identity fixing $b$ under conjugation.