If $T=\begin{bmatrix}A & C\\0 & B\end{bmatrix}$ is upper triangular with $A$ and $B$ upper triangular as well ($C$ is an arbitrary matrix), then what must $R$ be so that:
$$ S^{-1}TS = \begin{bmatrix}A & 0\\0 & B\end{bmatrix} \,, $$ where $S = \begin{bmatrix}I & R\\0 & I\end{bmatrix}$?
My attempt
I tried to compute the equivalent statement $TS = S\begin{bmatrix}A & 0\\0 & B\end{bmatrix}$ first, which gave me:
$$ \begin{gather*} \begin{bmatrix}A & C\\0 & B\end{bmatrix}\begin{bmatrix}I & R\\0 & I\end{bmatrix} = \begin{bmatrix}I & R\\0 & I\end{bmatrix}\begin{bmatrix}A & 0\\0 & B\end{bmatrix} \\ \begin{bmatrix}A & AR+C\\0 & B\end{bmatrix} = \begin{bmatrix}A & RB\\0 & B\end{bmatrix} \\ \Rightarrow AR+C = RB \end{gather*} $$ then I'm stuck. How do I extract matrix $R$ from this?
I think there's probably a need to use the fact that $A$ and $B$ are upper triangular, but I don't know how. Note that this is the question as posed, I do not have the dimensions of any of the matrices.
The fact that $A,B$ are triangular is not important.
We assume that $A\in M_p, spectrum(A)=(\lambda_i)_{i\leq p},B\in M_q,spectrum(B)=(\mu_i)_{i\leq q}$.
The equation in $R\in M_{p,q}$ to solve is $f(R)=RB-AR=C$; $f$ is a linear function s.t. $spectrum(f)=(-\lambda_i+\mu_j)_{i,j}$. Thus $f$ is invertible IFF for every $(i,j),$ $\lambda_i\not= \mu_j$; under these conditions, $R$ exists and is unique.
Conclusion. If $A,B$ have no common eigenvalues, then we can block-wise diagonalize $T$.
EDIT. If $g,h\in L(M_{p,q})$ are defined by $g(R)=RB,h(R)=AR$, then $f=g-h$; according to 1. and 2.2.1 in
https://en.wikipedia.org/wiki/Kronecker_product#Abstract_properties
$spectrum(g)=(spectrum(B))^p,spectrum(h)=(spectrum(A))^q\subset \mathbb{R}^{pq}$; since $g,h$ commute, they are simultaneously triangularizable and $spectrum(f)$ is as above given. If $f$ has no zero eigenvalues, then it is bijective.
There is no closed formula for $R$; yet, to obtain $R$ it suffices to solve a linear system, that is elementary. Note that if $f$ is not bijective and $C$ is randomly chosen, then $R$ does not exist.