Let $G$ be an algebraic, $B$ Borel subgroup, and $U$ unipoent subgroup of $G$. For example, we take $G=GL_n$, $B$ the subgroup of lower triangular matrices, and $U$ unipoent upper triangular matrices. Let $T$ be the Cartan subgroup consisting of all diagonal matrices in $G$. I think that the following formulas define a $B$-action on $U$. For $t \in T$, we define $t(u) = t u t^{-1}$. For $u' \in U$, we define $u'(u) = u'u$. Is this correct?
I think that the following formulas also define a $B$-action on $U$. \begin{align} t(u) = t u t^{-1}, t\in T, u\in U, \\ u'(u) = u u', u, u' \in U. \end{align}
I think that the following formulas don't define a $B$-action on $U$. \begin{align} t(u) = t^{-1} u t, t\in T, u\in U, \\ u'(u) = u u', u, u' \in U. \end{align} Is this correct? Thank you very much.
In your example you let $B$ be lower triangular matrices and $U$ be strict upper triangular matrices. That means you don't have $B = TU$ so defining how elements of $T$ and elements of $U$ act on $U$ is not enough information to tell you how elements of $B$ act on $U$.
Suppose instead you take $U$ to be the unipotent elements in $B$ and $T$ the torus in $B$ so that $B = TU$. Then every element of $B$ can be written as a product of an element of $U$ and an element of $T$. Letting $T$ act on $U$ by conjugation and letting $U$ act on $U$ by left multiplication defines actions of $T$ and $U$ on $U$. You still have to check that the two actions are compatible, but lucky you they are compatible and extend to an action of $B$ on $U$.
Note that by action I mean left action, so as you've gleaned right multiplication does not define a left action of $U$ on itself, it defines a right action. Similarly $t\cdot u = t^{-1}ut$ is not a left action, it's a right action.