This question is about a particular case of the special orthogonal groups considered in section 23.4 of Borel's Linear Algebraic Groups. Let $k$ be a field of characteristic $\neq 2$. Let $F$ be a nondegenerate isotropic symmetric bilinear form on $k^6$ of Witt index 2, so we can write the matrix of $F$ as $$ F = \begin{pmatrix} && I_2 \\ & F_o \\ I_2 \end{pmatrix} \qquad F_o =\begin{pmatrix} a_1 \\ & a_2 \end{pmatrix} \qquad a_1, a_2 \in k^\times $$ where $I_2$ is the $2 \times 2$ identity matrix, and $F_o$ is the anisotropic component of $F$. Let $G = \operatorname{SO}(F)$ be the associated special orthogonal group. $$ G = \{ X \in \operatorname{SL}_6(k) : X^t F X = F \} $$ $X^t$ denotes the matrix transpose. The Lie algebra of $G$ is $$ \mathfrak{g} = \{ A \in \mathfrak{gl}_6(k) : A F + FA^t = 0 \} $$ Let $S$ be the following maximal $k$-split torus in $G$ (note the rank of $S$ equals the Witt index of $F$, this is true in general). $$ S = \left\{ \begin{pmatrix} s_1 \\ & s_2 \\ && 1 \\ &&& 1 \\ &&&& s_1^{-1} \\ &&&&& s_2^{-1} \end{pmatrix} : s_1, s_2 \in k^\times \right\} $$ Let $y_i:S \to k^\times$ be the character which picks off the $i$th diagonal entry. The root space corresponding to the character $y_1$ is $$ \mathfrak{g}_{y_1} = \left\{ A(x_1, x_2) = \begin{pmatrix} 0 & & x_1 & x_2 \\ & 0 \\ && 0 && - a_1 x_1 \\ &&& 0 & -a_2 x_2 \\ &&&& 0 \\ &&&&& 0 \end{pmatrix} : x_1, x_2 \in k \right\} $$ As I understand it, it should be the case that the corresponding root subgroup of $G$ is the image of $\mathfrak{g}_{y_1}$ under the matrix exponential map. However, if $A \in \mathfrak{g}_{y_1}$, my calculations show that $\exp(A)$ need not lie in $G$ at all. In particular, taking $A = A(x_1, x_2)$, $$ \exp(A) = 1 + A + \frac 12 A^2 = \begin{pmatrix} 1 & & x_1 & x_2 & - \frac 12 (a_1 x_1^2 + a_2 x_2^2) \\ & 1 \\ && 1 && - a_1 x_1 \\ &&& 1 & -a_2 x_2 \\ &&&& 1 \\ &&&&& 1 \end{pmatrix} $$ The matrix $X = \exp(A)$ does not satisfy $X^t F X = F$. Concretely, the defect is $$ X^t F F - X = \begin{pmatrix} 0 \\ & 0 \\ && 0 && -x_1(a_1^2-1) \\ &&& 0 \\ && -x_1(a_1^2-1) && a_1 x_1^2( a_1^2 - 1) \\ &&&&& 0 \end{pmatrix} $$ Thus $X \in G$ if and only if $a_1^2 = 1$. But to the best of my knowledge, over a general field $k$ we cannot assume that the anisotropic nondegenerate form $F_o$ can be diagonalized with diagonal entries $\pm 1$. This is certainly possible if $k = \mathbb{R}$ (or slightly more generally), but in section 23.4 Borel makes no assumption about $k$ other than that the characteristic is not 2, so I don't think I should need additional assumptions either.
My question is: Why is it not the case that $\exp(A) \in G$? Have I just made a calculation error somewhere, or maybe I have misunderstood something about the relationship between an algebraic group and its Lie algebra.
This entire confusion is just due to some very minor transpose issues. I had missed the fact that Borel takes opposite conventions in where to put the matrix transpose in section 23.4 versus in 23.9. If $G = \operatorname{SO}(F)$ is defined as in the question, then the Lie algebra condition should be $A^t F + FA = 0$, not the slightly different $A F + FA^t = 0$. This small change means that the matrix $A = \exp(X)$ in the question does not actually belong to the Lie algebra.
An alternative fix is to keep the same Lie algebra condition, and replace $X^t FX = F$ in the definition of $G$ with the slightly different $XFX^t = F$. Then it does work out that $X = \exp(A)$ lies in $G$.