It is known that $$\frac{\partial\log|A|}{\partial A}=A^{-T}$$
However, if $L$ is a lower triangular positive definite matrix and take the log determinant, $\log |L|=\sum_i\log L_{ii}$.
Question is considering that the off diagonal elements of $L$ do not affect the determinant at all why would $$\frac{\partial\log|L|}{\partial L}=L^{-T}$$ $L^{-T}$ has off diagonal terms that are non-zero.
Am I somehow looking at the definition of matrix calculus wrong?
Yes, the problem is with interpretation. If we want to look at the partial derivative of the logdet with respect to a single element $(i,j)$ of $L$, we do this: $$\frac{\partial \log \det L}{\partial L_{ij}} = \left\langle \frac{d \log\det L}{d L}, E_{ij} \right\rangle = \langle L^{-T}, E_{ij} \rangle = \left(L^{-T}\right)_{ij} = \left(L^{-1}\right)_{ji}$$ ($E_{ij}$ is the matrix with a $1$ in the $(i,j)$ position and zeros everywhere else.) When $i>j$, $(L^{-1})_{ji}=0$. So just as expected, perturbations in the strict lower triangle do not change the determinant.
But perturbations in the upper triangle of $L$ certainly do change $\log\det L$! When $i<j$, we expect the partial derivative to be non-zero, typically.
And that is why the matrix derivative involves the entire matrix $L$, not just its diagonal. The vector space over which you are defining this function is the space of all $n\times n$ matrices, including those that are not lower triangular.
If you wanted to, you could define a function just on the vector space of lower triangular matrices: $$f_{LT}:\mathbb{R}^{n\times n}_{LT}\rightarrow\mathbb{R}, \quad f(L) = \log \det L$$ If you do this, then the derivative has to be a lower triangular matrix as well. And since we the know that the strict lower trianglular portion of the derivative is zero, you get a diagonal derivative: $$\frac{d f_{LT}(L)}{d L} = \mathop{\textrm{diag}}(L_{11}^{-1},L_{22}^{-1},\dots,L_{nn}^{-1})$$