I used the first formula to see if I would get the same answer as the second formula:
$$ \vec{r}'(t) = 2t\hat{i}+\hat{k} $$
$$ ||\vec{r}'(t)|| = \sqrt{4t^2+1} $$
$$ \vec{T}(t) = \frac{\vec{r}'(t)}{||\vec{r}'(t)||} = \frac{2t\hat{i}+\hat{k}}{\sqrt{4t^2+1}} $$
Therefore,
$$ \vec{T}'(t) = \frac{2\hat{i}(\sqrt{4t^2+1}) - 4t(\frac{2t\hat{i}+\hat{k}}{\sqrt{4t^2+1}}) }{4t^2+1} $$
Since the unit vector in the direction of a given vector is the same after multiplying by a positive scalar, I can simplify by multiplying $$ (4t^2+1)\;\;\; \sqrt{4t^2+1} $$
The first factor gets rid of the denominator and the second gets rid of fractional powers
$$ = 2\hat{i}(4t^2+1) - 4t(2t\hat{i}+\hat{k}) $$
$$ = 8t^2\hat{i}+2\hat{i} - 8t^2\hat{i} - 4t\hat{k}$$
$$ = 2\hat{i}-4t\hat{k} $$
$$ = \hat{i} - 2t\hat{k} $$
Thus, $$ \vec{T}'(t) = \hat{i} - 2t\hat{k} $$
and consequently,
$$ ||\vec{T}'(t)|| = \sqrt{1 + 4t^2} $$
So,
$$ \kappa = \frac{\sqrt{1 + 4t^2}}{\sqrt{4t^2+1}} = 1 $$
although this clearly does not make sense as the graph of the vector function is a parabola so there will be some point t that will be steeper than another point t2.
if my math is wrong i'd like to know where I made an error.
Otherwise, can someone please help explain how to decide between the formulas when they produce different results?


This statement:
is incorrect. Clearly, multiplying a vector by a scalar changes its length (hence it is called a scalar...it scales the length), and the length of $\vec T'(t)$ in this case is essential since it is part of the definition of curvature. If you must scale anything, you have to multiply both $\vec T'(t)$ and $\vec r'(t)$ by the same scalar to preserve the ratio of their lengths.
Thus, using the first definition, the correct computation is
$$\kappa=\frac{\|\vec T'(t)\|}{\|\vec r'(t)\|}=\frac{\|(4t^2+1)^{3/2}\vec T'(t)\|}{(4t^2+1)^{3/2}\|\vec r'(t)\|}=\frac{\|2(\hat i-2t\hat k)\|}{(4t^2+1)^{3/2}\sqrt{4t^2+1}}=\frac{2}{(4t^2+1)^{3/2}},$$
and all is right with the world.
Tip 1: The first formula may be less tractable when the unit tangent vector $\vec T(t)$ is a bit ugly, which happens in this problem since $\|\vec r'(t)\|$ is nonconstant, and hence requires using the quotient rule to compute $\vec T'(t)$.
Tip 2: The first formula essentially computes $\|\frac{d\vec T}{ds}\|$ where $s$ is arc length. So if the parameter $t$ represents arc length (e.g. $\frac{ds}{dt}=\|\vec r'(t)\|$ is constant), the first formula may then be more wieldy.