Sometimes I see the normal distribution written as $N(\mu,\sigma)$ sometimes as $N(\mu,\sigma^2)$ and sometimes, when the Central Limit Theorem is involved, as $N(\mu,\sigma^2/n)$
As a software developer I would not write a function that overloads the meaning of a parameter. It looks to me as if that is what is being done here.
How should I be thinking of the 2nd parameter here?
Are we specifying how the variance is to be created in a new normal distribution out of parameters of an old distribution?
As a software developer you have totally nailed it, in that the second parameter is overloaded, and that there is a
$$N( \text{mean}, \text{std_dev}) $$ constructor and a $$N( \text{mean}, \text{variance}) $$ constructor.
The mathematician parser is generally smart enough to be able to distinguish between the two when it encounters "$N(\cdot, \cdot)$". Sometimes it can tell from how it is written (i.e. $N(\mu,\sigma)$ versus $N(\mu,\sigma^2)$, other times it can tell from the units, e.g. if the mean is in $cm$, then it looks at whether the second parameter has units of $cm$ or $cm^2$. If it cannot disambiguate it throws an error and demands that the writer write it more clearly.
Your observation about the CLT, and the use of $N(\mu,\sigma^2/n)$, however, points out that mathematicians can be pretty lazy about notation, relying on context, which can be confusing to the beginner, who may not know all of the same assumed context. And statisticians are even worse than your average mathematician in this regard. (I've taught stats, and have found myself apologizing to students for this.)
In the context of the CLT there is an underlying random variable (call it $X$), with mean $\mu$ and standard deviation $\sigma$. This underlying random variable may or may not be normally distributed, and so $\mu$ is shorthand for $E[X]$, and similarly for $\sigma$.
In the CLT we construct a new random variable, the "sample mean (from $n$ samples)", and call it $\overline{X}$. We want to compare its distribution with a particular Normal Distribution, $N(\mu,\sigma^2/n)$. The content of the CLT is that $\overline{X}$ looks more and more like $N(\mu,\sigma^2/n)$ as $n$ gets larger.
So yes, you are right, in the CLT "we [are] specifying how the variance is to be created in a new normal distribution out of parameters of an old distribution". The only thing to add is that there is an easy CLT, where the original distribution is normal, and a more general CLT, where the original distribution is fairly arbitrary (within some broad limits). In the easy CLT, $\mu$ and $\sigma$ are the parameters of the original distribution, but in the more general version they are "computed properties" of the distribution. For example, if the original distribution is Poisson (with parameter $\lambda$), then the CLT holds, but with $\mu = \lambda$, and $\sigma = \sqrt \lambda$.