I need help in interpreting rigorously the following derivation from Jackson's Electrodynamics textbook: In particular I'm having trouble rigorously defining $\delta W$. I tried to interpret $W$ as a functional with $\rho$ as its argument but then I couldn't make sense as to how we integrate with respect to that variation (I'm familiar with taking variational derivatives not integrating with respect to variations (whatever that means)).
Yes, what he writes is a sloppy notation for the Frechet differential of a (typically nonlinear) functional $W$ defined on a vector space $V$. (The latter is a vector space of functions with some degree of smoothness and decay at inifnity and equipped with a suitable norm: In principle, one can do it all without norms, but then you have to introduce a formalism of topological vector spaces. Physicists usually do not worry about such details.) Now, let $\rho$ is a point in $V$ and $F: V\times V\to {\mathbb R}$ is a functional on $V$ (linear in the second variable). When you see a formula $$ \delta W = F(\rho, \delta \rho), $$ this means that the Frechet derivative of $W$ at $\rho$ in the direction of a vector $\delta\rho\in V$ equals $F(\rho, \delta \rho)$. (Implicitly, it is assumed or, at least, hoped, that the functional $F$ is differentiable at $\rho$.)
As for a justification of his computation: For such a computation to work, a mathematician would impose further assumptions on functions, such as compact support or fast decay at infinity, and make sure that the norm (topology) is chosen correctly, otherwise one has trouble interchanging limits (derivatives) and integration. Most physicists do not care about any of these and assume that formal computations will actually work.