Let $N \in \mathbb{N}$ such that $\phi(N) \sim N$, where $\phi$ is the Euler's totient function. Let $A \subset [N] := \{1, 2, \ldots, N\}$. For $n \in \mathbb{N}$ define the function $$ C_A(n) = \#\{ k \in [N] \mid (k, n) = 1 \text{ and } k \in A\}. $$ In particular $C_A(1) = \#A$. Assuming that $\#A \sim N$, can we give a good lower bound to the sum $$ S_A(N) = \sum_{d \mid N} C_A(d) $$ in terms of $\#A$ and $N$?
Note that $C_A(n) \leq C_A(d)$ whenever $d \mid n$; hence we can get a lower bound $$ S_A(N) \geq \#A + (\tau(N)-1) C_A(N), $$ where $\tau(N)$ is the number of distinct divisors of $N$ (pull out the term with $d = 1$ and use the inequality on the remaining terms). Since $\#A \sim N$ and $\phi(N) \sim N$, it is to be expected that $C_A(N) > 0$. Nevertheless this bound seems crude to me. I think there should be something better (preferably only in terms of $\#A$ and $N$ if this is possible).
Update:
Greg Martin's answer made me realize that things could be generalized a little.
Proposition: Let $N \in \mathbb{N}$ and let $A \subseteq [N] := \{1, \ldots, N\}$. Let $D$ be a subset of the positive divisors of $N$. Write $\#A = \delta(N)N$ and $\phi(N) = r(N)N$ for some functions $\delta, r : \mathbb{N} \to (0, 1]$. If $\delta(N) + r(N) > 1$, then $$ \sum_{d \in D} C_A(d) > N\sum_{d \in D} \dfrac{\phi(d)}{d} - \#D \phi(N). $$ In particular, if $\delta(N) \sim \delta$ is a constant and $r(N) \sim 1$, then the above inequality holds for all such $N$ large enough.
Proof: Observe that \begin{align*} C_A(d) &= N \dfrac{\phi(d)}{d} - \#\{\text{integers coprime to $d$ but not in $A$} \}\\ & \geq N\dfrac{\phi(d)}{d} - \#\{\text{integers in } [N] \text{ not in } A\}\\ &= N\dfrac{\phi(d)}{d} - (N - \#A). \end{align*} Thus \begin{align*} \sum_{d \in D} C_A(d) \geq N \sum_{d \in D} \dfrac{\phi(d)}{d} - \#D(N - \#A). \end{align*} Now the result follows by noticing that $$ \#D(N - \#A) = \#DN(1 - \delta(N)) < \#DNr(N) = \#D\phi(N). $$ QED
Question: When can we guarantee that $C_A(N) > 0$? In the case that $r(N) \sim r$ and $\delta(N) \sim \delta$ are constants such that $r + \delta \leq 1$, can we get a similar result? That is can we show there exists a subset $D$ of divisors of $N$ for which the inequality of the proposition is satisfied? Note that in the proof I merely used the fact that $C_A(d) \geq N\dfrac{\phi(d)}{d} - (N - \#A)$ on every single divisor $d \in D$. This seems crude to me and I think could be improved.