This is kind of vague question, but I'll try to make it more precise.
$T$ is a compact operator on a Hilbert Space, $H$, if $\overline{T(D)}$ is compact in $H$, where of course, $D$ is the closed unit ball in $H$. So we use the underlying Hilbert space in order to define these operators. Now $B(H)$ is a $C^*$-algebra. Can we abstractly define/characterize compact operators only in terms of $C^*$-notions, without appealing to the underlying Hilbert space? Stated otherwise, is it possible to define, say "compact" elements in a $C^*$-algebra $A$, which agree with the usual notion if we take $A=B(H)$?
I am not familiar with such a concept, but you could call a projection $p$ in a $C^*$-algebra $A$ finite if $pAp$ is finite-dimensional; and then consider the closed ideal generated by the finite projections.
I don't know if this is useful, however.