In Principia Mathematica's Introduction, there is a definition for "incomplete" symbol:
By an "incomplete" symbol we mean a symbol which is not supposed to have any meaning in isolation, but is only defined in certain contexts. -- Chapter III, Principia Mathematica, 1st edition, page 69.
I am not sure what "meaning" means in this definition. Judging by the passage that follows, I guess if a symbol has a "meaning," that symbol stands for an object, like "Socrates" stands for a certain man. As a matter of fact, in Russell's An Inquiry of Meaning and Truth, the meaning of an object-word is the thing it stands for. Please let me know if "meaning" in PM has the same definition as in An Inquiry of Meaning and Truth. Thanks.
Russell is developing his theory of descriptions here. Roughly, he takes an "incomplete symbol" to be one that does not refer -- one that does not have a denotation in the way that proper names do. So, for example, the expression "the author of Waverley" -- according to Russell's theory -- does not denote; you cannot ask what its referent is. (EDIT 2: Russell does use the notation $\psi(\iota x\phi (x))$, but this doesn't mean that $\iota x\phi(x)$ denotes an object that can serve as the argument of $\psi(y)$, because $\psi(\iota x\phi(x))$ is really shorthand for $\exists x(\phi(x)\land\forall y(\phi(y)\rightarrow y=x)\land \psi(x))$.)
So noun phrases like "the author of Waverley," or "the more famous author of Principia Mathematica," cannot be thought of as proper names; and since Russell thinks of "meaning" in terms of reference, the expression cannot be said to have a meaning "in isolation." (One reason Russell makes this argument is to help explain how to formalize language about objects that don't exist, e.g. "the present King of France is bald" or "the greatest prime number.") Only in a sentence like "Scott is the author of Waverley" does the expression acquire a referent: that is what he means by the meaning only being "defined in certain contexts." In short, some expressions that look like object-words or proper names -- i.e., expressions that look like they refer (to an object) -- actually turn out not to be, if you're careful about the logical analysis.
Whether or not you are a proponent of the Russellian theory of definite descriptions, the mathematical examples Russell gives at the beginning of chapter 3 should not be taken too seriously. He says, for example, that the symbol $\frac{d}{dx}$ should be thought of as an incomplete symbol -- that it should not be thought of as having a denotation or referent by itself, when it isn't "completed" by a function symbol: e.g $\frac{d}{dx}(x^2)$.
To a contemporary mathematician, though, this might sound silly. A contemporary mathematician would likely say that $\frac{d}{dx}$ denotes a certain operator on a function space (e.g. the space of smooth functions from $\mathbb{R}$ to $\mathbb{R}$). Of course, there may be some ambiguity about which function space is meant, but that doesn't pose much of a problem: it is usually clear from context which space of functions one is working over.