Let $\Sigma$ be some finite set of elements. A function $f$ is a mapping: $f: \Sigma\rightarrow\Sigma$. Let also define that elements of $\Sigma$ has some boolean property $c(x)\in\{T,F\}$.
Consider the set $X$ which is constructed recursively, using disjunction of two conditions: or $c(x)=T$ or $f(x)\in X$. The latter condition means that inclusion of some element $x$ depends on the current state of $X$.
Will it be mathematically correct to describe this set in the following form?
$X=\{x\in\Sigma | c(x)=T\text{ or }f(x)\in X\}$
I doubt that it is rigorous definition. For instance, it is probably needed to define an order in which elements of $\Sigma$ are considered for adding into $X$. Will it be enough strict in this case or it should be defined in some different way?
In this particular case you can probably get away with defining your set as $$ X = \{ x\in\Sigma \mid \exists n\ge 0: c(f^n(x)) = T \} $$ But that works only because your recursive definition has a very particular form.
For general recursive definitions of this kind, I can say with some certainty that there isn't a generally understood way to write them that is as short and convenient as set builder notation. I did a PhD and 3 years of post-doc work in an area of computer science where such definitions are our bread-and-butter and I never saw one. What we would usually do was write an inference system:
and expect the reader to know how to interpret that rigorously. This can usually be assumed when you write for computer scientists, but not with a general mathematical audience.
You can say
where what you wrote is now presented as a condition rather than a definition. However, this depends on the reader being able to convince himself on his own that there is indeed a smallest subset of $\Sigma$ that satisfies the condition.
If that won't fly with your audience you'll need start by defining a helper function $\Phi:\mathcal P(\Sigma)\to\mathcal P(\Sigma)$:
and then explicitly iterate it: