Having a function f which is neither convex nor coercive. What is the correct nominalization for these properties? I suppose something like
non-convexity and non-coercivity of f
Additionally, is it correct to say
f is non-coercive
f is non-convex
Maybe this is a question for MathOverflow...
Disclaimer: I'm not a native speaker of English
Yes, these are frequently used: "The non-convexity of $W(\nabla u)$ implies...", "in spite of the non-coercivity of $A$...", "due to non-coercivity of the Hamiltonian..." Google Scholar search brings up these and many others.
The combination of both may be appropriate in the introductory part: "additional challenges are presented by the non-convexity and non-coercivity of $f$". I would not use it in the course of a proof, because it packs too much mathematics into one sentence. The sentence "... fails due to the non-convexity and non-coercivity of $f$" is likely to be improved by separating it in a way that shows the role of those two factors.
You can say that, but I see no advantages compared to "$f$ is not coercive", "$f$ is not convex", "$f$ is neither coercive nor convex". These versions sound simpler and avoid any dispute about "non-convex" vs "nonconvex" (the journal you submit to may have a different preference from yours). It also handles line breaks better.