I understand that for every action of group $G$ on a set $X$, there is a homomorphism: $$G\rightarrow S_X$$ It seems to me that this can be used to rule out many possible actions. For example, a group of order 11 could not act on a set of cardinality 3, because then the kernel of the homomorphism would have to be of order $|\{e\}|+(|G|-|S_X|)=1+11-3!=1+11-6=6$. 6 does not divide 11, violating LaGrange's theorem.
Is this correct?
Firstly, to answer the question in the title "Can kernel of homomorphism tell you when a group action cannot be constructed?" No, its isomorphism class cannot: For example, $S_X\times H$ acts on the set $X$ with kernel $H$, no matter the group $H$.
Now, to address the rule you give: you have a slight error (aside from the missing non-triviality assumption). For example, $\mathbb{Z}_{12}$, the cyclic group of order $12$, acts on the set of three elements by permuting two of them (and leaving the third fixed). However, $1+12-6=7$ and $7$ does not divide $12$$^{\dagger}$.
I believe that the condition you are after is the following:
From this, you have the following necessary condition:
In my example, there is a homomorphism $\mathbb{Z}_{12}\twoheadrightarrow\mathbb{Z}_2$ and $\mathbb{Z}_2$ embeds into $S_3$, while in your example $\mathbb{Z}_{11}$ is simple and cannot embed into $S_3$, hence it cannot act non-trivially on the set with three elements (note that $11$ and $3$ are coprime).
Note that the necessary condition is not sufficient. For example, $A_5$, the alternating group of order $60$, is simple and so if it acts non-trivially on a set then the kernel of the action must be trivial. This means that if $A_5$ acts on $X$ then $A_5$ embeds into $S_X$. Then, $4$ is not coprime to $60$, but $A_5$ cannot embed into $S_4$ (a group of order $24$), and hence any action of $A_5$ on a set of order less than $5$ is trivial. The same is true for any $A_n$, $n\geq5$, acting on sets of order less than $n$.
$^{\dagger}$The "reason" that the formula does not work is because actions need a "multiplicative" formula, because they are related to $G/N$, but the given formula is additive. One way to exploit this additivity to get a counter-example would be to take larger and larger sets $X$ for fixed $G$, and see that the formula gives a negative number. Even if $|G|>|X|$, negative numbers are possible, because $|X|!\gg|X|$.