Let $N$ be a group. Suppose that $N$ has an action on a compact Hausdorff space $X$. Let $\phi: H \to \text{Aut}(N)$ be a non-trivial homomorphism. Let $G=N \rtimes_{\phi}H$. I want to define an action of $G$ on $X$. Let $g=(n,h) \in G$. I defined an action by $g.x=\phi_{h^{-1}}(n).x$.
Although it is well defined, the action fails to be associative: For $g_1=(n_1,h_1)$ and $g_2=(n_2,h_2)$, we have $g_1g_2.x=(n_1\phi_{h_1}(n_2),h_1h_2).x=\phi_{h_2^{-1}h_1^{-1}}(n_1\phi_{h_1}(n_2)).x=\phi_{h_2^{-1}h_1^{-1}}(n_1)\phi_{h_2^{-1}}(n_2).x$,where as $g_1.(g_2.x)=g_1.(\phi_{h_2^{-1}}(n_2).x)=\phi_{h_1^{-1}}(n_1)\phi_{h_2^{-1}}(n_2).x$
This seemed to be a natural way of defining it. How else should I define? Thanks for the help!!
Let us say I don't care about the action (actually I do, you take $K=Aut(X)$ and $f$ the structural morphism defining your action). Set $K$ to be some group and $f:N\to K$. What you want to do is to define a group morphism $g:N\rtimes_{\phi}H\to K$ so that $g_{|N}=f$. Assume $g$ exists. Let $h\in H$ then $g_{|H}$ is a group morphism I denote $s$.
$$f(n_1\phi(h_1)\cdot n_2)s(h_1h_2)=g(n_1\phi(h_1)\cdot n_2,h_1h_2)=g(n_1,h_1)g(n_2,h_2)=f(n_1)s(h_1)f(n_2)s(h_2)$$
Take in particular $h_2$ to be $1_H$ and $n_1=1_N$, then you end up with the condition for all $n_2\in N$ and $h_1\in H$:
$$f(\phi(h_1)\cdot n_2)=s(h_1)f(n_2)s(h_1)^{-1}$$
In other words you need to be able to find a group morphism $s$ such that the above always hold. I don't think it is always possible, unfortunately (but maybe you have some strong assumption on the action).
Edit: Here is a counter example. Consider $N=GL(2,\mathbb{C})=K$ and $f$ the identity automorphism. Let $H=\mathbb{Z}/2$ then $H$ acts on $N$ by complex conjugation. Then define $G:=N\rtimes_{\overline{\cdot}}H$. I claim that you cannot extend $f$ to $G$. If it were the case then any matrix in $GL(2,\mathbb{C})$ would be conjugate (group theoretically) to its complex conjugate and this is false.
To find a counter-example in your specific case, you should consider that being able to extend $f$ to the group $G$ implies that for each $n\in N$, $f(\phi(h)\cdot n)$ needs to be conjugate in $Aut(X)$ to $f(n)$ for all $h\in H$. This is a rather strong condition.